An rgb-t target tracking method based on multi-modal hierarchical relationship modeling

Abstract

The application discloses an RGB-T target tracking method based on multi-modal hierarchical relationship modeling, and the method gradually collects and fuses multi-modal image features in multiple stages of image feature learning by using a self-attention mechanism through a stacked multi-layer Transformer encoder structure. In the multi-modal interaction process of the whole network, the importance of the visible light information of each region in the tracking scene is dynamically solved by using an image block-based dynamic component feature fusion module, so that the interaction of the visible light information and the infrared information in the tracking process is adjusted, the complexity scene is better adapted, and better tracking performance is obtained.

Description

Technical Field

[0001] This invention relates to an RGB-T target tracking method based on multimodal hierarchical relationship modeling, which belongs to multimodal RGB-T target tracking technology and is an image processing technology. Background Technology

[0002] Target tracking, a crucial task in computer vision, aims to predict the target's position in subsequent frames given its coordinates in the first frame of a video. However, tracking tasks often face challenges from environmental interference, such as occlusion, low light, and rapid movement.

[0003] RGB-T tracking has become an important research direction in target tracking in recent years. It achieves good tracking performance even under the aforementioned challenges, mainly because it utilizes the advantages of both visible light and infrared (VLS) images. Due to their different imaging methods and principles, VLS and infrared images have different characteristics. Visible light images have high resolution and clear textures, providing complete visual information in normal environments. Infrared images, on the other hand, have lower resolution and provide limited information, but can supplement visible light images in extremely dark or other harsh environments. Therefore, the main purpose of RGB-T tracking is to leverage the complementary advantages of visible light and infrared images to overcome environmental interference, obtain more feature representations, and achieve robust video tracking. In recent years, many researchers have dedicated themselves to advancing RGB-T tracking, but due to insufficient utilization of RGB and thermal infrared information, there is still significant room for further research.

[0004] Transformer is a recently emerging architecture centered around an attention mechanism. Unlike convolutional networks, which are characterized by local dominance, Transformer excels in modeling global information. Retinex is a commonly used image enhancement method that decomposes an image into two components: a reflection image R and an illumination image L. Summary of the Invention

[0005] Objective: To improve the tracking performance of RGB-T tracking algorithms, this invention discloses an RGB-T target tracking method based on multimodal hierarchical relationship modeling. By stacking multiple layers of Transformer encoders, a self-attention mechanism is used to progressively aggregate and fuse multimodal image features at multiple stages of image feature learning. During the multimodal interaction process of the entire network, a dynamic particulate feature fusion module based on image patches is used to dynamically determine the importance of visible light information in each region of the tracking scene. This adjusts the interaction between visible light and infrared information during the tracking process, better adapting to complex scenes and achieving better tracking performance.

[0006] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0007] An RGB-T target tracking method based on multimodal hierarchical relationship modeling includes the following steps:

[0008] (1) Obtain the training sample set from the RGB-T data stream, and extract a corresponding visible light image R from each training sample. t And an infrared image T t From visible light image R t The visible light template image and the visible light search image are cropped from the infrared image T. t The infrared template image and the infrared search image are cropped from the image. The cropped image is then divided into blocks to obtain the visible light template block sequence z. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i ;

[0009] (2) Construct a dynamic component fusion module to obtain illumination information of visible light search image and estimate visible light weight value α∈(0,1). Visible light weight value α is used to describe the trustworthiness of visible light information in visible light search image. Visible light weight value α is used to dynamically guide the interaction between visible light mode and infrared mode throughout the process.

[0010] (3) The visible light template block sequence z is projected through a linear projection layer. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i Projecting the data into 2D features yields the visible light template label embedding sequence Zv and the visible light search label embedding sequence X. v Infrared template marker embedding sequence Z i and infrared search tag embedding sequence X i ;

[0011] (4) Construct an L-layer Transformer encoder and embed visible light template labels into sequence Z. v Infrared template marker embedding sequence Z i Visible light search tag embedding sequence X v and infrared search tag embedding sequence X i They are merged sequentially into a single sequence and input into the L-layer Transformer encoder in parallel for self-attention operation to obtain the total interactive embedding sequence E.

[0012] (5) Use the visible light weight value α to perform a weighted summation on the total interactive embedding sequence E to obtain the output embedding sequence E.fusion ;

[0013] (6) Embed the output sequence E fusion The spatial features are reconstructed into spatial features F. A score map M of spatial features F is obtained through a fully convolutional network. The position with the highest score is taken as the center of the target position to determine the final target box coordinates.

[0014] (7) Use a loss function to supervise the training process of the tracking network;

[0015] (8) Input the image to be tracked in the video stream into the trained tracking network and track the image to be tracked in the video stream. The image to be tracked belongs to a certain frame or part of a certain frame in the video stream.

[0016] Specifically, in step (1), the visible light template block sequence z is obtained. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i It includes the following steps:

[0017] (11) Extract the corresponding visible light image R t and infrared image T t From visible light image R t The visible light template image is cropped from the infrared image T, and the visible light search image is cropped by expanding the range of the visible light template image; t The infrared template image is cropped out from the image, and the range of the infrared template image is expanded to crop out the infrared search image;

[0018] (12) The cropped image is divided into blocks. According to the size, the visible light template image and the outer template image are divided into n×n blocks according to the template image block division rules. The visible light search image and the infrared search image are divided into N×N blocks according to the search image block division rules. The blocks are sorted to form a block sequence, which is the visible light template block sequence. Visible light search block sequence Infrared template block sequence and infrared search block sequence

[0019] Specifically, in step (2), obtaining the visible light weight value α through the dynamic component fusion module includes the following steps:

[0020] (21) Construct an illumination decoupling network and obtain the illumination component I of the visible light search image through the illumination decoupling network;

[0021] (22) According to the search image block division rules, the illumination component I is divided into N×N blocks;

[0022] (23) Construct a trainable neural network to map the segmented illumination components I to a range of (0,1) and estimate the reliability α of the visible light information in the l-th segment. l l = 1, 2, ..., N 2 .

[0023] Specifically, in step (21), the illumination component I of the visible light search image is obtained through an illumination decoupling network based on Retinex illumination enhancement theory (also known as low-light image enhancement theory), including the following steps:

[0024] (211) Based on the Retinex illumination enhancement theory, two convolutional network branches are set up for the visible light search image S, which are used to decompose the reflectance component R and the illumination component I of the visible light search image S, respectively, S = R·I; the visible light search image S first passes through a convolution-ReLU layer to obtain F temp ;

[0025] (212) In the first branch of the convolutional network, F temp The process involves downsampling through a max pooling layer, followed by downsampling through a first convolutional-ReLU layer and then another max pooling layer. This is then passed through a second convolutional-ReLU layer to obtain an intermediate result. This intermediate result undergoes a deconvolutional upsampling process, which is then concatenated with the output of the first convolutional-ReLU layer and fed into a third convolutional-ReLU layer. This third deconvolutional upsampling process is then performed, and the output is compared with F... temp The concatenation is performed and fed into the fourth convolutional-ReLU layer to obtain F. branch1 ;

[0026] (213) In the second convolutional network branch, F temp The output result is obtained by passing through a single convolutional-ReLU layer and comparing it with F. branch1 The light component I is obtained by splicing the components through a convolutional layer and a sigmoid activation function layer.

[0027] Specifically, in step (23), a trainable neural network is used to map the segmented illumination component I to a range of (0,1); the trainable neural network includes a global average pooling layer, a 1×1 convolutional layer, a fully connected layer, and a Sigmoid activation function, and the l-th block I of the illumination component I... l First, the size is adjusted using a global average pooling layer. Then, the channels are transformed using a 1×1 convolutional layer. Finally, a fully connected layer and a sigmoid activation function are used to map the result to a specific score, which is expressed as follows:

[0028] α l =σ(FC(Conv(GAP(I)l ))))

[0029] Among them: I l Let I represent the l-th block of illumination component I, Conv represent the convolution operation, GAP represent the global average pooling operation, FC represent the fully connected layer, and α represent the sigmoid activation function. l This indicates the level of trustworthiness of the visible light information in the l-th block.

[0030] Specifically, in step (3), obtaining the marker embedding sequence includes the following steps:

[0031] (31) The visible light template block sequence z v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i Flatten and project into 2D features;

[0032] (32) Add learnable template image position codes p to the block sequence z and search image location encoding p x The location information of each block is marked, and this process is described as follows:

[0033]

[0034]

[0035]

[0036]

[0037] Where: P is a learnable parameter of the linear projection layer.

[0038] Specifically, in step (4), an L-layer Transformer encoder is constructed to perform self-attention operations on the labeled embedding sequence. Parallel inputs are fed into an L-layer Transformer encoder for self-attention operations. The L-layer Transformer encoder is a multimodal structure composed of L Transformer encoder layers. Unlike multi-input cross-attention, self-attention is a process of pairwise interaction between input label embeddings, enhancing its own features by generating an attention matrix. The m-th layer Transformer encoder is described as follows:

[0039] A m =Encode(A m-1 )

[0040] Among them: A m and A m-1These represent the outputs of the m-th and (m-1)-th Transformer encoders, respectively, where m = 1, 2, ..., L. The total sequence of interactive embeddings E = A L The visible light template embedding sequence is extracted from the total interactive embedding sequence E. Visible light search embedding sequence Infrared template embedding sequence and infrared search embedding sequence

[0041] Specifically, the Transformer encoder adopts the existing ViT structure used in downstream tasks, including two normalization layers, a multi-head self-attention layer, and a multi-layer perceptron layer. It performs two residual connections during encoding. The encoding process is described as follows:

[0042]

[0043] Among them: A m-1 denoted as the input of the m-th layer Transformer encoder, LN represents layer normalization, MHA represents multi-head self-attention, and MLP represents multilayer perceptron, which consists of two fully connected layers and a GELU activation function.

[0044] Specifically, in step (5), according to the function Obtain the output embedded sequence E fusion .

[0045] Specifically, in step (6), the spatial feature F is input into the fully convolutional network to obtain the score map M, the local offset O, and the normalized target box size s; the score map M is linearly transformed (simply multiplied by a Hanning window of the same size) to obtain a score matrix of the same size as the search image; combined with the position information, the position with the highest score is selected as the center of the target position; combined with the normalized target box size s, and corrected using the local offset O, the final target box coordinates are obtained.

[0046] Specifically, in the fully convolutional network, each convolutional block includes a convolutional layer, a normalization layer, and a ReLU activation function layer.

[0047] Specifically, in step (7), a weighted focus loss is used. As the classification loss, GIoU loss is used. As a regression loss, it is combined with the mean absolute error loss. The total loss of the tracking network is calculated as follows:

[0048]

[0049]

[0050]

[0051] Where: λ1 and 2 are equilibrium parameters (set to λ1 = 2 and λ2 = 5 in the experiment); M xy This represents the predicted score at position (x, y) in the score map M. The score at position (x,y) in the ground truth heatmap generated using a Gaussian kernel is given, where β and μ are hyperparameters (set to β=2 and μ=4 during training), and H and W are the length and width of the input image (visible light image R). t and infrared image T t (length and width); IoU is the intersection-union ratio, A c ρ is the minimum bounding box area of ​​the ground truth bounding box and the predicted bounding box, and ρ is the union area of ​​the ground truth bounding box and the predicted bounding box.

[0052] Beneficial effects: The RGB-T target tracking method based on multimodal hierarchical relationship modeling provided by this invention, through stacking multi-layer Transformer encoder structures, utilizes a self-attention mechanism to progressively gather and fuse multimodal image features at multiple stages of image feature learning, which can fully exploit the usable information of multimodal data; at the same time, by using a dynamic part feature fusion module based on image patches to adjust the interaction between visible light information and infrared information during the tracking process, it can better adapt to complex scenes and improve tracking performance and tracking speed. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of a system for target tracking using the method of the present invention;

[0055] Figure 3 This is a schematic diagram of the structure of an illumination decoupling network. Detailed Implementation

[0056] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1This paper presents an RGB-T target tracking method based on multimodal hierarchical relationship modeling. The input is a pair of visible light and infrared images from a video stream. Template and search images are obtained through cropping, and after segmentation, corresponding image block sequences are obtained. These block sequences are projected into 2D labeled embedding sequences via a linear projection layer. These sequences are then cascaded and input in parallel to multiple ViT encoders, forming the backbone of a Siamese network. This network is used to perform feature learning and interaction between the template and search images, and between the visible light and infrared modalities. At any stage of feature interaction, including the encoding and weighted summation stages, the visible light image illumination map is decomposed, and a trainable network is used to calculate the corresponding weights for the visible light modalities for each region in the visible light image. Finally, the obtained visible light and infrared search region features are weighted and summed to reconstruct spatial features, which are then input into the prediction head for subsequent target classification and regression. The specific steps are explained below.

[0058] Step S01: Obtain the training sample set from the RGB-T data stream, and extract the corresponding visible light image R from each training sample. t And an infrared image T t From visible light image R t The visible light template image and the visible light search image are cropped from the infrared image T. t The infrared template image and the infrared search image are cropped from the image. The cropped image is then divided into blocks to obtain the visible light template block sequence z. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i .

[0059] (11) Extract the corresponding visible light image R t and infrared image T t From visible light image R t A visible light template image of size 192×192 was cropped from the image, and the visible light template image was expanded to crop a visible light search image of size 384×384; from the infrared image T... t An infrared template image of size 192×192 is cropped from the image, and the range of the infrared template image is expanded to crop an infrared search image of size 384×384.

[0060] (12) The cropped image is divided into blocks. According to the size, the visible light template image and the outer template image are divided into n×n blocks according to the template image block division rules. The visible light search image and the infrared search image are divided into N×N blocks according to the search image block division rules. The blocks are sorted to form a block sequence, which is the visible light template block sequence. Visible light search block sequence Infrared template block sequence and infrared search block sequence

[0061] Step S02: Construct a dynamic component fusion module (e.g.) Figure 2 (Right side) Obtain illumination information of the visible light search image and estimate the visible light weight value α∈(0,1). The visible light weight value α is used to describe the trustworthiness of the visible light information in the visible light search image. The visible light weight value α is used to dynamically guide the interaction between the visible light mode and the infrared mode throughout the process.

[0062] (21) Construct an illumination decoupling network based on Retinex illumination enhancement theory (e.g.) Figure 3 As shown), the illumination component I of the visible light search image is obtained through an illumination decoupling network; the steps include the following.

[0063] (211) Based on the Retinex illumination enhancement theory, two convolutional network branches are set up for the visible light search image S, which are used to decompose the reflectance component R and the illumination component I of the visible light search image S, respectively, S = R·I; the visible light search image S first passes through a convolution-ReLU layer to obtain F temp .

[0064] (212) In the first branch of the convolutional network, F temp The process involves downsampling through a max pooling layer, followed by downsampling through a first convolutional-ReLU layer and then another max pooling layer. This is then passed through a second convolutional-ReLU layer to obtain an intermediate result. This intermediate result undergoes a deconvolutional upsampling process, which is then concatenated with the output of the first convolutional-ReLU layer and fed into a third convolutional-ReLU layer. This third deconvolutional upsampling process is then performed, and the output is compared with F... temp The concatenation is performed and fed into the fourth convolutional-ReLU layer to obtain F. branch1 .

[0065] (213) In the second convolutional network branch, F temp The output result is obtained by passing through a single convolutional-ReLU layer and comparing it with F. branch1 The light component I is obtained by splicing the components through a convolutional layer and a sigmoid activation function layer.

[0066] (22) According to the search image block rules, the illumination component I is divided into N×N blocks.

[0067] (23) Construct a trainable neural network to map the segmented illumination component I to a range of (0,1). The trainable neural network includes a global average pooling layer, a 1×1 convolutional layer, a fully connected layer, and a Sigmoid activation function. The l-th block I of the illumination component I... lFirst, the size is adjusted using a global average pooling layer, then the channels are converted using a 1×1 convolutional layer, and finally mapped to a specific score value using a fully connected layer and a sigmoid activation function. The reliability α of the visible light information in the i-th block is estimated. i for:

[0068] α l =σ(FC(Conv(GAP(I) l ))))

[0069] Where: l = 1, 2, ..., N 2 I l Let I represent the l-th block of illumination component I, Conv represent the convolution operation, GAP represent the global average pooling operation, FC represent the fully connected layer, σ represent the sigmoid activation function, and α represent the sigmoid activation function. l The reliability of visible light information in the l-th block is represented by α = [α1; α2; ...; N2 ].

[0070] Step S03: Project the visible light template block sequence z through a linear projection layer. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i Projecting the data into 2D features yields the visible light template label embedding sequence Zv, the visible light search label embedding sequence Xv, and the infrared template label embedding sequence Z. i and infrared search tag embedding sequence X i .

[0071] (31) The visible light template block sequence z v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i Flatten and project into 2D features.

[0072] (32) Add learnable template image position codes p to the block sequence z and search image location encoding p x The location information of each block is marked, and this process is described as follows:

[0073]

[0074]

[0075]

[0076]

[0077] Where: P is a learnable parameter of the linear projection layer.

[0078] Step S04: Construct an L-layer Transformer encoder and embed the visible light template markers into sequence Z. v Infrared template marker embedding sequence Z i Visible light search tag embedding sequence X v and infrared search tag embedding sequence X i They are merged into a single sequence in sequence and input into the L-layer Transformer encoder in parallel for self-attention operation to obtain the total interactive embedding sequence E.

[0079] like Figure 2 As shown in the middle section, an L-layer Transformer encoder is constructed to perform self-attention operations on the labeled embedding sequence. First, the... Parallel inputs are fed into an L-layer Transformer encoder for self-attention operations. The L-layer Transformer encoder is a multimodal structure composed of L Transformer encoder layers. Unlike multi-input cross-attention, self-attention is a process of pairwise interaction between input label embeddings, enhancing its own features by generating an attention matrix. The m-th layer Transformer encoder is described as follows:

[0080] A m =Encode(A m-1 )

[0081] Among them: A m and A m-1 These represent the outputs of the m-th and (m-1)-th Transformer encoders, respectively, where m = 1, 2, α, L. The total sequence of interactive embeddings E = A L The visible light template embedding sequence is extracted from the total interactive embedding sequence E. Visible light search embedding sequence Infrared template embedding sequence and infrared search embedding sequence

[0082] The Transformer encoder employs a ViT architecture for downstream tasks, comprising two normalization layers, a multi-head self-attention layer, and a multi-layer perceptron layer. It performs two residual connections during encoding. The encoding process is described as follows:

[0083] Encode′(A m-1 )=MHA(LN(E))+A m-1

[0084] Encode(A m-1=MLP(LN(Encode′(A)) m-1 )))+Encode'(A m-1 )

[0085] Among them: A m-1 denoted as the input of the m-th layer Transformer encoder, LN represents layer normalization, MHA represents multi-head self-attention, and MLP represents multilayer perceptron, which consists of two fully connected layers and a GELU activation function.

[0086] Step S05: Use the visible light weight value α to perform a weighted summation on the total interactive embedding sequence E to obtain the output embedding sequence E. fusion .

[0087] Calculate the output embedded sequence E fusion At that time, only visible light is used to search for embedded sequences. and infrared search embedding sequence The calculation formula is

[0088] Step S06: Embed the output sequence E fusion The data is reconstructed into spatial features F. A score map M of spatial features F is obtained through a fully convolutional network. The position with the highest score is taken as the center of the target position to determine the final target box coordinates.

[0089] Spatial features F are input into a fully convolutional network, in which each convolutional block includes a convolutional layer, a normalization layer, and a ReLU activation function layer. The fully convolutional network is used to obtain a score map M, a local offset O, and a normalized target box size s. A linear transformation is performed on the score map M to obtain a score matrix of the same size as the search image. Combining the position information, the position with the highest score is selected as the center of the target position. The normalized target box size s is combined with the local offset O for correction to obtain the final target box coordinates.

[0090] Step S07: Use a loss function to supervise the training process of the tracking network.

[0091] Using weighted focus loss As the classification loss, GIoU loss is used. As a regression loss, it is combined with the mean absolute error loss. The total loss of the tracking network is calculated as follows:

[0092]

[0093]

[0094]

[0095] Where: λ1 and 2 are equilibrium parameters (set to λ1 = 2 and λ2 = 5 in the experiment); M xy This represents the predicted score at position (x, y) in the score map M. The score at position (x,y) in the ground truth heatmap generated using a Gaussian kernel is given, where β and μ are hyperparameters (set to β=2 and μ=4 during training), and H and W are the length and width of the input image (visible light image R). t and infrared image T t (length and width); IoU is the intersection-union ratio, A c ρ is the minimum bounding box area of ​​the ground truth bounding box and the predicted bounding box, and ρ is the union area of ​​the ground truth bounding box and the predicted bounding box.

[0096] Step S08: Input the image to be tracked in the video stream into the trained tracking network, and track the image to be tracked in the video stream. The image to be tracked belongs to a certain frame or a part of a certain frame in the video stream.

[0097] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. An RGB-T target tracking method based on multimodal hierarchical relationship modeling, characterized in that: Includes the following steps: (1) Obtain the training sample set from the RGB-T data stream, and extract the corresponding visible light image R from each training sample. t And an infrared image T t From visible light image R t The visible light template image and the visible light search image are cropped from the infrared image T. t The infrared template image and the infrared search image are cropped from the image. The cropped image is then divided into blocks to obtain the visible light template block sequence z. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i ; (2) Construct a dynamic component fusion module to obtain illumination information of visible light search image and estimate visible light weight value (0,1). Visible light weight value α is used to describe the trustworthiness of visible light information in visible light search image. Visible light weight value α is used to dynamically guide the interaction between visible light mode and infrared mode throughout the process. (3) The visible light template block sequence z is projected through a linear projection layer. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i Projecting the data into 2D features yields the visible light template label embedding sequence Z. v Visible light search tag embedding sequence X v Infrared template marker embedding sequence Z i and infrared search tag embedding sequence X i ; (4) Construct an L-layer Transformer encoder and embed visible light template labels into sequence Z. v Infrared template marker embedding sequence Z i Visible light search tag embedding sequence X v and infrared search tag embedding sequence X i They are merged sequentially into a single sequence and input into the L-layer Transformer encoder in parallel for self-attention operation to obtain the total interactive embedding sequence E. (5) Use the visible light weight value α to perform a weighted summation on the total interactive embedding sequence E to obtain the output embedding sequence E. fusion ; (6) Embed the output sequence E fusion The spatial features are reconstructed into spatial features F. A score map M of spatial features F is obtained through a fully convolutional network. The position with the highest score is taken as the center of the target position to determine the final target box coordinates. (7) Use a loss function to supervise the training process of the tracking network; (8) Input the image to be tracked in the video stream into the trained tracking network and track the image to be tracked in the video stream. The image to be tracked belongs to a certain frame or part of a certain frame in the video stream.

2. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (1), the visible light template block sequence z is obtained. v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i It includes the following steps: (11) Extract the corresponding visible light image R t and infrared image T t From visible light image R t The visible light template image is cropped from the infrared image T, and the visible light search image is cropped by expanding the range of the visible light template image; t The infrared template image is cropped out from the image, and the range of the infrared template image is expanded to crop out the infrared search image; (12) The cropped image is divided into blocks. According to the size, the visible light template image and the outer template image are divided into n×n blocks according to the template image block division rules. The visible light search image and the infrared search image are divided into N×N blocks according to the search image block division rules. The blocks are sorted to form a block sequence, which is the visible light template block sequence. Visible light search block sequence Infrared template block sequence and infrared search block sequence 3. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (2), the visible light weight value α is obtained through the dynamic component fusion module, including the following steps: (21) Construct an illumination decoupling network and obtain the illumination component I of the visible light search image through the illumination decoupling network; (22) According to the search image block division rules, the illumination component I is divided into N×N blocks; (23) Construct a trainable neural network to map the segmented illumination components I to a range of (0,1) and estimate the reliability α of the visible light information in the l-th segment. l l = 1, 2, ..., N 2 .

4. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 3, characterized in that: In step (21), the illumination component I of the visible light search image is obtained through an illumination decoupling network based on Retinex illumination enhancement theory, including the following steps: (211) Based on the Retinex illumination enhancement theory, two convolutional network branches are set up for the visible light search image S, which are used to decompose the reflectance component R and the illumination component I of the visible light search image S, respectively, S = R·I; the visible light search image S first passes through a convolution-ReLU layer to obtain F temp ; (212) In the first branch of the convolutional network, F temp The process involves downsampling through a max pooling layer, followed by downsampling through a first convolutional-ReLU layer and then another max pooling layer. This is then passed through a second convolutional-ReLU layer to obtain an intermediate result. This intermediate result undergoes a deconvolutional upsampling process, which is then concatenated with the output of the first convolutional-ReLU layer and fed into a third convolutional-ReLU layer. This third deconvolutional upsampling process is then performed, and the output is compared with F... temp The concatenation is performed and fed into the fourth convolutional-ReLU layer to obtain F. branch1 ; (213) In the second convolutional network branch, F temp The output result is obtained by passing through a single convolutional-ReLU layer and comparing it with F. branch1 The light component I is obtained by splicing the components through a convolutional layer and a sigmoid activation function layer.

5. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 3, characterized in that: In step (23), a trainable neural network is used to map the segmented illumination component I to a range of (0,1); the trainable neural network includes a global average pooling layer, a 1×1 convolutional layer, a fully connected layer, and a Sigmoid activation function, and the l-th block I of the illumination component I... l First, the size is adjusted using a global average pooling layer. Then, the channels are transformed using a 1×1 convolutional layer. Finally, a fully connected layer and a sigmoid activation function are used to map the result to a specific score, which is expressed as follows: a l =σ(FC(Conv(GAP(I l )))) Among them: I l Let I represent the l-th block of illumination component I, Conv represent the convolution operation, GAP represent the global average pooling operation, FC represent the fully connected layer, σ represent the sigmoid activation function, and α represent the sigmoid activation function. l This indicates the level of trustworthiness of the visible light information in the l-th block.

6. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (3), obtaining the marker embedding sequence includes the following steps: (31) The visible light template block sequence z v Visible light search block sequence x v Infrared template block sequence z i and infrared search block sequence x i Flatten and project into 2D features; (32) Add learnable template image position codes p to the block sequence z and search image location encoding p x The location information of each block is marked, and this process is described as follows: Where: P is a learnable parameter of the linear projection layer.

7. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (4), an L-layer Transformer encoder is constructed to perform self-attention operations on the labeled embedding sequence. Parallel inputs are fed into an L-layer Transformer encoder for self-attention computation. The L-layer Transformer encoder is a multimodal structure composed of L Transformer encoder layers. The m-th layer Transformer encoder is described as follows: A m =Encode(A m-1 ) Among them: A m and A m-1 These represent the outputs of the m-th and (m-1)-th Transformer encoders, respectively, where m = 1, 2, ..., L. The total sequence of interactive embeddings E = A L The visible light template embedding sequence is extracted from the total interactive embedding sequence E. Visible light search embedding sequence Infrared template embedding sequence and infrared search embedding sequence 8. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (5), according to the function Obtain the output embedded sequence E fusion .

9. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (6), the spatial feature F is input into the fully convolutional network to obtain the score map M, the local offset O and the normalized target box size s; the score map M is linearly transformed to obtain a score matrix with the same size as the search image; combined with the position information, the position with the highest score is selected as the center of the target position; combined with the normalized target box size s and corrected using the local offset O, the final target box coordinates are obtained.

10. The RGB-T target tracking method based on multimodal hierarchical relationship modeling according to claim 1, characterized in that: In step (7), a weighted focus loss is used. As the classification loss, GIoU loss is used. As a regression loss, it is combined with the mean absolute error loss. The total loss of the tracking network is calculated as follows: Where: λ1 and λ2 are equilibrium parameters; M xy This represents the predicted score at position (x, y) in the score map M. The score at position (x,y) in the ground truth heatmap generated using a Gaussian kernel is given by β and μ, which are hyperparameters, and H and W are the length and width of the input image, respectively; IoU is the intersection-over-union ratio, and A is the cross-hatching ratio. c ρ is the minimum bounding box area of ​​the ground truth bounding box and the predicted bounding box, and ρ is the union area of ​​the ground truth bounding box and the predicted bounding box.

Images