Dual-modal target tracking method and device based on infrared and visible light images

Abstract

The present application relates to a dual-modal target tracking method and device based on infrared and visible light images. The method comprises: pre-processing the acquired data of the same pair of visible light-infrared dual-modal images, and performing template and search area cropping on the first frame image of the sequence according to the obtained visible light-infrared image; using the obtained template and search area to train the dual-modal target tracking model to obtain a trained dual-modal target tracking model; the model comprises a feature extraction network, a dual-feature aggregation module and a detection module; using the initial frame target position as a template, the trained dual-modal target tracking model is used for reasoning to determine the target position in the search area. Based on visible light target tracking, this method introduces infrared modal data, which increases the robustness in dark scenes, helps drones track targets in low-light, high-brightness and other scenes, and effectively expands the scope of application scenarios of drone target tracking algorithms.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a dual-modal target tracking method and device based on infrared and visible light images. Background Art

[0002] Target tracking is a fundamental task in computer vision and has widespread applications in everyday life. In the drone industry, target tracking is a prerequisite for many drone applications, so an accurate and reliable single-target tracking algorithm is crucial. As the price and weight of infrared sensors gradually decrease, infrared sensors, visible light sensors, and laser rangefinders have been integrated into a three-light module as a drone's sensor payload. Current drone tracking algorithms are primarily based on visible light target tracking, which does not fully utilize the potential of drone infrared-visible light data. Furthermore, visible light tracking algorithms fail to track properly in many scenarios (such as those with varying lighting conditions and background occlusion). In these situations, infrared image information can effectively supplement visible light image information, thereby improving the robustness of the tracking process.

[0003] In recent years, with the widespread adoption of infrared sensors, an increasing number of drones have begun carrying infrared payloads. Infrared images are produced by measuring the heat released by an object. Compared to visible light images, infrared images have relatively poor visual quality but are rich in target location information. Furthermore, infrared images are insensitive to changes in scene brightness, maintaining excellent imaging quality even in scenes with poor lighting conditions. However, infrared images suffer from poor imaging quality, with limited spatial resolution and grayscale dynamic range, and lacking image detail and texture information. In contrast, visible light images are rich in detail and texture features. Due to the complementary nature of infrared and visible light imaging, visible-light-infrared target tracking has garnered increasing attention.

[0004] As drone missions become more diverse, more and more of them place higher demands on drone target tracking. This requires robust tracking algorithms to maintain accurate target tracking in challenging scenarios. Currently, while optical tracking algorithms provide excellent tracking performance, they struggle under conditions of drastic lighting changes and in darkness. Therefore, the inclusion of infrared modal information to enhance the robustness of tracking algorithms is particularly beneficial. Infrared images rely on the target's inherent thermal radiation and are insensitive to illumination, making them effective even in the dark. This makes them particularly useful for drones tracking targets in the dark.

[0005] Target tracking technology is a key technology for the application of unmanned aerial vehicles (UAVs) in both civilian and military applications, and has gradually become a prerequisite for many downstream UAV tasks. Many UAV tasks rely on target pixel coordinates provided by target tracking. Therefore, the reliability and accuracy of target tracking methods are crucial for achieving multiple tasks, such as target positioning. Researchers have long conducted extensive research on this topic, achieving fruitful results. Target tracking methods based on correlation filters are widely used in the UAV field due to their fast computational speed and low complexity. Existing visual target tracking methods operate on single-light data, failing to exploit the potential of dual-light data. Accurate target tracking is difficult in some scenarios, such as at night. To address this issue, researchers have begun studying dual-modal target tracking algorithms. Compared to the aforementioned single-light target tracking algorithms, dual-modal target tracking algorithms utilize both visible and infrared image information. Leveraging the complementary nature of visible and infrared image information, these algorithms overcome the shortcomings of visible light, which is sensitive to illumination changes, and infrared images, which lack texture information. However, compared with the single-light algorithm, the current dual-light tracking algorithm has the characteristics of large data volume and complex network structure, which makes its operation speed relatively slow. Therefore, a visible light-infrared dual-modal target tracking algorithm with higher accuracy and faster speed is very important for UAV applications.

[0006] Most existing target tracking methods use single-light data. However, due to the limitations of visible light or infrared images in some scenarios, tracking performance can be poor or even lost. Existing dual-light tracking algorithms, while utilizing dual-modal data, are slow and inaccurate, and their tracking performance needs further improvement. Summary of the Invention

[0007] Based on this, it is necessary to provide a dual-modal target tracking method and device based on infrared and visible light images to address the above technical problems.

[0008] A dual-modal target tracking method based on infrared and visible light images, the method comprising:

[0009] The obtained same pair of visible light-infrared dual-modal image data is preprocessed to obtain a visible light-infrared image pair sequence.

[0010] The first frame of the visible light-infrared image pair sequence is cropped to obtain a template and a search area of the first frame of the visible light-infrared image pair.

[0011] The template and search area of the visible light-infrared image pair are used as input to train a dual-modal target tracking model to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module and a detection module; the feature extraction network is used to extract the visible light image features and infrared image features of the template and search area of the visible light-infrared image pair respectively; the dual-feature aggregation module is used to perform feature aggregation on the visible light image features and the infrared image features using a spatial aggregation module and a channel aggregation module to obtain an aggregated template and a search area feature map; the detection module is used to perform correlation processing on the aggregated template and search area feature map, send the obtained processing results to the detection head, and regress the predicted bounding box of the target.

[0012] The infrared-visible multi-frame video pair of the target to be tracked is input into the trained dual-modal target tracking model, and the target position in the initial frame is used as a template to determine the target position in the search area.

[0013] In one embodiment, preprocessing the acquired same pair of visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence includes:

[0014] Acquire the same pair of visible light and infrared dual-modal image data of the target.

[0015] Grayscale processing and augmentation processing are sequentially performed on the same pair of visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence.

[0016] In one embodiment, a template of a visible light-infrared image pair and a search area are used as input to train a dual-modal target tracking model to obtain a trained dual-modal target tracking model, including:

[0017] The template and search area of the visible light-infrared image pair are input into the feature extraction network to obtain visible light image features and infrared image features.

[0018] The visible light image features and the infrared image features are input into the dual-feature aggregation module to obtain an aggregated template and a search area feature map.

[0019] The aggregated template and the search area feature map are input into the detection module to obtain the predicted bounding box of the target.

[0020] The dual-modal target tracking model is reversely trained according to the predicted bounding box and the true position to obtain a trained dual-modal target tracking model.

[0021] In one embodiment, the feature extraction network is two independent feature extraction modules with the same structure, and the feature extraction module includes the first three layers of the Swin-transformer network.

[0022] The template and search area of the visible light-infrared image pair are input into the feature extraction network to obtain visible light image features and infrared image features, including:

[0023] The target and search area of the visible light image are patch-segmented and input into the first feature extraction module of the feature extraction network, where attention operation is performed on the small patch blocks to obtain the template features and search area features of the visible light image.

[0024] The target and search area of the infrared image are segmented into patches and then input into the second feature extraction module of the feature extraction network to perform attention operation on the small patch blocks to obtain the template features and search area features of the infrared image.

[0025] In one embodiment, the dual-feature aggregation module includes a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a visible light branch and an infrared branch; the visible light branch includes a multi-head attention module and a normalization processing layer;

[0026] Inputting the visible light image features and the infrared image features into the dual-feature aggregation module to obtain an aggregated template and a search area feature map, including:

[0027] The visible light image features and the infrared image features are input into the spatial aggregation module of the dual-feature aggregation module to obtain calibrated visible light and infrared image features:

[0028] SA RGB =F I +(MCA(F I ,Concat s (F RGB ,F I )))

[0029] SA I =F RGB +(MCA(F RGB ,Concat s (F I ,F RGB )))

[0030] Among them, F RGB and F I They are the visible light image features and infrared image features extracted by the feature extraction network, MCA(.) represents a multi-head mutual attention module, Concat s(.) indicates splicing in the spatial dimension.

[0031] The calibrated visible light and infrared image features are input into the channel aggregation module, and the aggregated template and search area feature map are obtained as follows:

[0032] weight RGB =Softmax(Γ RGB (Concat c (SA RGB ,SA I )))

[0033] weight I =Softmax(Γ I (Concat c (SA RGB ,SA I )))

[0034] Feature=weight RGB ×SA RGB +weught I ×SA I

[0035] Among them, SA RGB and SA I are the outputs of the spatial aggregation module, Γ RGB and Γ I The pooling layer is used to aggregate channel information, Concat c (.) indicates concatenation in the channel dimension, and Feature is the output of the dual-feature aggregation module.

[0036] In one embodiment, the detection module includes an image-related operation module and a detection head, and the image-related operation module includes a cascade-based encoder and decoder in SwinTrack; the encoder consists of a multi-head self-attention module and a feedforward network, and a residual structure is used in the multi-head self-attention module and the feedforward network module; the structure of the decoder is obtained by replacing the multi-head self-attention module in the encoder with a multi-head mutual attention module.

[0037] The aggregated template and search area feature map are input into the detection module to obtain the predicted bounding box of the target, including:

[0038] The aggregated template and search area feature map are spliced and input into the encoder of the image correlation operation module of the detection module to obtain the template and search area after separation and splicing features:

[0039] f d=DeConcat(F FFN ), d=x,z

[0040] F MSA =Concat(f z ,f x )+MSA(LN(Concat(f z ,f x )))

[0041] F FFN =MLP(LN(F MSA ))+F MSA

[0042] Among them, f x and f z They are respectively the template feature and the search area feature after dual feature aggregation, F MSA is the output of the residual block composed of multiple self-attention modules, F FFN is the output of the residual block composed of the feedforward network, MLP is the feedforward network, MSA is the multi-head self-attention module, Concat is the splicing operation, and LN is the layer normalization.

[0043] The template and search area obtained by separating and splicing the features are input into the decoder of the image correlation operation module of the detection module, and the decoded output is:

[0044] F=MLP(LN(F MCA ))+F MCA

[0045] F MCA =f x +MCA(LN(f x ),LN(Concat(f x ,f z )))

[0046] Where F is the output of the decoder, F MCA It is the output of the residual block composed of multiple mutual attention modules.

[0047] The decoded output is input into the detection head of the detection module to obtain a predicted bounding box of the target.

[0048] In one embodiment, the detection head includes a classification head and a regression head, and the classification head is a multi-layer perception network.

[0049] In one embodiment, the feedforward network includes a two-layer multilayer perceptron.

[0050] In one embodiment, a multi-frame infrared-visible light video pair of a target to be tracked is input into a trained dual-modal target tracking model, and the target position in the initial frame is used as a template to determine the target position in the search area, including:

[0051] The target position of the initial frame of the infrared-visible light multi-frame video pair of the target to be tracked is intercepted as a template, the visible light-infrared features of the template are extracted, and the visible light-infrared branch features are aggregated to obtain the template graph.

[0052] In the video frames after the initial frame, the search area features are extracted, the visible light-infrared features of the search area are aggregated, and correlated with the template image to obtain the feature map with the largest response. The feature map is sent to the detection head to classify the target.

[0053] A Hamming window is used to add prior information of the target position in the image and penalize sudden changes in the target position.

[0054] The obtained target position is used as the center position of the new search area for cropping to obtain a new search area. The above process is repeated until the preset conditions are met and the target position is determined in the search area.

[0055] A dual-modal target tracking device based on infrared and visible light images, the device comprising:

[0056] The dual-modal image preprocessing module is used to preprocess the acquired same pair of visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence.

[0057] The template and search area determination module is used to crop the first frame of the visible light-infrared image pair sequence to obtain the template and search area of the first frame of the visible light-infrared image pair.

[0058] A dual-modal target tracking model training module is used to take the template and search area of the visible light-infrared image pair as input to train the dual-modal target tracking model to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module and a detection module; the feature extraction network is used to extract the visible light image features and infrared image features of the template and search area of the visible light-infrared image pair respectively; the dual-feature aggregation module is used to perform feature aggregation on the visible light image features and the infrared image features using a spatial aggregation module and a channel aggregation module to obtain an aggregated template and a search area feature map; the detection module is used to perform correlation processing on the aggregated template and search area feature map, send the obtained processing results to the detection head, and regress the predicted bounding box of the target.

[0059] The target tracking module is used to input the infrared-visible light multi-frame video pair of the target to be tracked into the trained dual-modal target tracking model, and determine the target position in the search area using the initial frame target position as a template.

[0060] The aforementioned dual-modal target tracking method and device based on infrared and visible light images includes: preprocessing the acquired visible-infrared dual-modal image data to obtain a visible-infrared image pair sequence; cropping the first frame of the visible-infrared image pair sequence to obtain a template and search area for the first frame of the visible-infrared image pair; training a dual-modal target tracking model using the template and search area of the visible-infrared image pair as input to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module, and a detection module; inputting a multi-frame infrared-visible light video pair of the target to be tracked into the trained dual-modal target tracking model, and determining the target position within the search area using the target position in the initial frame as a template. This method, based on a visible light target tracking algorithm, introduces infrared modal data, increasing the algorithm's robustness in dark scenes. It can help drones track targets in low-light and high-brightness scenarios, effectively expanding the application scenarios of drone target tracking algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 1 is a flow chart of a dual-modal target tracking method based on infrared and visible light images in one embodiment;

[0062] Figure 2 A schematic diagram of a feature extraction network structure in another embodiment;

[0063] Figure 3 This is a schematic diagram of the structure of a dual-feature aggregation module in another embodiment;

[0064] Figure 4 A schematic diagram of the detection module structure in another embodiment;

[0065] Figure 5 is a flowchart of a dual-modal target tracking method based on infrared and visible light images in another embodiment;

[0066] Figure 6This is a display of measured data in different scenarios in another embodiment, where (a) and (b) are the visible light image and red image in a low-light scenario, respectively; (c) and (d) are the visible light image and red image in a strong-light scenario, respectively; (e) and (f) are the visible light image and red image in a scene with drastic lighting changes, respectively; (g) and (h) are the visible light image and red image in a scene with tree occlusion, respectively; (i) and (j) are the visible light image and red image in a scene with cloud occlusion, respectively; and (k) and (l) are the visible light image and red image in a scene with appearance changes, respectively.

[0067] Figure 7 In another embodiment, a heat map is used to analyze the difference between using only channel aggregation and dual-feature aggregation, where (a) is direct channel aggregation and (b) is dual-feature aggregation.

[0068] Figure 8 This is a heat map visualization of the input of each module of the method of the present application for an image pair in another embodiment, where (a) is a Bike data sequence, (b) is a pedestrian-211 data sequence, and (c) is a Tricy-006 data sequence;

[0069] Figure 9 Here are some failure cases of the method of the present application in another embodiment, where (a) is similar in appearance, (b) is shape change, and (c) is image noise;

[0070] Figure 10 1 is a structural diagram of a dual-modal target tracking device based on infrared and visible light images in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0072] In one embodiment, Figure 1 As shown, a dual-modal target tracking method based on infrared and visible light images is provided, and the method includes the following steps:

[0073] Step 100: Preprocessing the acquired same pair of visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence.

[0074] Specifically, the same pair of visible light and infrared dual-modal image data is read; a grayscale change with a probability of 5% is performed on the image pair to enhance the generalization of the algorithm for various scenarios; the pixels of the visible light and infrared image pair are normalized using the mean [0.485, 0.456, 0.406] and variance [0.229, 0.2240.225] obtained by statistics on the ImageNet dataset; and image augmentation operations are performed on the image pair with a brightness jitter of 3 and a scale jitter of 0.5.

[0075] Step 102: cropping the first frame of the visible light-infrared image pair sequence to obtain a template and a search area for the first frame of the visible light-infrared image pair.

[0076] Specifically, the first frame of the image pair sequence is annotated, and the template (Template) and search area (Search) data used for image training are extracted to complete the algorithm initialization; the template is cropped with the annotated target position as the center, and its area multiplier is 2 (target annotation box + background padding), and resampled to obtain a template input of size 112×112×6; the search area is cropped with the target position as the center, and its area multiplier is 5, and resampled to obtain a search area input of size 224×224×6; the cropped template and search area are used as network input to start network training.

[0077] Step 104: The template and search area of the visible light-infrared image pair are used as input to train the dual-modal target tracking model to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module and a detection module; the feature extraction network is used to extract the visible light image features and infrared image features of the template and search area of the visible light-infrared image pair respectively; the dual-feature aggregation module is used to perform feature aggregation on the visible light image features and the infrared image features using the spatial aggregation module and the channel aggregation module to obtain an aggregated template and a search area feature map; the detection module is used to perform correlation processing on the aggregated template and the search area feature map, send the obtained processing results to the detection head, and regress the predicted bounding box of the target.

[0078] Specifically, first, the visible light and infrared features of the template and search area input to the dual-modal target tracking model are extracted respectively, and the visible light and infrared template and search area feature maps are obtained respectively.

[0079] A dual feature aggregation module is used to perform feature aggregation on the visible light and infrared feature maps respectively to obtain the aggregated template and search area feature map.

[0080] In the detection module, the search area is correlated with the template image and then fed into a multi-layer perception network to regress the position and bounding box of the target in the current frame.

[0081] Compare with the true position and bounding box, calculate the error, backpropagate the error to the network, and update the network weight value.

[0082] Step 106: Input the infrared-visible light multi-frame video pair of the target to be tracked into the trained dual-modal target tracking model, and use the initial frame target position as a template to determine the target position in the search area.

[0083] The aforementioned dual-modal target tracking method based on infrared and visible light images includes: preprocessing the acquired visible-infrared dual-modal image data to obtain a visible-infrared image pair sequence; cropping the first frame of the visible-infrared image pair sequence to obtain a template and search area for the first frame of the visible-infrared image pair; training a dual-modal target tracking model using the template and search area of the visible-infrared image pair as input to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module, and a detection module; inputting a multi-frame infrared-visible light video pair of the target to be tracked into the trained dual-modal target tracking model, and determining the target position within the search area using the target position in the initial frame as a template. This method, based on a visible light target tracking algorithm, introduces infrared modal data, increasing the algorithm's robustness in dark scenes. It can help drones track targets in low-light and high-brightness scenarios, effectively expanding the application scenarios of drone target tracking algorithms.

[0084] In one embodiment, step 100 includes: acquiring a same pair of visible light-infrared dual-modal image data of the target; performing grayscale processing and augmentation processing on the same pair of visible light-infrared dual-modal image data in sequence; and obtaining a visible light-infrared image pair sequence.

[0085] In one embodiment, step 104 includes the following specific steps:

[0086] Step 200: Input the template and search area of the visible light-infrared image pair into a feature extraction network to obtain visible light image features and infrared image features;

[0087] Step 202: Input the visible light image features and the infrared image features into a dual-feature aggregation module to obtain an aggregated template and a search area feature map;

[0088] Step 204: Input the aggregated template and the search area feature map into the detection module to obtain the predicted bounding box of the target;

[0089] Step 206: Perform reverse training on the bimodal target tracking model based on the predicted bounding box and the true position to obtain a trained bimodal target tracking model.

[0090] In one embodiment, the feature extraction network is composed of two independent feature extraction modules with the same structure. The feature extraction module includes the first three layers of the Swin-transformer network. The structure of the feature extraction module is as follows: Figure 2 As shown; step 200 includes: patch segmenting the target and search area of the visible light image and inputting them into the first feature extraction module of the feature extraction network to perform an attention operation on the small patch block to obtain the template feature and search area feature of the visible light image; patch segmenting the target and search area of the infrared image and inputting them into the second feature extraction module of the feature extraction network to perform an attention operation on the small patch block to obtain the template feature and search area feature of the infrared image.

[0091] Specifically, we built a target tracking network based entirely on Transformer, using a cascade-based encoder-decoder to reduce the model's graphics memory usage and inference speed.

[0092] In one embodiment, Figure 3 As shown, the dual-feature aggregation module includes a spatial aggregation module and a channel aggregation module. The spatial aggregation module includes a visible light branch and an infrared branch. The visible light branch includes a multi-head attention module and a normalization processing layer. Step 202 includes: inputting the visible light image features and the infrared image features into the spatial aggregation module of the dual-feature aggregation module to obtain calibrated visible light and infrared image features as follows:

[0093] SA RGB =F I +(MCA(F I ,Concat s (F RGB ,F I )))

[0094] SA I =F RGB +(MCA(F RGB ,Concat s (F I ,F RGB )))

[0095] Among them, F RGB and F I They are the visible light image features and infrared image features extracted by the feature extraction network, MCA(.) represents a multi-head mutual attention module, Concat s (.) indicates splicing in the spatial dimension.

[0096] The calibrated visible light and infrared image features are input into the channel aggregation module, and the aggregated template and search area feature map are obtained as follows:

[0097] weight RGB =Softmax(Γ RGB (Concat c (SA RGB ,SA I )))

[0098] weight I =Softmax(Γ I (Concat c (SA RGB ,SA I )))

[0099] Feature=weight RGB ×SA RGB +weught I ×SA I

[0100] Among them, SA RGB and SA I are the outputs of the spatial aggregation module, Γ RGB and Γ I The pooling layer is used to aggregate channel information, Concat c (.) indicates concatenation in the channel dimension, and Feature is the output of the dual-feature aggregation module.

[0101] Specifically, the dual-feature aggregation network first uses single modal information to strengthen the other modal information in space, and finally aggregates them on the channel to obtain better visible light-infrared aggregation features.

[0102] In one embodiment, Figure 4 As shown, the detection module includes an image correlation operation module and a detection head. The image correlation operation module includes a cascade-based encoder and decoder in SwinTrack. The encoder consists of a multi-head self-attention module and a feedforward network. A residual structure is used in the multi-head self-attention module and the feedforward network module. The structure of the decoder is obtained by replacing the multi-head self-attention module in the encoder with a multi-head mutual attention module. Step 204 includes: splicing the aggregated template and the search area feature map and inputting them into the encoder of the image correlation operation module of the detection module to obtain the separated and spliced features. The template and search area are:

[0103] f d =DeConcat(F FFN ), d=x,z

[0104] F MSA =Concat(f z ,f x )+MSA(LN(Concat(f z ,f x )))

[0105] F FFN =MLP(LN(F MSA ))+F MSA

[0106] Among them, f x and f z They are respectively the template feature and the search area feature after dual feature aggregation, F MSA is the output of the residual block composed of multiple self-attention modules, F FFN is the output of the residual block composed of the feedforward network, MLP is the feedforward network, MSA is the multi-head self-attention module, Concat is the splicing operation, and LN is the layer normalization.

[0107] The template and search area obtained by separating and splicing the features are input into the decoder of the image correlation operation module of the detection module, and the decoded output is:

[0108] F=MLP(LN(F MCA ))+F MCA

[0109] F MCA =f x +MCA(LN(f x ),LN(Concat(f x ,f z )))

[0110] Where F is the output of the decoder, F MCA It is the output of the residual block composed of multiple mutual attention modules.

[0111] The decoded output is fed into the detection head of the detection module to obtain the predicted bounding box of the object.

[0112] In one embodiment, the detection head includes a classification head and a regression head, and the classification head is a multi-layer perception network.

[0113] In one embodiment, the feedforward network includes a two-layer multilayer perceptron.

[0114] In one embodiment, step 106 includes: intercepting the initial frame target position of the infrared-visible light multi-frame video pair of the target to be tracked as a template, extracting the visible light-infrared features of the template, and aggregating the visible light-infrared branch features to obtain a template map; in the video frame after the initial frame, extracting the search area features, aggregating the visible light-infrared features of the search area, and correlating them with the template image to obtain a feature map with the largest response, and sending the feature map to the detection head to classify the target; using a Hamming window to add prior information of the target position in the image, and penalizing the sudden change of the target position; using the obtained target position as the center position of the new search area for cropping to obtain a new search area, and repeating the above process until the preset conditions are met and the target position is determined in the search area.

[0115] In one embodiment, Figure 5 As shown, a flowchart of a target tracking method for a visible light-infrared dual-modal image sequence of a UAV is provided, which specifically includes the following steps:

[0116] In the first step, the visible light-infrared image is preprocessed and the data is augmented to enhance the generalization of the algorithm.

[0117] 1.1 Read the same pair of visible light and infrared dual-modal image data;

[0118] 1.2 Perform grayscale changes with a probability of 5% on image pairs to enhance the generalization of the algorithm for various scenarios.

[0119] 1.3 Use the mean [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225] obtained from the ImageNet dataset to normalize the pixels of the visible-infrared image.

[0120] 1.4 Perform image augmentation operations on the image pairs with a brightness jitter of 3 and a scale jitter of 0.5.

[0121] In the second step, the first frame of the image sequence is annotated, and the template (Template) and search area (Search) data used for image training are extracted to complete the algorithm initialization.

[0122] 2.1 The template is cropped with the target location as the center, with an area multiple of 2 (target annotation box + background padding), and resampled to obtain a template input of size 112×112×6;

[0123] 2.2 The search area is cropped with the target position as the center, with an area multiplier of 5, and resampled to obtain a search area input of size 224×224×6;

[0124] 2.3 Use the cropped template and search area as network input and start network training.

[0125] The third step is to train the network and obtain the network weight value.

[0126] 3.1 Here we use Swin-Transformer as the feature extraction network. Compared with the traditional ResNet network, it has more powerful feature extraction capabilities. Its network structure is as follows Figure 2 As shown in the figure, to speed up computation, the image is segmented into patches and fed into the network. Here, we use the first three layers of the Swin-Transformer network with a stride size of 16. Our tracker follows the Siamese network architecture and requires an image template and a search region. The bimodal image pair is first segmented into small patches and fed into the network. Attention is then applied to these patches, significantly reducing computational complexity. Finally, template feature patches and search region feature patches are generated.

[0127] For a single modality, the template and search region can be extracted using only a single parameter-sharing feature extraction network. However, since our task is dual-modality tracking involving visible light and infrared, we need to extract both visible and infrared features separately. This requires two independent feature extraction modules.

[0128] 3.2 Perform feature aggregation on the visible light and infrared feature maps respectively to obtain the aggregated template and search area feature map.

[0129] Here we use Figure 3 The dual-feature aggregation module shown in the figure performs feature fusion, and its structure is shown in the figure. Inspired by the work of CBAM, SENet, and other researchers, we use attention mechanisms in both the spatial and channel dimensions to enhance useful feature information during the fusion of visible and infrared information. The dual-feature aggregation module mainly consists of two modules: the spatial aggregation module and the channel aggregation module. Unlike other feature fusion methods, ours simultaneously enhances bimodal information during the fusion process, which is very useful in scenarios where a single modality is limited.

[0130] Spatial aggregation focuses mainly on the spatial features of the image. Inspired by the Transformer structure, the attention mechanism can search for areas of interest at any position in the image. If the input template is an infrared image and the search area is a visible light image, the attention mechanism can use the information of the infrared image to search for the area in the visible light image that is most similar to the infrared image. This is equivalent to calibrating the visible light image information using the infrared image information, highlighting the parts of the visible light image that are similar to the infrared image. In this way, the image information of another modality can be introduced while retaining the original modality image information to the greatest extent. Similarly, by exchanging the inputs of visible light and infrared images, the visible light information can be used to calibrate the infrared image, thus obtaining the calibrated infrared and visible light image features. This is why the module is called dual features. This process is described by the formula:

[0131] SA RGB =F I +(MCA(F I ,Concat s (F RGB ,F I )))

[0132] SA I =F RGB +(MCA(F RGB ,Concat s (F I ,F RGB )))

[0133] Among them, F RGB and F I They are the visible light and infrared image features extracted by the feature extraction network, MCA(.) represents a multi-head mutual attention module, Concat s (.) indicates splicing in the spatial dimension.

[0134] Channel aggregation focuses on the features of image channels. After spatial aggregation, we obtain calibrated visible and infrared image features. If directly fed into the subsequent encoder, the channel dimension would be twice that of the original features, seriously affecting the encoder's operating efficiency. This is why channel aggregation is necessary to select the most important channel features in visible and infrared images, thereby reducing the channel dimension and further fusing the visible and infrared image information at the channel level. This is expressed as:

[0135] weight RGB =Softmax(Γ RGB (Concat c (SA RGB ,SA I )))

[0136] weight I =Softmax(Γ I (Concat c (SA RGB ,SA I )))

[0137] Feature=weight RGB ×SA RGB +weught I ×SA I

[0138] Among them, SA RGB and SA I are the outputs of the spatial aggregation module, Γ RGB and Γ I The pooling layer is used to aggregate channel information, Concat c (.) indicates concatenation in the channel dimension, and Feature is the output of the dual-feature aggregation module.

[0139] 3.3 Correlate with the template image in the search area, then feed it into a multi-layer perception network to regress the position and bounding box of the target in the current frame.

[0140] After the dual feature aggregation network, we get the fused template and search area features. In the correlation process, the encoder and decoder are used to perform correlation operations on the template and search area. The structure diagram is as follows: Figure 4 As shown in the figure, the encoder encodes the template and search area to obtain a more compact feature representation, and the decoder decodes the encoded features into the original feature space to obtain relevant results. Finally, the results are fed into the detection head to regress the target bounding box.

[0141] Here we use the cascade-based encoder and decoder in SwinTrack to reduce the size of the model parameters. After testing, the use of the cascade-based method can significantly reduce the model's memory usage, which makes our algorithm easier to deploy on some drone edge computing devices.

[0142] The encoder consists of a multi-head self-attention module and a feedforward network. The feedforward network contains a two-layer multi-layer perceptron. GELU is used as the activation function. Layer normalization is used in the encoder to avoid overfitting. Residual structures are used in the multi-head self-attention module and the feedforward network module to facilitate gradient backpropagation. The encoder process can be expressed as:

[0143] Combine the template and search area:

[0144] F=Concat(fz ,f x )

[0145] The concatenated features are enhanced using the attention mechanism:

[0146] F MSA =F+MSA(LN(F))

[0147] F FFN =MLP(LN(F MSA ))+F MSA

[0148] Separate the spliced features to obtain the template and search area:

[0149] f d =DeConcat(F FFN ), d=x,z

[0150] Among them, f x and f z They are respectively the template feature and the search area feature after dual feature aggregation, and MSA is the multi-head self-attention module.

[0151] The decoder and encoder structures are roughly the same, except that the multi-head mutual attention module is used instead of the multi-head self-attention module. Other parts remain the same, so the decoder process can be expressed as:

[0152] F MCA =f x +MCA(LN(f x ),LN(Concat(f x ,f z )))

[0153] F=MLP(LN(F MCA ))+F MCA

[0154] Among them, f x and f z They are the template feature and search area feature after dual feature aggregation, MCA is the multi-head mutual attention module, and F is the output of the decoder, which is sent to the detection head to generate the classification response map and the boundary response map.

[0155] 3.5 Compare with the true position and bounding box, calculate the error, backpropagate the error to the network, and update the network weight value.

[0156] In the fourth step, the trained network is used for inference, the target position in the initial frame is used as a template, and the target tracking network is used to determine the target position in the search area.

[0157] 4.1 Intercept the target position of the initial frame of the infrared-visible light multi-frame video pair of the target to be tracked as a template, extract the visible light-infrared features of the template, and aggregate the visible light-infrared branch features to obtain the template graph;

[0158] 4.2 In the subsequent video frames, extract the search area features, aggregate the visible light and infrared features of the search area, and correlate them with the template image to obtain the feature map with the largest response. The feature map is sent to the head detector to classify the target;

[0159] 4.3 The Hamming window is introduced to add the prior information of the target position in the image and penalize the sudden change of the target position.

[0160] Here, a Hamming window is used to introduce prior information about the target position, penalize sudden changes in the target position, and prevent sudden changes in the target position. The expression is:

[0161] cls=(1-γ)×r cls +γ×h

[0162] Here r cls Represents the target classification response map, γ is the penalty coefficient, which is generally taken as 0.49, and h is the value of r cls Hamming windows of the same size.

[0163] 4.4 The obtained target position is used as the center position of the new search area for cropping to obtain a new search area. The above process is repeated to continuously infer the image sequence.

[0164] The training process uses multi-frame time-series videos. When tracking the target, the input is an infrared-visible light multi-frame video pair.

[0165] It should be understood that although Figure 1 and Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 and Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0166] In a verification example, the method of the present application was tested on a measured data set collected by the Zenmuse H20T optoelectronic pod experimental equipment carried by the M300 UAV. The robustness and accuracy of the algorithm compared with similar algorithms were analyzed through experimental comparison. The tracking effect of our algorithm was then demonstrated, and the effects of each module in the algorithm were displayed through heat maps.

[0167] (1) Description of measured data

[0168] The data set used in the experiment of this embodiment is collected by DJI’s industrial-grade drone M300 equipped with a Zenmuse H20T optoelectronic pod. It contains a total of 500 visible-infrared image sequences, of which 250 image sequences are used as training sets and the remaining image sequences are used as test sequences. The test sequences contain image sequence pairs collected by drones in various scenarios, including multiple common drone mission scenarios such as lighting changes, occlusion, shadows, and rapid drone movement. The measured data in different scenarios are shown as follows: Figure 6 As shown, Figure 6 (a) and (b) are visible light images and red images in low-light scenes, respectively. Figure 6 (c) and (d) are the visible light image and red image under strong light scene respectively. Figure 6 (e) and (f) are the visible light image and red image under the scene of drastic lighting changes, respectively. Figure 6 (g) and (h) are the visible light image and red image in the tree occlusion scene respectively. Figure 6 (i) and (j) are the visible light image and red image in the cloud-covered scene, respectively. Figure 6 (k) and (l) are visible light images and red images in scenes with appearance changes, respectively.

[0169] (2) Comparative experiments and analysis

[0170] The test used 175 dual-modal video sequences captured by a DJI M300 drone equipped with a Zenmuse H20T camera. The experimental environment was Linux 20.04, with an Intel Xeon E5-2678 v3 @ 2.50 GHz processor, 78.5 GB of RAM, and two Nvidia TiTAN GPUs with 24 GB of graphics. The programming environment was Python 3.7 and PyTorch 1.11.

[0171] Using the maximum success rate (MSR) and maximum precision rate (MPR) as evaluation indicators, a comparative experiment was conducted between the visible light-infrared dual-modal target tracking algorithm and the mainstream non-temporal visible light-infrared dual-modal algorithms (DAFNet, ADRNet, FSRPN). The experimental results are shown in the figure:

[0172] Table 1 shows the statistics of the comparative experiments of the aforementioned algorithms. As can be seen from Table 1, our algorithm outperforms both ADRNet and DAFNet in terms of results. Because our algorithm uses a splicing-based Transformer structure as the encoder and decoder, our algorithm runs slightly faster than the FSRPN algorithm. Overall, our algorithm achieves good results in data captured by the Zenmuse H20T visible light and infrared dual-modal camera equipped with the M300.

[0173] Table 1 Comparison results of algorithms

[0174]

[0175] (3) Ablation experiment and analysis

[0176] Ablation experiments were conducted on each module of the method of this application to verify the effectiveness of the algorithm. The experimental environment was the Linux 20.04 platform, the processor was an Inter Xeon E5-2678 v3 @ 2.50 GHz, the memory was 78.5 GB, the graphics card was two Nvidia TiTAN GPUs 24 GB, and the programming environment was Python 3.7 and PyTorch 1.11. The ablation experiment results of the algorithm are shown in Table 2.

[0177] Table 2 Ablation experiments of the algorithms

[0178]

[0179] We conducted ablation experiments and comparative analysis on four different algorithms: simple bimodal concatenation, using a ResNet network instead of the Swin-Trnsformer as the backbone network, using only channel aggregation to fuse bimodal feature information, and our algorithm. The results in Table 2 show that simple bimodal concatenation significantly degrades performance, with the maximum success rate and maximum accuracy dropping by 10.3% and 13.2%, respectively, compared to using the dual-feature aggregation module.

[0180] Using the ResNet network instead of the Swin-Transformer as the backbone network, its performance dropped by 7.7% and 8.3% compared to the splicing of dual-modal features. First, Swin-Transformer has stronger feature extraction capabilities than ResNet. At the same time, since the step size of the ResNet network is 8, this algorithm is suitable for the Transformer structure. If ResNet is used as the backbone network, a smaller template and search area size must be used, which is also a factor affecting the performance of the algorithm. The difference between using only channel aggregation and dual-feature aggregation using heat maps is shown in the figure below. Figure 7As shown, (a) is direct channel aggregation, and (b) is dual-feature aggregation.

[0181] Finally, we only use channel aggregation to aggregate visible light and infrared information. The information is slightly lower than that of the stitching algorithm. This is because channel aggregation only weights the channels. Due to the different imaging mechanisms of visible light and infrared images, their channel responses vary greatly in different environments. Directly using channel aggregation may focus on the heterogeneous information of visible light and infrared images while ignoring their homogeneous information, which affects the aggregation effect of dual-modal features. Here we use heat maps for visual analysis, such as Figure 6 shown.

[0182] Figure 7 (a) is to directly perform channel aggregation. Figure 7 Middle (b) uses dual-feature aggregation. From the effect analysis, the dual-feature aggregation model can enhance the visible light and infrared image information respectively, showing more homogeneous information, and the network's attention is more likely to focus on the target itself.

[0183] The dual-feature aggregation module used in the method of this application achieved the best results, which proves that the dual-feature aggregation module we proposed can effectively aggregate visible light-infrared dual-modal information and thus complete the tracking of the target.

[0184] (4) Network heat map analysis

[0185] In order to more clearly see the information processing capabilities of specific modules of the algorithm, we use heat maps to show the heat map visualization effects of the input of each module of the algorithm for image pairs, such as Figure 8 As shown, (a) is the Bike data sequence, (b) is the pedestrian-211 data sequence, and (c) is the Tricy-006 data sequence.

[0186] From the heat map, we can see that the backbone network extracts features from visible light and infrared images respectively. Since visible light and infrared images have certain differences, the features extracted by the network are also different. The extracted features are sent to the dual feature aggregation network, which will integrate the visible light and infrared image information to obtain a fused feature map. Figure 8 As shown in (b) and (c) in the Pedestrian-211 and Tricy-006 data sequences, it is clearer that the aggregated feature map combines all features from the image pair. This fused feature map then passes through the encoder and decoder modules, focusing the network's attention on the target. Finally, the image with the largest target response is fed into the detection head for classification and regression.

[0187] (5) Actual tracking effect of the algorithm and failure cases

[0188] The algorithm in this application can achieve relatively good tracking results in a variety of scenarios, including daytime, nighttime, and occluded scenes. This is especially true in dark scenes, where the target is difficult to detect using only visible light. However, in dual-modal scenes, the algorithm in this application still achieves a good tracking effect. Furthermore, the algorithm in this application can maintain a good target tracking effect even in small and densely packed target scenes, which is another advantage of our algorithm.

[0189] At the same time, the algorithm will fail in some scenarios. Here are some examples of algorithm failure, such as Figure 9 As shown in the figure, (a) is similar in appearance, (b) is shape change, and (c) is image noise. This helps us pay attention to the usage scenario when using the algorithm, and also facilitates subsequent improvement and enhancement of the algorithm.

[0190] Figure 9 In (a), objects with similar appearances interfere with each other in both visible and infrared image pairs. For the Siamese network, its core is matching the template and the search region, and it lacks the ability to distinguish similar objects. Although we use a Hanning window to introduce prior information about the target's position and penalize sudden changes in the target's position, tracking similar objects at close range is very likely to fail. Figure 9 (b) in the middle is shape change. Our work is non-temporal. When the appearance of the target in the search area is very different from the appearance in the template, the tracker cannot track it. Therefore, temporal information is very important in some long sequence tracking processes. Figure 9 (c) is image noise. Due to process problems, thermal aggregation will occur when the infrared camera collects infrared data, which seriously affects the quality of the infrared image. When the single-modal noise is too large, the tracker will introduce noise, affecting the tracking effect.

[0191] In one embodiment, Figure 10 As shown, a dual-modal target tracking device based on infrared and visible light images is provided, comprising: a dual-modal image preprocessing module, a template and search area determination module, a dual-modal target tracking model training module and a target tracking module, wherein:

[0192] A dual-modal image preprocessing module is used to preprocess the acquired visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence;

[0193] a template and search area determination module, configured to crop the first frame of the visible light-infrared image pair sequence to obtain a template and a search area for the first frame of the visible light-infrared image pair;

[0194] The dual-modal target tracking model training module is used to take the template and search area of the visible light-infrared image pair as input to train the dual-modal target tracking model to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module and a detection module; the feature extraction network is used to extract the visible light image features and infrared image features of the template and search area of the visible light-infrared image pair respectively; the dual-feature aggregation module is used to perform feature aggregation on the visible light image features and the infrared image features using the spatial aggregation module and the channel aggregation module to obtain the aggregated template and the search area feature map; the detection module is used to perform correlation processing on the aggregated template and the search area feature map, send the obtained processing results to the detection head, and regress the predicted bounding box of the target;

[0195] The target tracking module is used to input the infrared-visible light multi-frame video pair of the target to be tracked into the trained dual-modal target tracking model, and determine the target position in the search area using the initial frame target position as a template.

[0196] In one embodiment, the dual-modal image preprocessing module is further used to obtain the same pair of visible light-infrared dual-modal image data of the target; perform grayscale processing and augmentation processing on the same pair of visible light-infrared dual-modal image data in sequence; and obtain a visible light-infrared image pair sequence.

[0197] In one embodiment, the dual-modal target tracking model training module is further used to input the template and search area of the visible light-infrared image pair into the feature extraction network to obtain visible light image features and infrared image features; input the visible light image features and infrared image features into the dual-feature aggregation module to obtain the aggregated template and search area feature map; input the aggregated template and search area feature map into the detection module to obtain the predicted bounding box of the target; and reversely train the dual-modal target tracking model based on the predicted bounding box and the true position to obtain a trained dual-modal target tracking model.

[0198] In one embodiment, the feature extraction network is composed of two independent feature extraction modules with the same structure, and the feature extraction modules include the first three layers of the Swin-transformer network; the dual-modal target tracking model training module is also used to perform patch segmentation on the target and search area of the visible light image and input them into the first feature extraction module of the feature extraction network to perform an attention operation on the small patch blocks to obtain the template features and search area features of the visible light image; the target and search area of the infrared image are patch segmented and input into the second feature extraction module of the feature extraction network to perform an attention operation on the small patch blocks to obtain the template features and search area features of the infrared image.

[0199] In one embodiment, the dual-feature aggregation module includes a spatial aggregation module and a channel aggregation module. The spatial aggregation module includes a visible light branch and an infrared branch. The visible light branch includes a multi-head attention module and a normalization processing layer. The dual-modal target tracking model training module is further used to input the visible light image features and the infrared image features into the spatial aggregation module of the dual-feature aggregation module, and obtain the calibrated visible light and infrared image features as follows:

[0200] SA RGB =F I +(MCA(F I ,Concat s (F RGB ,F I )))

[0201] SA I =F RGB +(MCA(F RGB ,Concat s (F I ,F RGB )))

[0202] Among them, F RGB and F I They are the visible light image features and infrared image features extracted by the feature extraction network, MCA(.) represents a multi-head mutual attention module, Concat s (.) indicates splicing in the spatial dimension.

[0203] The calibrated visible light and infrared image features are input into the channel aggregation module, and the aggregated template and search area feature map are obtained as follows:

[0204] weight RGB =Softmax(Γ RGB (Concat c (SA RGB ,SA I )))

[0205] weight I =Softmax(Γ I (Concat c (SA RGB ,SA I )))

[0206] Feature=weight RGB ×SA RGB +weught I ×SA I

[0207] Among them, SARGB and SA I are the outputs of the spatial aggregation module, Γ RGB and Γ I The pooling layer is used to aggregate channel information, Concat c (.) indicates concatenation in the channel dimension, and Feature is the output of the dual-feature aggregation module.

[0208] In one embodiment, the detection module includes an image correlation operation module and a detection head, the image correlation operation module includes a cascade-based encoder and decoder in SwinTrack; the encoder consists of a multi-head self-attention module and a feedforward network, and a residual structure is used in the multi-head self-attention module and the feedforward network module; the decoder structure is obtained by replacing the multi-head self-attention module in the encoder with a multi-head mutual attention module; the dual-modal target tracking model training module is further used to splice the aggregated template and search area feature map and input them into the encoder of the image correlation operation module of the detection module, and the separated and spliced features are obtained. The template and search area are:

[0209] f d =DeConcat(F FFN ), d=x,z

[0210] F MSA =Concat(f z ,f x )+MSA(LN(Concat(f z ,f x )))

[0211] F FFN =MLP(LN(F MSA ))+F MSA

[0212] Among them, f x and f z They are respectively the template feature and the search area feature after dual feature aggregation, F MSA is the output of the residual block composed of multiple self-attention modules, F FFN is the output of the residual block composed of the feedforward network, MLP is the feedforward network, MSA(.) is the multi-head self-attention module, Concat is the splicing operation, and LN is the layer normalization;

[0213] The template and search area obtained by separating and splicing the features are input into the decoder of the image correlation operation module of the detection module, and the decoded output is:

[0214] F=MLP(LN(F MCA ))+F MCA

[0215] F MCA =f x +MCA(LN(f x ),LN(Concat(f x ,f z )))

[0216] Where F is the output of the decoder, F MCA It is the output of the residual block composed of multiple mutual attention modules.

[0217] The decoded output is fed into the detection head of the detection module to obtain the predicted bounding box of the object.

[0218] In one embodiment, the detection head includes a multi-layer perception network.

[0219] In one embodiment, the feedforward network includes a two-layer multilayer perceptron.

[0220] In one embodiment, the target tracking module is further used to intercept the initial frame target position of the infrared-visible light multi-frame video pair of the target to be tracked as a template, extract the visible light-infrared features of the template, and aggregate the visible light-infrared branch features to obtain a template map; in the video frame after the initial frame, extract the search area features, aggregate the visible light-infrared features of the search area, and correlate them with the template image to obtain a feature map with the largest response, and send the feature map to the detection head to classify the target; use a Hamming window to add prior information of the target position in the image, and penalize sudden changes in the target position; use the obtained target position as the center position of the new search area for cropping to obtain a new search area, and repeat the above process until the preset conditions are met and the target position is determined in the search area.

[0221] Regarding the specific definition of the dual-modal target tracking device based on infrared and visible light images, please refer to the definition of the dual-modal target tracking method based on infrared and visible light images above, and no further details will be given here. Each module in the above-mentioned dual-modal target tracking device based on infrared and visible light images can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0222] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0223] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A dual-modal target tracking method based on infrared and visible light images, characterized in that: The method comprises: Preprocessing the acquired same pair of visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence; Cropping the first frame of the visible light-infrared image pair sequence to obtain a template and a search area for the first frame of the visible light-infrared image pair; The template and search area of the visible light-infrared image pair are used as input to train a dual-modal target tracking model to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module and a detection module; the feature extraction network is used to extract visible light image features and infrared image features of the template and search area of the visible light-infrared image pair respectively; the dual-feature aggregation module is used to perform feature aggregation on the visible light image features and the infrared image features using a spatial aggregation module and a channel aggregation module to obtain an aggregated template and a search area feature map; the detection module is used to perform correlation processing on the aggregated template and the search area feature map, and then send the obtained processing results to a detection head to regress a predicted bounding box of the target; Input the infrared-visible multi-frame video pair of the target to be tracked into the trained dual-modal target tracking model, and use the initial frame target position as a template to determine the target position in the search area; The template and search area of the visible light-infrared image pair are used as input to train the dual-modal target tracking model, and the trained dual-modal target tracking model is obtained, including: The template and search area of the visible light-infrared image pair are input into the feature extraction network to obtain visible light image features and infrared image features; Inputting the visible light image features and the infrared image features into the dual-feature aggregation module to obtain an aggregated template and a search area feature map; the dual-feature aggregation module includes a spatial aggregation module and a channel aggregation module, the spatial aggregation module includes a visible light branch and an infrared branch; the visible light branch includes a multi-head attention module and a normalization processing layer; The aggregated template and search area feature map are input into a detection module to obtain a predicted bounding box of the target; the detection module includes an image correlation operation module and a detection head, and the image correlation operation module includes a cascade-based encoder and decoder in SwinTrack; the encoder consists of a multi-head self-attention module and a feedforward network, and a residual structure is used in the multi-head self-attention module and the feedforward network module; the decoder structure is obtained by replacing the multi-head self-attention module in the encoder with a multi-head mutual attention module; The dual-modal target tracking model is reversely trained according to the predicted bounding box and the true position to obtain a trained dual-modal target tracking model.

2. The method according to claim 1, characterized in that The obtained same pair of visible light-infrared dual-modal image data is preprocessed to obtain a visible light-infrared image pair sequence, including: Acquire the same pair of visible light-infrared dual-modal image data of the target; Grayscale processing and augmentation processing are sequentially performed on the same pair of visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence.

3. The method according to claim 1, characterized in that The feature extraction network is composed of two independent feature extraction modules with the same structure, and the feature extraction module includes the first three layers of the Swin-transformer network; The template and search area of the visible light-infrared image pair are input into the feature extraction network to obtain visible light image features and infrared image features, including: The target and search area of the visible light image are segmented into patches and then input into the first feature extraction module of the feature extraction network, where attention operation is performed on the small patches to obtain the template features and search area features of the visible light image. The target and search area of the infrared image are segmented into patches and then input into the second feature extraction module of the feature extraction network to perform attention operation on the small patch blocks to obtain the template features and search area features of the infrared image.

4. The method according to claim 1, wherein Inputting the visible light image features and the infrared image features into the dual-feature aggregation module to obtain an aggregated template and a search area feature map, including: The visible light image features and the infrared image features are input into the spatial aggregation module of the dual-feature aggregation module to obtain calibrated visible light and infrared image features: IN RGB =F I +(MCA(F I ,Concat s (F RGB ,F I ))) IN I =F RGB +(MCA(F RGB ,Concat s (F I ,F RGB ))) Among them, SA RGB and SA I are the outputs of the spatial aggregation module, F RGB and F I They are the visible light image features and infrared image features extracted by the feature extraction network, MCA(.) represents a multi-head mutual attention module, Concat s (.) indicates splicing in the spatial dimension; The calibrated visible light and infrared image features are input into the channel aggregation module, and the aggregated template and search area feature map are obtained as follows: Feature=weight RGB ×SA RGB +weught I ×SA I weight RGB =Softmax(Γ RGB (Concat c (SA RGB ,SA I ))) weight I =Softmax(Γ I (Concat c (SA RGB ,SA I ))) Among them, Feature is the output of the dual feature aggregation module, Γ RGB and Γ I It is the channel information that the pooling layer uses to aggregate, Concat c (.) indicates concatenation in the channel dimension.

5. The method according to claim 1, wherein The aggregated template and search area feature map are input into the detection module to obtain the predicted bounding box of the target, including: The aggregated template and search area feature map are spliced and input into the encoder of the image correlation operation module of the detection module to obtain the template and search area after separation and splicing features: f d =DeConcat(F FFN ),d=x,z F MSA =Concat(f z ,f x )+MSA(LN(Concat(f z ,f x ))) F FFN =MLP(LN(F MSA ))+F MSA Among them, f x and f z They are respectively the template feature and the search area feature after dual feature aggregation, F MSA is the output of the residual block composed of multiple self-attention modules, F FFN is the output of the residual block composed of the feedforward network, MLP is the feedforward network, MSA is the multi-head self-attention module, Concat is the splicing operation, and LN is the layer normalization; The template and search area obtained by separating and splicing the features are input into the decoder of the image correlation operation module of the detection module, and the decoded output is: F=MLP(LN(F MCA ))+F MCA F MCA =f x +MCA(LN(f x ),LN(Concat(f x ,f z ))) Where F is the output of the decoder, F MCA is the output of the residual block composed of multiple mutual attention modules; The decoded output is input into the detection head of the detection module to obtain a predicted bounding box of the target.

6. The method according to claim 5, characterized in that The detection head includes a classification head and a regression head, and the classification head is a multi-layer perception network.

7. The method according to claim 5, characterized in that The feedforward network includes a two-layer multilayer perceptron.

8. The method according to claim 1, characterized in that Input the infrared-visible multi-frame video pair of the target to be tracked into the trained dual-modal target tracking model. Using the target position in the initial frame as a template, the target position is determined in the search area, including: The target position of the initial frame of the infrared-visible light multi-frame video pair of the target to be tracked is intercepted as a template, the visible light-infrared features of the template are extracted, and the visible light-infrared branch features are aggregated to obtain the template graph; In the video frames after the initial frame, the search area features are extracted, the visible light-infrared features of the search area are aggregated, and correlated with the template image to obtain the feature map with the largest response. The feature map is then sent to the detection head to classify the target. A Hamming window is used to add prior information about the target position in the image and penalize sudden changes in the target position. The obtained target position is used as the center position of the new search area for cropping to obtain a new search area. The above process is repeated until the preset conditions are met and the target position is determined in the search area.

9. A dual-modal target tracking device based on infrared and visible light images, characterized in that: The device comprises: A dual-modal image preprocessing module is used to preprocess the acquired visible light-infrared dual-modal image data to obtain a visible light-infrared image pair sequence; a template and search area determination module, configured to crop a first frame of the visible light-infrared image pair sequence to obtain a template and a search area for the first frame of the visible light-infrared image pair; A dual-modal target tracking model training module is used to take the template and search area of the visible light-infrared image pair as input to train the dual-modal target tracking model to obtain a trained dual-modal target tracking model; the dual-modal target tracking model includes a feature extraction network, a dual-feature aggregation module and a detection module; the feature extraction network is used to extract visible light image features and infrared image features of the template and search area of the visible light-infrared image pair respectively; the dual-feature aggregation module is used to perform feature aggregation on the visible light image features and the infrared image features using a spatial aggregation module and a channel aggregation module to obtain an aggregated template and a search area feature map; the detection module is used to perform correlation processing on the aggregated template and the search area feature map, send the obtained processing results to the detection head, and regress the predicted bounding box of the target; The target tracking module is used to input the infrared-visible light multi-frame video pair of the target to be tracked into the trained dual-modal target tracking model, and determine the target position in the search area using the initial frame target position as a template; Among them, the dual-modal target tracking model training module is also used to input the template and search area of the visible light-infrared image pair into the feature extraction network to obtain visible light image features and infrared image features; input the visible light image features and the infrared image features into the dual-feature aggregation module to obtain the aggregated template and search area feature map; input the aggregated template and search area feature map into the detection module to obtain the predicted bounding box of the target; reversely train the dual-modal target tracking model according to the predicted bounding box and the true position to obtain a trained dual-modal target tracking model; the dual-feature aggregation module The block includes a spatial aggregation module and a channel aggregation module, and the spatial aggregation module includes a visible light branch and an infrared branch; the visible light branch includes a multi-head attention module and a normalization processing layer; the detection module includes an image correlation operation module and a detection head, and the image correlation operation module includes a cascade-based encoder and decoder in SwinTrack; the encoder consists of a multi-head self-attention module and a feedforward network, and a residual structure is used in the multi-head self-attention module and the feedforward network module; the structure of the decoder is obtained by replacing the multi-head self-attention module in the encoder with a multi-head mutual attention module.

Images