Multi-target tracking method and system based on visible light thermal infrared fusion

Abstract

This invention provides a multi-target tracking method based on visible light and thermal infrared fusion, belonging to the field of deep learning. It obtains visible light and thermal infrared images of the current and previous frames; inputs these images and the thermal image from the previous frame into a convolutional neural network to obtain visible light and thermal infrared features of the current and previous frames, as well as features from the previous frame's thermal image; using the previous frame's thermal image features as a positional condition, it performs temporal feature fusion to obtain visible light and thermal infrared temporal features; adds these features to obtain initial multimodal features; interacts with these initial multimodal features to obtain enhanced modality-specific features and multimodal features; merges these features and inputs them into a feedforward neural network to obtain refined multimodal features; inputs these features into a tracking network to obtain tracking results, providing a multi-target tracking system based on visible light and thermal infrared fusion; and improves the robustness of multi-target tracking in complex environments by fusing temporal and complementary information from visible light and thermal infrared modes.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more particularly to multimodal multi-target tracking. Background Technology

[0002] Multi-object tracking is a fundamental task in computer vision, aiming to analyze video to identify and track objects belonging to one or more categories, such as people, vehicles, animals, and inanimate objects, without any prior knowledge of the object's appearance or number. Due to its engineering practicality in real-world scenarios such as video analysis and autonomous driving, it has received increasing attention from the computer vision community. While significant progress has been made in multi-object tracking in recent years, it still faces considerable challenges. Currently, most mainstream trackers are based on visible light images. Due to the limitations of visible light imaging, the tracking performance of these trackers is not particularly stable in complex environments such as low light, smoke, and haze. To address these challenges, integrating visible light and thermal infrared data has become a promising solution. Visible light images can provide rich color and texture information, but the data quality is poor in low-light and hazy environments. Conversely, thermal infrared data exhibits good quality in such environments but lacks color and texture information. Numerous studies have demonstrated that integrating visible light and thermal infrared data can significantly improve the performance of visual tasks such as single-target tracking, semantic segmentation, saliency detection, and target detection. For example, the paper "Quality-aware aggregation network for robust RGBT tracking" (Yabin Zhu, Chenglong Li, Jin Tang, and Bin Luo, IEEE Transactions on Intelligent Vehicles 6, 1 (2020)) significantly improves the performance of single-target tracking by integrating visible light and thermal infrared data. However, this single-target tracking method usually cannot analyze and utilize temporal information to associate multiple targets.

[0003] Currently, most target tracking technologies are based on visible light data, and there are few technologies exploring the fusion of visible light and thermal infrared data for multi-target tracking. Among existing technologies, the Chinese invention patent application CN116993781A, entitled "Multimodal Visual Tracking Method and System Based on Surveillance Scenes," first extracts common features from the visible light and thermal infrared modalities, then extracts independent features from these two modalities, and finally fuses them. However, this simple integration of visible light and thermal infrared data cannot provide effective discriminative features, thus failing to significantly maintain the robustness of multi-target tracking in complex environments. Summary of the Invention

[0004] The technical problem to be solved by this invention is how to improve the robustness of multi-target tracking in complex environments.

[0005] This invention solves the above-mentioned technical problems through the following technical solution: a multi-target tracking method based on visible light and thermal infrared fusion, the method comprising:

[0006] Data is acquired using a visible light camera to obtain visible light images of the current frame and the previous frame; data is also acquired using a thermal infrared camera to obtain thermal infrared images of the current frame and the previous frame.

[0007] The visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, are respectively input into the convolutional neural network for feature extraction to obtain the visible light features and thermal infrared features of the current frame and the previous frame, as well as the heat map features of the previous frame.

[0008] Using the heatmap features of the previous frame as a location condition, the visible light features and thermal infrared features of the current frame and the previous frame are fused temporally to obtain visible light temporal features and thermal infrared temporal features. The visible light temporal features and thermal infrared temporal features are added together to obtain initial multimodal features. Using the initial multimodal features as bridging features, the visible light temporal features and thermal infrared temporal features are interacted with the initial multimodal features to obtain enhanced modality-specific features and enhanced multimodal features. The enhanced modality-specific features and enhanced multimodal features are merged, input into a feedforward neural network, and then normalized to obtain refined multimodal features.

[0009] The refined multimodal features are input into the baseline CenterTrack tracking network to obtain multi-target tracking results.

[0010] This invention fuses visible light and thermal infrared data for multi-target tracking and proposes a progressively fused multimodal multi-target tracking framework, PFTrack. This framework effectively fuses the temporal and complementary information of visible light and thermal infrared modes in a progressive manner, enabling the tracker to achieve robust tracking of multiple targets in complex scenarios.

[0011] Preferably, the visible light camera and the thermal infrared camera are handheld devices, drones, or monitoring platforms.

[0012] Preferably, the convolutional neural network includes 7×7 convolutional layers, batch normalization, and correction linear units.

[0013] Preferably, the process of inputting the visible light image and thermal infrared image of the current frame and the previous frame and the heat map of the previous frame into a convolutional neural network for feature extraction includes: inputting the visible light image and thermal infrared image of the current frame and the previous frame and the heat map of the previous frame into a 7×7 convolutional layer to obtain feature maps with unchanged height and width and 16 channels; dividing the visible light feature map and thermal infrared feature map of the current frame and the previous frame and the heat map of the previous frame into 16×16 image blocks and planarizing them into sequences to obtain the visible light feature sequence and thermal infrared feature sequence of the current frame and the previous frame and the heat map feature sequence of the previous frame.

[0014] Preferably, the process of temporal feature fusion of the visible light features of the current frame and the previous frame is the same as the process of temporal feature fusion of the thermal infrared features of the current frame and the previous frame, including:

[0015] Visible light features of the current frame Visible light features of the previous frame The input is fed into the cross-attention module for fusion, yielding the median value of the fused visible light features, x. v ;

[0016] The median value of the fused visible light features is x v Visible light features of the current frame After addition, normalization is performed. The normalized value is then compared with the heatmap features of the previous frame. Adding them together yields the intermediate value of the fused visible light characteristics.

[0017] The median value of the fused visible light features Input to a feedforward network, output value and the median of the fused visible light features are two values. The sums are then normalized to obtain the final visible light time characteristics.

[0018] Preferably, the process of obtaining refined multimodal features includes:

[0019] visible light time characteristics and thermal infrared time characteristics Adding them together yields the initial multimodal features x. f ;

[0020] With initial multimodal features x f For keys and values, visible light temporal characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. With initial multimodal features x f For bonds and values, thermal infrared time characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. Based on visible light time characteristics For keys and values, the initial multimodal feature x f The query input is fused using a cross-attention module to obtain enhanced multimodal features. Based on thermal infrared time characteristics For keys and values, the initial multimodal feature x f The query input is fused using a cross-attention module to obtain enhanced multimodal features.

[0021] With enhanced modality-specific features Enhanced modality-specific features 2 Enhanced multimodal features Enhanced multimodal features II Construct a multidimensional array to obtain multimodal features Multimodal features The input is fed forward into a network, and the output value is normalized to obtain the final refined multimodal features.

[0022] This invention also provides a multi-target tracking system based on visible light and thermal infrared fusion, the system comprising:

[0023] The data acquisition module is used to acquire data based on a visible light camera to obtain visible light images of the current frame and the previous frame; and to acquire data based on a thermal infrared camera to obtain thermal infrared images of the current frame and the previous frame.

[0024] The feature extraction module is used to input the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, into the convolutional neural network for feature extraction, so as to obtain the visible light feature and thermal infrared feature of the current frame and the previous frame, as well as the heat map feature of the previous frame.

[0025] The progressive fusion module is used to fuse the visible light and thermal infrared features of the current frame and the previous frame respectively, using the heat map features of the previous frame as the location condition, to obtain visible light temporal features and thermal infrared temporal features. The visible light temporal features and thermal infrared temporal features are added together to obtain initial multimodal features. Using the initial multimodal features as bridging features, the visible light temporal features and thermal infrared temporal features are fused with the initial multimodal features respectively to obtain enhanced modality-specific features and enhanced multimodal features. The enhanced modality-specific features and enhanced multimodal features are merged, input into the feedforward neural network, and then normalized to obtain refined multimodal features.

[0026] The data processing module is used to input refined multimodal features into the baseline CenterTrack tracking network to obtain multi-target tracking results.

[0027] Preferably, the process of inputting the visible light image and thermal infrared image of the current frame and the heat map of the previous frame into the convolutional neural network for feature extraction in the feature extraction module includes: inputting the visible light image and thermal infrared image of the current frame and the heat map of the previous frame into a 7×7 convolutional layer to obtain feature maps with unchanged height and width and 16 channels; dividing the visible light feature map and thermal infrared feature map of the current frame and the previous frame into 16×16 image blocks and planarizing them into a sequence to obtain the visible light feature sequence and thermal infrared feature sequence of the current frame and the previous frame, as well as the heat map feature sequence of the previous frame, which are used as input features of the progressive fusion module.

[0028] Preferably, the process of temporal feature fusion of the visible light features of the current frame and the previous frame in the progressive fusion module is the same as the process of temporal feature fusion of the thermal infrared features of the current frame and the previous frame, including:

[0029] Visible light features of the current frame Visible light features of the previous frame The input is fed into the cross-attention module for fusion, yielding the median value of the fused visible light features, x. v ;

[0030] The median value of the fused visible light features is x v Visible light features of the current frame After addition, normalization is performed. The normalized value is then compared with the heatmap features of the previous frame. Adding them together yields the intermediate value of the fused visible light characteristics.

[0031] The median value of the fused visible light features Input to a feedforward network, output value and the median of the fused visible light features are two values. The sums are then normalized to obtain the final visible light time characteristics.

[0032] Preferably, the process of obtaining refined multimodal features in the progressive fusion module includes:

[0033] visible light time characteristics and thermal infrared time characteristics Adding them together yields the initial multimodal features x. f ;

[0034] With initial multimodal features x f For keys and values, visible light temporal characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. With initial multimodal features x f For bonds and values, thermal infrared time characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. Based on visible light time characteristics For keys and values, the initial multimodal feature x f The query input is fused using a cross-attention module to obtain enhanced multimodal features. Based on thermal infrared time characteristics For keys and values, the initial multimodal feature x f The query input is fused using a cross-attention module to obtain enhanced multimodal features.

[0035] With enhanced modality-specific features Enhanced modality-specific features 2 Enhanced multimodal features Enhanced multimodal features II Construct a multidimensional array to obtain multimodal features Multimodal features The input is fed forward into a network, and the output value is normalized to obtain the final refined multimodal features.

[0036] The advantages provided by this invention are:

[0037] 1. This invention fuses visible light and thermal infrared data for multi-target tracking, proposing a progressive fusion multimodal multi-target tracking framework, PFTrack. This framework effectively fuses the temporal and complementary information of visible light and thermal infrared modes in a progressive manner. In the first fusion stage of progressive fusion, an attention mechanism that does not rely on strict spatial alignment of adjacent frames is used to integrate temporal information well. In the second fusion stage of the progressive fusion module, addition is used to obtain coarse multimodal features. These features are used as bridging features that interact with single-modal features, avoiding the problems caused by direct interaction of single-modal features, and realizing robust tracking of multiple targets in complex scenes.

[0038] 2. In the first fusion stage of progressive fusion, this invention uses a cross-attention module to fuse features from the current frame and the previous frame to capture spatiotemporal context information. To enhance target localization, the heatmap of the previous frame is used as a location condition and integrated into the fused features. Finally, a feedforward network is used to obtain the final temporal features to enhance the representation of target features.

[0039] 3. In the second fusion stage of progressive fusion, this invention uses four cross-attention modules to interact and fuse features and single-modal features, further enhancing the fused features and modality-specific features. Finally, the enhanced features obtained from these four interactions are concatenated and fed into a feedforward neural network to obtain a powerful multimodal feature representation.

[0040] 4. The visible light images and thermal infrared images of this invention can be collected by handheld devices, drones and monitoring platforms. The collected dataset includes data from different angles and environmental conditions. Combining these data from different sources can further enhance the robustness and generalization ability of the tracker, enabling it to adapt to various complex application scenarios. Attached Figure Description

[0041] Figure 1 A flowchart of a multi-target tracking method based on visible light thermal infrared fusion provided for embodiments of the present invention;

[0042] Figure 2 A network architecture diagram of a multi-target tracking system based on visible light thermal infrared fusion provided for embodiments of the present invention;

[0043] Figure 3 The figure shows the experimental results of a multi-target tracking method based on visible light and thermal infrared fusion provided for an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] Example 1

[0046] like Figure 1 As shown, this embodiment provides a multi-target tracking method based on visible light and thermal infrared fusion. This method can be implemented by electronic devices such as terminals or servers, and the processing flow of this method can include the following steps:

[0047] S1. Data acquisition is performed based on a visible light camera to obtain visible light images of the current frame and the previous frame; data acquisition is performed based on a thermal infrared camera to obtain thermal infrared images of the current frame and the previous frame.

[0048] Visible light images provide rich color and texture information, but their data quality is poor in low-light and hazy environments. Thermal infrared data, on the other hand, exhibits good quality in such environments but lacks color and texture information. This invention combines the advantages and disadvantages of both, collecting visible light and thermal infrared data to construct a visual dataset for multi-target tracking. The complementary information from visible light and thermal infrared data enhances the robustness of multi-target tracking in complex environments.

[0049] Visible light and thermal infrared images can both be collected through handheld devices, drones, and monitoring platforms. Handheld devices can provide usage scenarios that are closer to the real world, capturing people's daily activities and dynamic changes. Drones can provide an aerial perspective and a wide field of view, which helps to track the movement and behavior of targets over a large area. Monitoring platforms can provide data for specific environments, such as surveillance cameras in public spaces. Visual datasets collected through handheld devices, drones, and monitoring platforms include data from different angles and environmental conditions. Combining data from these different sources can significantly enhance the robustness and generalization ability of trackers, enabling them to adapt to various complex application scenarios.

[0050] In practical applications, the HIKMICRO HM-TP2ZL-HD10 camera can be used to capture handheld scene data in two modalities. The thermal infrared image resolution is 640×480, while the visible light image resolution is even higher, reaching 1600×1200. The DJI Matrice 300RTK drone is equipped with a ZENMUSE H20T camera for capturing drone data. The thermal infrared image resolution is 640×512, and the visible light image resolution is also 640×512. Data can be collected from surveillance scenes using the Hikvision DS-2TD4136-25 / V2. The visible light and thermal infrared modes have resolutions of 1920×1080 and 384×288, respectively. Furthermore, partial datasets can be collected from existing single-target tracking datasets RGB-T234, LasHeR, and VTUAV, and the collected single-target tracking datasets can be re-annotated to support the multi-target tracking of this invention.

[0051] S2. Input the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, into the convolutional neural network for feature extraction to obtain the visible light features and thermal infrared features of the current frame and the previous frame, as well as the heat map features of the previous frame.

[0052] The heatmap of the previous frame can be created based on the position of the object center in the previous frame. The convolutional neural network (CNN) includes a 7×7 convolutional layer, batch normalization (BN), and rectified linear unit (ReLU). The process of inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heatmap of the previous frame, into the convolutional neural network for feature extraction includes: inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heatmap of the previous frame, into a 7×7 convolutional layer to obtain a feature map with unchanged height and width and 16 channels; dividing the visible light feature map and thermal infrared feature map of the current frame and the previous frame, as well as the heatmap of the previous frame, into 16×16 image blocks and planarizing them into a sequence to obtain the visible light feature sequence, thermal infrared feature sequence, and heatmap feature sequence of the current frame and the previous frame, which are used as input features for the progressive fusion module.

[0053] S3. Based on the progressive fusion module, using the heat map features of the previous frame as the location condition, the visible light features and thermal infrared features of the current frame and the previous frame are fused temporally to obtain visible light temporal features and thermal infrared temporal features. The visible light temporal features and thermal infrared temporal features are added together to obtain initial multimodal features. Using the initial multimodal features as bridging features, the visible light temporal features and thermal infrared temporal features are interacted with the initial multimodal features to obtain enhanced modality-specific features and enhanced multimodal features. The enhanced modality-specific features and enhanced multimodal features are merged, input into the feedforward network, and then normalized to obtain refined multimodal features.

[0054] The present invention describes a fusion process for obtaining visible light temporal features by fusing the visible light features of the current frame and the previous frame. This process is identical to the fusion process for obtaining thermal infrared temporal features by fusing the thermal infrared features of the current frame and the previous frame. The visible light features of the current frame and the previous frame are represented as follows: The thermal infrared features of the current frame and the previous frame are respectively represented as follows:

[0055] The process of fusing the visible light features of the current frame and the previous frame to obtain the visible light temporal features includes:

[0056] 3.1.1. The visible light features of the current frame Visible light features of the previous frame The input is fused into the cross-attention module, which can use the visible light features of the current frame. For query (Q), the visible light features of the previous frame Using keys (K) and values ​​(V) to capture spatiotemporal context information, an intermediate value x of the fused visible light features is obtained. v The calculation method is as follows: in, This indicates a cross-attention module. are the visible light features of the current frame and the previous frame, respectively, and p is the position code.

[0057] 3.1.2. Take the median value of the fused visible light features obtained in step 3.1.1, x v Visible light features of the current frame After addition, normalization is performed, and then the features from the previous frame are fused together. Specifically, it refers to the features of a previous frame of heatmap. As a location condition, the normalized value is compared with the features of the previous frame's heatmap. Adding them together yields the intermediate value of the fused visible light characteristics. The calculation method is as follows: Where LN() represents the normalization operation, x represents the visible light characteristics of the current frame. v The intermediate value of the fused visible light feature obtained in step 3.1.1 is one. Features of the previous frame's heatmap.

[0058] 3.1.3. The intermediate value of the fused visible light feature obtained in step 3.1.2 is taken as the second value. The median value between the output value after inputting into the feedforward network and the fused visible light characteristics. The sums are then normalized to obtain the final visible light time characteristics. The calculation method is as follows: in, FFN() represents the intermediate value of the fused visible light features, and LN() represents the normalization operation.

[0059] Similarly, the process of fusing the thermal infrared features of the current frame and the previous frame to obtain the thermal infrared temporal features includes:

[0060] 3.2.1. Analyze the thermal infrared features of the current frame. Thermal infrared features of the previous frame The input is fused into the cross-attention module, which can use the thermal infrared features of the current frame. For query (Q), the thermal infrared features of the previous frame Using keys (K) and values ​​(V) to capture spatiotemporal context information, an intermediate value of x is obtained from the fused thermal infrared features. ir The calculation method is as follows: in, This indicates a cross-attention module. , respectively, are the thermal infrared features of the current frame and the previous frame, and p is the position code.

[0061] 3.2.2. Take the intermediate value x of the fused thermal infrared features obtained in step 3.2.1. ir Thermal infrared features of the current frame After addition, normalization is performed, and then the features from the previous frame are fused together. Specifically, it refers to the features of a previous frame of heatmap. As a location condition, the normalized value is compared with the features of the previous frame's heatmap. Adding them together, we obtain the intermediate value of the fused thermal infrared characteristics. The calculation method is as follows: Where LN() represents the normalization operation, x represents the visible light characteristics of the current frame. ir The intermediate value of the fused thermal infrared feature obtained in step 3.2.1 is 1. Features of the previous frame's heatmap.

[0062] 3.2.3, The intermediate value of the fused thermal infrared feature obtained in step 3.2.2 is... The median value between the output value after inputting into the feedforward network and the fused thermal infrared characteristics. The sums are then normalized to obtain the final thermal infrared time characteristics. The calculation method is as follows: in, The intermediate value of the fused thermal infrared features is 2, where FFN() represents the feedforward network and LN() represents the normalization operation.

[0063] For target tracking tasks, mining temporal information is crucial for improving performance. This invention's multi-target tracking method based on visible light and thermal infrared fusion employs the baseline method CenterTrack. Since CenterTrack only uses addition operations to integrate temporal features, the current frame lacks strict spatial alignment with the previous frame, making it difficult to effectively integrate information from the previous frame. To address this issue, this invention uses the thermal image features of the previous frame as positional conditions and performs temporal feature fusion on the visible light and thermal infrared features of the current and previous frames respectively. A progressive fusion approach is used to first obtain single-modal temporal features. In the first fusion stage of progressive fusion, an attention mechanism that does not rely on strict spatial alignment between adjacent frames is used to effectively integrate temporal information. The thermal image of the previous frame is taken as a positional condition and integrated into the fused features, enhancing the target localization capability.

[0064] The process of obtaining refined multimodal features specifically includes the following steps:

[0065] 3.3.1. Temporal characteristics of visible light and thermal infrared time characteristics Adding them together yields the initial multimodal features x.f The calculation method is as follows:

[0066] 3.3.2. Using the initial multimodal feature x f Visible light temporal characteristics for key (K) and value (V). The query (Q) input is fused using a cross-attention module to obtain enhanced modality-specific features. With initial multimodal features x f Thermal infrared time characteristics for bond (K) and value (V) The query (Q) input is fused using a cross-attention module to obtain enhanced modality-specific features. Based on visible light time characteristics Given keys (K) and values ​​(V), the initial multimodal features x f The query (Q) input is fused using a cross-attention module to obtain enhanced multimodal features. Based on thermal infrared time characteristics Given keys (K) and values ​​(V), the initial multimodal features x f The query (Q) input is fused using a cross-attention module to obtain enhanced multimodal features.

[0067] 3.3.3 Enhanced modality-specific features can be generated by constructing multidimensional arrays. Enhanced modality-specific features 2 Enhanced multimodal features Enhanced multimodal features II Connect them together to obtain multimodal features The calculation method is as follows: Then multimodal features The input is fed forward into a network, and the output value is normalized to obtain the final refined multimodal features. The calculation method is as follows:

[0068] In multimodal feature fusion, effective multimodal interactions are crucial for enhancing feature representations. While cross-attention modules are a common method for directly interacting two modalities, they often enhance similarity information (i.e., homogeneous information) between modalities and may ignore modality-specific information (i.e., heterogeneous information). Therefore, in the second fusion stage of the progressive fusion module, this invention uses addition to obtain coarse multimodal features, which are used as bridging features to interact with single-modal features, avoiding the problems caused by direct interaction of single-modal features. Specifically, we use four cross-attention modules to interact with fused features and single-modal features to further enhance the fused features and modality-specific features. Finally, the enhanced features obtained from these four interactions are concatenated and fed into a feedforward neural network to obtain a robust multimodal feature representation.

[0069] S4. Refine multimodal features Input the data into the baseline CenterTrack tracking network to obtain multi-target tracking results.

[0070] Example 2

[0071] The present invention also provides a multi-target tracking system based on visible light and thermal infrared fusion, the system comprising:

[0072] The data acquisition module is used to acquire data based on a visible light camera to obtain visible light images of the current frame and the previous frame; and to acquire data based on a thermal infrared camera to obtain thermal infrared images of the current frame and the previous frame.

[0073] The feature extraction module is used to input the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, into the convolutional neural network for feature extraction, so as to obtain the visible light feature and thermal infrared feature of the current frame and the previous frame, as well as the heat map feature of the previous frame.

[0074] The feature extraction module's process of inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heatmap of the previous frame, into a convolutional neural network for feature extraction includes: inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heatmap of the previous frame, into a 7×7 convolutional layer to obtain feature maps with unchanged height and width and 16 channels; dividing the visible light feature map and thermal infrared feature map of the current frame and the previous frame, as well as the heatmap of the previous frame, into 16×16 image blocks and planarizing them into sequences to obtain the visible light feature sequence, thermal infrared feature sequence, and heatmap feature sequence of the current frame and the previous frame, which serve as the input features for the progressive fusion module.

[0075] The progressive fusion module uses the heatmap features of the previous frame as location conditions to perform temporal feature fusion on the visible light features and thermal infrared features of the current frame and the previous frame, respectively, to obtain visible light temporal features and thermal infrared temporal features. The visible light temporal features and thermal infrared temporal features are added together to obtain initial multimodal features. Using the initial multimodal features as bridging features, the visible light temporal features and thermal infrared temporal features are interacted with the initial multimodal features to obtain enhanced modality-specific features and enhanced multimodal features. The enhanced modality-specific features and enhanced multimodal features are merged, input into the feedforward neural network, and then normalized to obtain refined multimodal features.

[0076] The progressive fusion module, during the first stage of fusion, includes a fusion process that fuses the visible light features of the current frame and the previous frame to obtain visible light temporal features, and a fusion process that fuses the thermal infrared features of the current frame and the previous frame to obtain thermal infrared temporal features. The two fusion processes are identical. The visible light features of the current frame and the previous frame are represented as follows: The thermal infrared features of the current frame and the previous frame are respectively represented as follows:

[0077] The first stage of fusion involves temporal feature fusion of the visible light features of the current frame and the previous frame to obtain the visible light temporal features. This process includes:

[0078] 3.1.1. The visible light features of the current frame Visible light features of the previous frame The input is fused into the cross-attention module, which can use the visible light features of the current frame. For query (Q), the visible light features of the previous frame Using keys (K) and values ​​(V) to capture spatiotemporal context information, an intermediate value x of the fused visible light features is obtained. v The calculation method is as follows: in, This indicates a cross-attention module. are the visible light features of the current frame and the previous frame, respectively, and p is the position code.

[0079] 3.1.2. Take the median value of the fused visible light features obtained in step 3.1.1, x v Visible light features of the current frame After addition, normalization is performed, and then the features from the previous frame are fused together. Specifically, it refers to the features of a previous frame of heatmap. As a location condition, the normalized value is compared with the features of the previous frame's heatmap. Adding them together yields the intermediate value of the fused visible light characteristics. The calculation method is as follows: Where LN() represents the normalization operation, x represents the visible light characteristics of the current frame. v The intermediate value of the fused visible light feature obtained in step 3.1.1 is one. Features of the previous frame's heatmap.

[0080] 3.1.3. The intermediate value of the fused visible light feature obtained in step 3.1.2 is taken as the second value. The median value between the output value after inputting into the feedforward network and the fused visible light characteristics. The sums are then normalized to obtain the final visible light time characteristics. The calculation method is as follows: in, FFN() represents the intermediate value of the fused visible light features, and LN() represents the normalization operation.

[0081] The second phase of the progressive fusion module's fusion process includes the following steps:

[0082] 3.3.1. Temporal characteristics of visible light and thermal infrared time characteristics Adding them together yields the initial multimodal features x. f The calculation method is as follows:

[0083] 3.3.2. Using the initial multimodal feature x f Visible light temporal characteristics for key (K) and value (V). The query (Q) input is fused using a cross-attention module to obtain enhanced modality-specific features. With initial multimodal features x f Thermal infrared time characteristics for bond (K) and value (V) The query (Q) input is fused using a cross-attention module to obtain enhanced modality-specific features. Based on visible light time characteristics Given keys (K) and values ​​(V), the initial multimodal features x f The query (Q) input is fused using a cross-attention module to obtain enhanced multimodal features. Based on thermal infrared time characteristics Given keys (K) and values ​​(V), the initial multimodal features x f The query (Q) input is fused using a cross-attention module to obtain enhanced multimodal features.

[0084] 3.3.3 Enhanced modality-specific features can be generated by constructing multidimensional arrays. Enhanced modality-specific features 2 Enhanced multimodal features Enhanced multimodal features II Connect them together to obtain multimodal features The calculation method is as follows: Then multimodal features The input is fed forward into a network, and the output value is normalized to obtain the final refined multimodal features. The calculation method is as follows:

[0085] The data processing module is used to input refined multimodal features into the baseline CenterTrack tracking network to obtain multi-target tracking results.

[0086] Quantitative evaluation

[0087] In the field of computer vision tracking, visible light data and near-infrared data are selected as the two modalities of this application. In order to evaluate the performance of the multimodal multi-target tracking algorithm, this invention evaluates the algorithm of this application by comparing its tracking performance with some classic and state-of-the-art single-modal trackers on the multimodal multi-target tracking dataset VT-MOT, so as to verify the effectiveness of the proposed method (see Table 1).

[0088] Table 1 Comparison of Method Results

[0089]

[0090] This invention extends a single-modal tracker into a multimodal tracker by modal summation. The evaluation phase focuses on two main metrics: Multi-Object Tracking Accuracy (MOTA) and Higher-Order Tracking Accuracy (HOTA). MOTA is a traditional benchmark for detection performance but lacks representation of correlation performance. To address this issue, Luxent et al. introduced HOTA, which separately evaluates detection (DetA) and correlation (AssA) performance, effectively integrating these two aspects into a single metric. This invention also uses IDF1 for object correlation analysis, DetA for detection analysis, and MOTP for object position accuracy during tracking. The results show that the proposed method significantly improves the performance in computer vision tracking.

[0091] Qualitative evaluation

[0092] like Figure 3As shown, the qualitative comparison in the VT-MOT test set is as follows: Challenge attributes are displayed at the bottom, including similar targets (O), low illumination (L), strong light interference (H), partial occlusion (P), and small targets (S). The first frame is the ground truth tracking result under visible light, and the other two frames are the tracking results of the method of this invention and other algorithms under different temporal thermal infrared conditions. The tracking results of the tracker of this invention are compared with those of two other trackers in two consecutive partial frames selected from the test set sequence, and better results are obtained.

[0093] exist Figure 3 In the upper part of the video sequence, challenges such as high-light interference, small objects, partial occlusion, and similar objects exist. Compared to OC-SORT and ByteTrack, this invention still provides good tracking results in such a complex mixed challenge scenario. Figure 3 In the lower half of the video sequence, challenges such as low light and small objects exist. ByteTrack and OC-SORT suffer from severe missed detections in these situations. However, this invention effectively addresses these challenges. Its core advantage lies in its ability to effectively fuse complementary information between different modalities and to extract contextual information about small objects, thereby enhancing target tracking capabilities.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target tracking method based on visible light and thermal infrared fusion, characterized in that: The method includes: Data is acquired using a visible light camera to obtain visible light images of the current frame and the previous frame; data is also acquired using a thermal infrared camera to obtain thermal infrared images of the current frame and the previous frame. The visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, are respectively input into the convolutional neural network for feature extraction to obtain the visible light features and thermal infrared features of the current frame and the previous frame, as well as the heat map features of the previous frame. Using the thermal image features of the previous frame as a location condition, temporal feature fusion is performed on the visible light features and thermal infrared features of the current frame and the previous frame respectively to obtain visible light temporal features and thermal infrared temporal features. The visible light temporal features and thermal infrared temporal features are added to obtain initial multimodal features. Using the initial multimodal features as bridging features, the visible light temporal features and thermal infrared temporal features are interacted with the initial multimodal features respectively to obtain enhanced modality-specific features and enhanced multimodal features. The enhanced modality-specific features and enhanced multimodal features are merged, input into a feedforward neural network, and then normalized to obtain refined multimodal features. The process of temporal feature fusion of the visible light features of the current frame and the previous frame is the same as the process of temporal feature fusion of the thermal infrared features of the current frame and the previous frame, including: The visible light features of the current frame Visible light characteristics of the previous frame The input is fed into the cross-attention module for fusion, yielding the median value of the fused visible light features. ; The median value of the fused visible light features Visible light features of the current frame After addition, normalization is performed. The normalized value is then compared with the heatmap features of the previous frame. Adding them together yields the intermediate value of the fused visible light characteristics. ; The median value of the fused visible light features Input to a feedforward network, output value and the median of the fused visible light features are two values. After summing, normalization is performed to obtain the final visible light time characteristics. ; The refined multimodal features are input into the baseline CenterTrack tracking network to obtain multi-target tracking results.

2. The multi-target tracking method based on visible light and thermal infrared fusion according to claim 1, characterized in that: The visible light camera and thermal infrared camera are handheld devices, drones, or monitoring platforms.

3. The multi-target tracking method based on visible light and thermal infrared fusion according to claim 1, characterized in that: The convolutional neural network includes 7×7 convolutional layers, batch normalization, and correction linear units.

4. The multi-target tracking method based on visible light and thermal infrared fusion according to claim 3, characterized in that: The process of inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, into a convolutional neural network for feature extraction includes: inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, into a 7×7 convolutional layer to obtain feature maps with unchanged height and width and 16 channels; dividing the visible light feature map and thermal infrared feature map of the current frame and the previous frame, as well as the heat map of the previous frame, into 16×16 image blocks and planarizing them into sequences to obtain the visible light feature sequence and thermal infrared feature sequence of the current frame and the previous frame, as well as the heat map feature sequence of the previous frame.

5. The multi-target tracking method based on visible light and thermal infrared fusion according to claim 1, characterized in that: The process of obtaining refined multimodal features includes: visible light temporal characteristics and thermal infrared time characteristics Adding them together yields the initial multimodal features. ; With initial multimodal features For keys and values, visible light temporal characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. With initial multimodal features For bonds and values, thermal infrared time characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. Based on visible light time characteristics Initial multimodal features for keys and values The query input is fused using a cross-attention module to obtain enhanced multimodal features. Based on thermal infrared time characteristics Initial multimodal features for keys and values The query input is fused using a cross-attention module to obtain enhanced multimodal features. ; With enhanced modality-specific features Enhanced modal-specific features II Enhanced multimodal features Enhanced multimodal features Construct a multidimensional array to obtain multimodal features Multimodal features The input is fed forward into a network, and the output value is normalized to obtain the final refined multimodal features. .

6. A multi-target tracking system based on visible light, thermal, and infrared fusion, characterized in that: The system includes: The data acquisition module is used to acquire data based on a visible light camera to obtain visible light images of the current frame and the previous frame; and to acquire data based on a thermal infrared camera to obtain thermal infrared images of the current frame and the previous frame. The feature extraction module is used to input the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heat map of the previous frame, into the convolutional neural network for feature extraction, so as to obtain the visible light feature and thermal infrared feature of the current frame and the previous frame, as well as the heat map feature of the previous frame. The progressive fusion module uses the thermal image features of the previous frame as a location condition to perform temporal feature fusion on the visible light and thermal infrared features of the current and previous frames, respectively, to obtain visible light temporal features and thermal infrared temporal features. The visible light and thermal infrared temporal features are then added together to obtain initial multimodal features. Using these initial multimodal features as bridging features, the visible light and thermal infrared temporal features are interacted with the initial multimodal features to obtain enhanced modality-specific features and enhanced multimodal features. These enhanced modality-specific features and enhanced multimodal features are then merged, input into a feedforward neural network, and normalized to obtain refined multimodal features. The process of temporal feature fusion on the visible light features of the current and previous frames is the same as the process of temporal feature fusion on the thermal infrared features of the current and previous frames, including: The visible light features of the current frame Visible light characteristics of the previous frame The input is fed into the cross-attention module for fusion, yielding the median value of the fused visible light features. ; The median value of the fused visible light features Visible light features of the current frame After addition, normalization is performed. The normalized value is then compared with the heatmap features of the previous frame. Adding them together yields the intermediate value of the fused visible light characteristics. ; The median value of the fused visible light features Input to a feedforward network, output value and the median of the fused visible light features are two values. After summing, normalization is performed to obtain the final visible light time characteristics. ; The data processing module is used to input refined multimodal features into the baseline CenterTrack tracking network to obtain multi-target tracking results.

7. The multi-target tracking system based on visible light and thermal infrared fusion according to claim 6, characterized in that: The feature extraction module's process of inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heatmap of the previous frame, into a convolutional neural network for feature extraction includes: inputting the visible light image and thermal infrared image of the current frame and the previous frame, as well as the heatmap of the previous frame, into a 7×7 convolutional layer to obtain feature maps with unchanged height and width and 16 channels; dividing the visible light feature map and thermal infrared feature map of the current frame and the previous frame, as well as the heatmap of the previous frame, into 16×16 image blocks and planarizing them into a sequence to obtain the visible light feature sequence, thermal infrared feature sequence, and heatmap feature sequence of the current frame and the previous frame, which serve as the input features of the progressive fusion module.

8. The multi-target tracking system based on visible light and thermal infrared fusion according to claim 6, characterized in that: The process of obtaining refined multimodal features in the progressive fusion module includes: visible light temporal characteristics and thermal infrared time characteristics Adding them together yields the initial multimodal features. ; With initial multimodal features For keys and values, visible light temporal characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. With initial multimodal features For bonds and values, thermal infrared time characteristics The query input is fused using a cross-attention module to obtain enhanced modality-specific features. Based on visible light time characteristics Initial multimodal features for keys and values The query input is fused using a cross-attention module to obtain enhanced multimodal features. Based on thermal infrared time characteristics Initial multimodal features for keys and values The query input is fused using a cross-attention module to obtain enhanced multimodal features. ; With enhanced modality-specific features Enhanced modal-specific features II Enhanced multimodal features Enhanced multimodal features Construct a multidimensional array to obtain multimodal features Multimodal features The input is fed forward into a network, and the output value is normalized to obtain the final refined multimodal features. .

Images