Graph-guided differential fusion RGB-T target detection method

Abstract

The invention provides a graph-guided differential fusion RGB-T target detection method, and the frame of the method comprises two innovative modules: a graph enhancement module (GEM), which achieves the feature refinement in a modal through space-channel graph convolution, and a graph fusion module, which achieves the feature refinement in the modal through space-channel graph convolution; and the second part is a graph difference fusion module (Graph Difference Fusion, GDF) which uses the design thought of a feedback amplification circuit for reference and uses the iteration relation graph difference to realize cross-modal fusion. A large number of experiments on two reference data sets of KAIST and LLVIP show that compared with a reference model YOLOv9, the performance of the framework is remarkably improved, improvement of 5.95% and 3.39% is achieved on the index of mAP at 0.5: 0.95, and meanwhile training loss is reduced by 10.2%. Besides, the framework can effectively balance the detection precision and the recall rate in various complex scenes such as extremely low light, shielding and scale change, and the robustness of the framework in actual scene deployment is verified.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to a graph-guided differential fusion RGB-T target detection method. Background Technology

[0002] RGB-T target detection integrates visible light (RGB) and infrared (IR) images, leveraging the rich texture details of the former and the thermal radiation characteristics of the latter, which are unaffected by visible light illumination. This aims to achieve robust perception in all weather conditions and complex environments. The technology has broad application prospects in fields such as autonomous driving, intelligent security, and military reconnaissance.

[0003] However, effectively fusing information from the two heterogeneous modalities of RGB and infrared remains a core challenge for RGB-T target detection. While existing fusion strategies have made some progress, they still have several shortcomings, mainly in the following aspects: (1) Limitations of early and mid-stage fusion: Early fusion methods (such as pixel-level stitching) do not fully consider the semantic and physical differences between modalities, and are prone to introducing a lot of background noise, leading to feature confusion. Mid-stage fusion methods (such as feature-level stitching or weighted summation) alleviate this problem to some extent, but their fusion rules are often static or heuristic and cannot adapt to complex and ever-changing scenarios.

[0004] (2) Shortcomings of attention-based fusion: In recent years, attention-based fusion methods (such as CFT, ICAFusion, etc.) have become mainstream. These methods improve the flexibility of fusion by calculating attention maps within or between modalities to weight features. However, most of them focus on modeling the "commonality" or "cooperation" relationship between modalities, while ignoring a key issue: RGB and infrared modalities have an asymmetric, complementary, or even competitive relationship in specific scenarios. For example, RGB information may dominate during the day; while infrared information is more reliable in extreme low-light environments. Existing attention mechanisms are difficult to accurately quantify and utilize this dynamic asymmetry.

[0005] (3) The challenge of balancing background noise suppression and complementary feature preservation: RGB and infrared images not only contain complementary features that are beneficial for target detection, but also share a large amount of redundant background information (such as sky, vegetation, buildings, etc.). Many existing fusion methods cannot effectively suppress these shared background noises while fully preserving and enhancing the unique complementary features in their respective modalities. This leads to a significant decrease in the model's discriminative ability in complex scenes such as occlusion, extreme lighting changes, and small targets, resulting in an increased false alarm rate or insufficient recall.

[0006] Specifically, while existing technologies such as the IRDFusion method introduce the concept of relational graph differences, their utilization of difference information is relatively direct, lacking a learnable and structured guiding mechanism to finely process difference signals. This results in sensitivity to noise and room for improvement in fusion efficiency. Other Transformer-based fusion methods, due to the high complexity of global computation and insufficient modal relationship modeling, face the dilemma of balancing efficiency and accuracy in practical deployments. Summary of the Invention

[0007] The purpose of this invention is to provide a graph-guided differential fusion RGB-T target detection method that can guide the fusion of multimodal information in a more refined and intelligent way, thereby improving the detection robustness and accuracy in complex scenarios.

[0008] To address the aforementioned technical problems, this invention provides a graph-guided differential fusion RGB-T target detection method, comprising the following steps: S1. Obtain the input RGB-T image pair, including the RGB image and the infrared image; S2. Extract RGB feature map and infrared feature map ; S3. The feature map is refined within the modality using the graph enhancement module; S4. The refined features are fused across modalities using the graph difference fusion module to generate fused features. S5. Perform target detection based on the fused features and output the detection results.

[0009] More preferably, the graph enhancement module includes:

[0010] in, The graph convolution operator is implemented as follows:

[0011] In the formula, and For science departments convolution kernels, This represents the sigmoid activation function and its output. Used to capture relationship patterns; Channel graph relationship modeling unit, used to calculate inter-modal relationships along the channel dimension:

[0012] in, For average pooling, and It is a fully connected projection matrix.

[0013] More preferably, the graph difference fusion module includes: The relationship graph difference calculation unit is used to calculate the difference in a two-way relationship graph.

[0014] in, and Correlation diagrams for RGB to infrared and infrared to RGB are shown below. Differential coding units are used to encode the differences in the relational graph:

[0015] Where DiffEncoder is a learnable differential encoder; Differential weight generation unit, used to generate adaptive fusion weights:

[0016]

[0017] Where BaseWeight is the basic weight calculation function, and DiffCorrection is the difference correction term, defined as:

[0018] It is an adjustable hyperparameter to control the degree of influence of differential guidance.

[0019] More preferably, the base weight BaseWeight is calculated as follows:

[0020] Here, selfAm represents the self-attention mechanism, and MLP stands for Multilayer Perceptron.

[0021] More preferably, the graph enhancement module and the graph difference fusion module work together in an iterative manner to optimize the feature representation through multiple iterations.

[0022] More preferably, the object detection is implemented based on the YOLOv9 dual-stream architecture and trained and validated on the KAIST or LLVIP dataset.

[0023] More preferably, the method achieves robust RGB-T target detection in extreme low-light, occlusion, or scale-changing scenarios.

[0024] Compared with the prior art, the present invention has the following beneficial effects: Firstly, it enhances the discriminative power and detection accuracy of fusion: By introducing a graph difference fusion module, this invention can explicitly model and utilize the asymmetric complementary relationship between RGB and infrared modes to dynamically evaluate the "dominance" of each mode in different spatial locations and feature channels, thereby achieving more refined and adaptive feature fusion. Experiments on publicly available benchmark datasets such as KAIST and LLVIP show that the complete framework of this invention achieves significant improvements of 5.95% and 3.39% respectively on the benchmark model YOLOv9, at the more stringent evaluation metrics mAP@0.5 and 0.95.

[0025] Secondly, it effectively suppresses common background noise and enhances model robustness: The graph enhancement module in this invention uses spatial-channel graph convolution to structurally refine intra-modal features, effectively filtering background interference unrelated to the target. Combined with the differential fusion mechanism to suppress insignificant differences, the entire system maintains high-precision detection capabilities in challenging scenarios such as extreme low light, severe occlusion, scale variations, and complex backgrounds, achieving a good balance between precision and recall, demonstrating its excellent environmental robustness.

[0026] Thirdly, it achieves more efficient feature optimization and faster model convergence: the graph enhancement module and graph difference fusion module work iteratively and collaboratively, forming an effective feedback optimization mechanism. This mechanism guides the model to focus on discriminative features more quickly during training, thereby accelerating the model's convergence process. Experimental data shows that the training loss (including bounding box loss and classification loss) of this invention is reduced by more than 10% compared to the baseline model, which means that while achieving the same or better performance, it reduces the computational resources and time costs required for training.

[0027] Fourth, it provides an interpretable fusion mechanism and flexible scalability: the fusion weights of this invention are not directly output through a "black box" model, but rather originate from interpretable differences in the relational graph. This design makes the fusion process more transparent and facilitates analysis and debugging. Simultaneously, the modular design (GEM and GDF) does not depend on a specific backbone network and can be flexibly integrated into other dual-stream detection architectures, providing a convenient technical foundation for future expansion to more modalities (such as depth maps and radar point clouds). Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the TF1 module; Figure 2 This is a schematic diagram of the TF2 module; Figure 3 These are representative detection results of the YOLOv9-based RGB-T detection framework on the KAIST dataset; Figure 4These are the detection results from the KAIST dataset. Detailed Implementation

[0029] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more detailed explanation of the graph-guided differential fusion RGB-T target detection method proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise scales, and are only used to facilitate and clarify the illustration of the embodiments of this invention. The same or similar reference numerals in the drawings represent the same or similar parts.

[0030] Example: A graph-guided differential fusion RGB-T target detection method This invention proposes a novel graph-guided differential fusion framework, the core design of which includes: (1) a graph enhancement module (GEM): constructing a spatial-channel graph, where nodes represent feature locations, edges encode local consistency patterns, and intra-modal feature refinement is achieved through graph convolution; (2) a graph differential fusion module (GDF): extending the relational graph difference idea in the IRDFusion method, introducing a learnable differential encoder and an adaptive gating mechanism (Equations 6 and 12) to achieve modality-aware weight correction, rather than using fixed fusion rules. Validation results of this graph-based method on the KAIST and LLVIP benchmark datasets show that it can achieve better discriminative fusion results.

[0031] The framework proposed in this invention comprises two core modules: (A) Graph Differential Cross Attention Module (GEM / DiffTF1) – nodes represent spatial locations {(h,w)}, and edges capture pixel-level similarity through feature difference weighting (Equations 2-4); (B) Differential Adaptive Fusion Module (GDF / DifferentialAdaptiveTF2) – extending the IRDFusion method, adding a learnable encoder (Equation 9) and a gating correction term (Equation 12) to achieve adaptive weight modulation, which differs from the design of directly applying difference information in IRDFusion.

[0032] A. Graph Difference Cross Attention 1) Graph Relationship Analysis RGB-T fusion requires modeling spatial-semantic relationships as a directed graph: each node in the graph Corresponding position in the feature map ( Edge weights Local consistency patterns (such as texture-thermal imaging alignment) are captured by quantizing cross-modal correlation strengths through learnable graph convolution (Equation 3). Given an RGB feature map... With infrared feature map First, flatten it into a sequence representation: ; in It is a spatial position function.

[0033] By calculating pixel-level differences and applying graph convolution, the spatial relationships are analyzed. ; in, The graph convolution operator is implemented as follows: ; In the formula, and For science departments convolution kernels, This represents the sigmoid activation function and its output. Used to capture relationship patterns.

[0034] Channel graph relationships and spatial analysis complement each other, modeling gradient relationships along the channel dimension through global up-down subdivision: ; in, For average pooling, and This is a fully connected projection matrix. Output results. Used to encode channel-level correlation strength.

[0035] TF1 modules such as Figure 1 As shown.

[0036] 2) Graph Difference Coding The core innovation of this invention lies in encoding the differences between bidirectional relationships. The specific calculation is as follows: ; This difference value captures the asymmetric dependency between modes: a positive value indicates a stronger influence from the RGB mode, while a negative value indicates that the infrared mode is dominant. This asymmetric relationship is encoded using a gating encoder. ; Among them, gating mechanism It can selectively emphasize patterns of difference that have informational value while suppressing noise.

[0037] B. Differential Adaptive TF2 1) Calculation of differences in the relationship diagram Based on the bidirectional relation graph difference of the IRDFusion method, this invention improves upon it in the following two ways: (1) by using a learnable differential encoder (Equation 9) instead of a manually designed differential aggregation method; (2) by introducing a gating correction mechanism (Equation 12) to achieve adaptive weight modulation, rather than the fixed linear combination in IRDFusion. For each pair of ordered modes, the correlation graph is first calculated: ; The differences in the relationship diagram are calculated as follows: ; This difference signal This is used to quantify modal dominance at each spatial location. The signal is encoded using a multilayer perceptron (MLP): ; C. Differential Weight Generation Traditional fusion methods use fixed weights or simple adaptive weights, while the differential weight generation strategy proposed in this invention can dynamically adjust the weights based on the differences in the relationship graph. ; The base weight follows the original TF2 mechanism and is calculated as follows: ; The difference correction term (DiffCorrection) adaptively adjusts the weights based on modality dominance: ; in, Used to control the degree of influence of differential guidance.

[0038] TF2 modules such as Figure 2 As shown.

[0039] experiment To fully verify the effectiveness of the graph-guided differential fusion module proposed in this invention, extensive experiments were conducted on two widely used RGB-T detection benchmark datasets (KAIST and LLVIP). Eight model configurations were designed based on the YOLOv9 two-stream architecture to compare system performance: (1) Baseline model – the original two-stream architecture without any enhancements; (2) +Graph Enhancement Module (GEM) – incorporating the graph-based feature enhancement module proposed in this invention; (3) +Graph Differential Fusion Module (GDF) – adding a differential cross-attention mechanism; (4) +GDF+GEM – a complete framework fusing the two modules. Through this ablation experiment design, the contribution of each module can be separated, and its synergistic effect in complex scenarios such as low light, occlusion, and changes in thermal contrast can be verified.

[0040] Detection results on the KAIST dataset Figure 3 This presentation showcases representative detection results of the YOLOv9-based RGB-T detection framework on the KAIST dataset. The visualizations cover 18 complex scenes, including day-night variations, occlusion, dense crowds, and changes in illumination. Each image set includes both RGB (left) and infrared (right) modalities, with green boxes representing predicted bounding boxes and confidence levels labeled within each box.

[0041] Figure 4 The detection results on the KAIST dataset show that each image set contains RGB-infrared paired images, with green boxes marking detected pedestrians and confidence scores within the boxes. The results demonstrate the robustness of the framework in the following scenarios: (Row 1) daytime pedestrian crossings (good visibility); (Row 2) dense urban environments (multiple pedestrians); (Row 3) nighttime low-light scenes; (Row 4) extreme dark environments (dependent on thermal imaging features); and (Row 5) complex lighting conditions with mixed visible / infrared contrast.

[0042] Quantitative performance analysis Table 1 provides a comprehensive quantitative evaluation of the performance of all model configurations on the KAIST and LLVIP datasets. All models were trained for 50 epochs, and the key metrics reported include precision, recall, mean precision at different IoU thresholds (mAP@0.5, mAP@0.5:0.95), and training loss and validation loss (if any) (see Table 1 for details).

[0043] Table 1. Quantitative performance comparison on KAIST and LLVIP datasets (all models were trained for 50 epochs, and the best results for each dataset are marked in bold). ; Table 2 summarizes the performance improvements of the complete framework (GDF + GEM) compared to the baseline model: Table 2: Performance advantages compared to the baseline model ; Experimental results validate the effectiveness of the proposed framework while demonstrating dataset-specific performance trade-offs: on the KAIST dataset, the GDF+GEM framework achieves consistent improvements across all metrics; on the LLVIP dataset, the framework prioritizes optimizing localization accuracy (mAP@0.5:0.95, a 3.39% improvement), while the slight fluctuations in precision / recall (-0.41% / -0.29%) stem from more rigorous bounding box optimization rather than missed detections. Qualitative visualizations and quantitative metrics both indicate that the framework outperforms the benchmark model YOLOv9. Ablation experiments further confirm that the synergistic effect of the graph augmentation and differential fusion modules is key to achieving robust RGB-T object detection.

[0044] This invention proposes a novel graph-guided differential fusion framework to address the core issue of effective RGB-infrared information fusion in RGB-T object detection. Inspired by the iterative relational graph difference mechanism in the IRDFusion method, two collaborative modules are designed: a graph enhancement module (GEM) – which refines intra-modal features through spatial and channel graph convolution; and a graph differential fusion module (GDF) – which adaptively fuses cross-modal features using iterative differential feedback. Extensive experiments on the KAIST and LLVIP datasets demonstrate that this framework significantly improves performance compared to the benchmark model YOLOv9: a 5.95% improvement in mAP on the KAIST dataset and a 3.39% improvement on the LLVIP dataset (the precision-recall tradeoff stems from more refined localization optimization), while simultaneously reducing training loss (a 10.2% reduction in bounding box loss and a 13.6% reduction in classification loss).

[0045] Both quantitative and qualitative results confirm that graph-based feature processing and differential-guided fusion can effectively suppress background noise while enhancing the features of significant targets—performing excellently in complex scenarios such as extreme low light, occlusion, and scale variations. Future work will focus on the following directions: (1) lightweight optimization to meet real-time deployment requirements; (2) extending the framework to more modalities (such as depth perception and radar perception) to achieve comprehensive environmental perception in scenarios such as autonomous driving.

[0046] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A graph-guided differential fusion RGB-T target detection method, characterized in that, Includes the following steps: S1. Obtain the input RGB-T image pair, including the RGB image and the infrared image; S2. Extract RGB feature map and infrared feature map ; S3. The feature map is refined within the modality using the graph enhancement module; S4. The refined features are fused across modalities using the graph difference fusion module to generate fused features. S5. Perform target detection based on the fused features and output the detection results.

2. The graph-guided differential fusion RGB-T target detection method according to claim 1, characterized in that, The graph enhancement module includes: ; in, The graph convolution operator is implemented as follows: ; In the formula, and For science departments convolution kernels, This represents the sigmoid activation function and its output. Used to capture relationship patterns; Channel graph relationship modeling unit, used to calculate inter-modal relationships along the channel dimension: ; in, For average pooling, and It is a fully connected projection matrix.

3. The graph-guided differential fusion RGB-T target detection method according to claim 1, characterized in that, The graph difference fusion module includes: The relationship graph difference calculation unit is used to calculate the difference in a two-way relationship graph. ; in, and Correlation diagrams for RGB to infrared and infrared to RGB are shown below. Differential coding units are used to encode the differences in the relational graph: ; Where DiffEncoder is a learnable differential encoder; Differential weight generation unit, used to generate adaptive fusion weights: ; ; Where BaseWeight is the basic weight calculation function, and DiffCorrection is the difference correction term, defined as: ; It is an adjustable hyperparameter to control the degree of influence of differential guidance.

4. The graph-guided differential fusion RGB-T target detection method according to claim 3, characterized in that, The calculation method for the base weight BaseWeight is as follows: ; Here, selfAm represents the self-attention mechanism, and MLP stands for Multilayer Perceptron.

5. The graph-guided differential fusion RGB-T target detection method according to claim 1, characterized in that, The graph enhancement module and the graph difference fusion module work together in an iterative manner, optimizing the feature representation through multiple iterations.

6. The graph-guided differential fusion RGB-T target detection method according to claim 1, characterized in that, The object detection is implemented based on the YOLOv9 two-stream architecture and trained and validated on the KAIST or LLVIP datasets.

7. The graph-guided differential fusion RGB-T target detection method according to claim 1, characterized in that, The method achieves robust RGB-T target detection in extreme low-light, occlusion, or scale-changing scenarios.

Images