Asymmetric dual-path gating and vlm arbitration based rgb-infrared target detection method

Abstract

The application discloses an RGB-infrared target detection method based on asymmetric dual-channel gating and VLM arbitration, and relates to multi-modal target detection and cross-modal representation fusion technology in embodied intelligent scenes. Multi-modal input data in embodied scenes is acquired and preprocessed to generate cross-modal input pairs; an asymmetric topology is constructed based on the cross-modal input pairs, and a constraint relationship of "semantic mainstream-structure prompt flow-semantic arbitration flow" is established; the infrared modality in the multi-modal input data is converted into an injectable structure prompt, and StarFusion star fusion is introduced into the semantic mainstream to realize efficient global modeling; semantic arbitration is performed by using a frozen VLM to generate a semantic consistency graph as an interpretable arbitration signal for gating fusion; gating fusion is performed through semantic-guided denoising fusion to realize a unified closed loop of night recovery and thermal noise suppression. The application has advantages in cross-modal manifold consistency, noise robustness and multi-scene generalization ability.

Description

Technical Field

[0001] This invention relates to multimodal target detection and cross-modal characterization fusion technology in embodied intelligence scenarios, and more specifically to an RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration. Background Technology

[0002] In the fields of multimodal object detection (MMOD) and visible-infrared object detection (MSOD), leveraging the texture semantic information of visible light (RGB) sensors and the radiometric imaging characteristics of thermal infrared (IR) sensors can improve the perception robustness of intelligent systems under complex conditions such as low light and severe weather. However, cross-modal fusion remains a key challenge. Existing technologies typically follow a "dual-stream / multi-stream network + feature fusion" approach, feeding aligned (RGB) and (IR) data into a coding backbone network and performing interactive fusion at multi-scale feature layers to obtain fused features that combine semantic and structural characteristics for downstream detection tasks. These methods often employ a "symmetric architecture" in engineering, meaning that the RGB and IR branches are designed isomorphically or nearly isomorphically in the network topology, and dual-modal information integration is achieved at the neck / head through cross-modal attention or splicing. Representative approaches such as CMX and ICA-Fusion, which focus on cross-modal fusion, belong to this symmetric fusion paradigm.

[0003] The aforementioned symmetrical fusion approach implicitly assumes that "RGB and IR information are equivalent and complementary," and tends to drive the learning of both modal branches simultaneously under the same optimization objective through joint training. However, this type of symmetrical design is prone to the so-called "symmetry fallacy": due to the inherent heterogeneity of the two modalities in statistical distribution and representational properties, the joint training process can lead to gradient conflicts, resulting in mutual interference and degradation of their respective feature manifolds. Especially when modern mainstream backbone networks have "RGB-centric" priors, forcibly inputting texture-deficient thermal infrared tensors into isomorphic backbones can induce manifold mismatches, making the fusion no longer complementary enhancement but feature contamination and performance degradation.

[0004] To mitigate the impact of cross-modal domain differences and noise, existing technologies have developed several enhancement paths based on "thermal domain adaptation + complex interaction": one is to introduce generative or contrastive domain transfer outside the thermal infrared region (e.g., mapping the IR distribution to a form closer to visible light through contrastive generation / translation mechanisms); the other is to employ strategies such as Frequency Domain Adaptation (FDA) for spectral domain transformation to alleviate modal distribution differences. Simultaneously, in the fusion stage, some schemes tend to introduce attention mechanisms with large parameters or complex interactions (e.g., DeformableAttention) to enhance cross-modal alignment and target region interaction modeling capabilities. While these schemes can improve performance under certain conditions, their fusion logic is still largely based on "blind fusion" of local statistics or strongly interactive modules, lacking explicit semantic discrimination criteria for "thermal rather than target" background interference, which may still lead to noise accumulation.

[0005] With the development of basic models, Visual Language Models (VLMs), especially models like CLIP, have shown outstanding performance in visual-text semantic alignment. Existing techniques have begun to explore incorporating VLMs into object detection or multimodal fusion processes: for example, extracting semantic features from VLMs for channel attention transfer or as fusion guidance signals to improve semantic discrimination and open-vocabulary generalization capabilities. However, this approach of "directly involving VLM features in fusion / inference" often faces two types of engineering constraints: firstly, additional feature extraction overhead is required during the inference stage, increasing computation and latency; secondly, fine-tuning VLMs may lead to catastrophic forgetting, disrupting their original pre-trained semantic manifold and generalization priors.

[0006] Based on the above technological evolution, existing technologies can be summarized as follows: constructing a symmetric RGB-IR dual-stream / multi-stream detection network in the MSOD task, using cross-modal attention or feature splicing for fusion, and further superimposing domain adaptation (such as generative translation or frequency domain adaptation) and VLM semantic features / attention guidance to improve detection performance in low-light scenes. The essence of its fusion can usually still be abstracted as the linear superposition or strong coupling interaction of the two modal representations.

[0007] In the fields of multispectral target detection and visible-infrared (RGB-IR) cross-modal perception, existing technologies generally employ symmetrical two-stream / multi-stream network structures, achieving joint training and fusion inference at the feature layer through cross-modal attention or feature concatenation / overlay. However, this type of "symmetrical fusion" implicitly assumes that RGB and IR are equivalent and complementary in information, which easily leads to the so-called "symmetry fallacy": because the two modalities are naturally heterogeneous in statistical distribution and representational properties, when using symmetrical topology and performing symmetrical joint training, cross-modal gradient conflicts will destroy their respective feature manifolds, turning the originally complementary learning process into mutual interference and feature degradation. At the same time, modern mainstream backbone networks have obvious "RGB-centric" priors. In this context, forcibly inputting texture-sparse thermal infrared tensors into isomorphic backbones will induce manifold mismatch, resulting in a decrease in the separability and stability of the fused features, making it difficult to meet the robust detection requirements in complex scenes.

[0008] Furthermore, when existing technologies attempt to introduce a Vision-Language Model (VLM) as a semantic prior to improve detection semantic capabilities, the aforementioned structural problems are amplified in the form of "semantic space pollution." The document points out from the perspective of distribution drift: RGB input can be naturally projected onto the semantic manifold S defined by the VLM. The original thermal infrared image is in a heterogeneous domain T, and due to the lack of texture information in thermal infrared, the effective intersection of T and S is approximately empty. Under these conditions, if the traditional linear superposition model of symmetric fusion is still used, feature space pollution and increased semantic uncertainty will occur under low-light degradation, leading to instability or decline in detection performance under critical conditions. In addition to structural and representational mismatch, existing adaptive fusion strategies have semantic gaps at the "fusion logic" level: most methods rely on local statistics or low-level heuristic information for "blind fusion," lacking high-level semantic verification and reliability constraints. This makes the fusion process more prone to accumulating modal noise rather than achieving noise suppression in complex backgrounds, ultimately resulting in fusion degradation, manifested as incorrect amplification of background thermal clutter and dilution of the target region response. Especially during the fusion phase, if semantic consistency checks are lacking, background clutter in thermal infrared light and visual ambiguity in visible light will propagate together and couple with each other, causing the network to have the risk of false detection or false detection.

[0009] Furthermore, existing technologies have shortcomings in thermal infrared feature processing and global context modeling. On the one hand, conventional convolutions often struggle to distinguish between high-frequency structural edges and low-frequency thermal radiation halos on thermal infrared inputs, resulting in weakened target boundaries and difficulty in effectively extracting and integrating structural information, thus reducing the recognition capability driven by shape bias. On the other hand, semantic backbones are mostly based on local receptive field structures such as CNNs, which limit their ability to capture long-range dependencies and global semantics. Using self-attention to enhance global modeling often introduces secondary complexity and high computational overhead, hindering real-time deployment. Meanwhile, some VLM fusion routes require additional feature extraction during the inference stage, incurring inference overhead, or triggering catastrophic forgetting during VLM fine-tuning, thereby disrupting their pre-trained semantic manifold and open-vocabulary priors, making it more difficult for the system to maintain semantic stability under limited thermal infrared data conditions.

[0010] In summary, the existing technical problems and defects can be categorized as follows: At the structural level, over-reliance on symmetric topology and symmetric joint training leads to cross-modal gradient conflicts and manifold mismatches, making it difficult to ensure that the fused features are in a stable semantic manifold; at the logical level, the fusion decision lacks high-level semantic verification, making noise more likely to accumulate and propagate; at the implementation level, thermal infrared structures and background thermal radiation are difficult to decouple effectively, and global modeling and the introduction of VLM are often accompanied by high computational costs or semantic forgetting risks, ultimately resulting in engineering defects such as missed detections, false detections, and unstable localization in low-light and complex background scenes. Summary of the Invention

[0011] In view of this, the present invention provides an RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration to solve the problems existing in the background art.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: An RGB-infrared target detection method based on asymmetric dual-path gating and VLM arbitration includes the following steps: Acquire and preprocess multimodal input data in the embodied scenario to generate cross-modal input pairs; An asymmetric topology is constructed based on cross-modal input pairs, and a constraint relationship of "semantic mainstream - structural cue flow - semantic arbitration flow" is established. Infrared modes in multimodal input data are converted into injectable structural cues, and StarFusion star fusion is introduced into semantic mainstream to achieve efficient global modeling; Semantic arbitration is performed using a frozen VLM to generate a semantic consistency graph as an interpretable adjudication signal for gated fusion; By using semantically guided denoising fusion to perform gated fusion, a unified closed loop for nighttime recovery and thermal noise suppression is achieved.

[0013] Optionally, acquire and preprocess multimodal input data in the embodied scenario to generate cross-modal input pairs, specifically including the following steps: Acquiring visible light images With thermal infrared images Based on sensor calibration parameters, spatiotemporal alignment and normalization are performed. A spatial transformation matrix is ​​calculated according to the calibration parameters, and geometric correction is applied to the thermal infrared image to ensure that for any three-dimensional scene point... The projection satisfies ,in and These are the homogeneous coordinates of the point in the visible light and infrared images, respectively; right Perform adaptive histogram equalization and dynamic range stretching; Noise suppression and intensity normalization are performed to obtain spatially aligned and radiation-normalized cross-modal input pairs. .

[0014] Optional, for Noise suppression is performed using a noise suppression operator based on local statistics: ; in For Gaussian kernel, For local variance, This is the inhibition coefficient.

[0015] Optionally, the constraint relationship between "semantic mainstream - structural cue flow - semantic arbitration flow" is achieved by introducing intermodal gradient isolation loss and semantic anchor constraint function, where the semantic mainstream feature is ; the structural cue flow feature is ; and the semantic arbitration flow outputs a semantic consistency graph.

[0016] Optionally, it may also include designing an intermodal gradient isolation loss. : in This represents the layer index in the network. This represents the total number of floors. To detect loss, element-wise multiplication is represented. The standard detection loss includes bounding box regression loss and classification loss; Is the detection loss for the first Mainstream features of layer semantics The partial derivatives, Is the detection loss for the first Layer structure suggests flow characteristics The partial derivatives of .

[0017] Optional, semantic anchor constraint function : ; in For spatial location index, For a lightweight linear transformation, Represents a semantic consistency graph. The representative structure indicates the flow characteristics. This represents a difference in the mainstream semantic features.

[0018] Optionally, the infrared modes in the multimodal input data are converted into injectable structural cues. Specifically, the structure-aware thermal imaging cue uses gradient prior extraction, edge-gated filtering, and learnable cues to convert the original infrared image into injectable structural cues. Convert to structural hints .

[0019] Optionally, StarFusion can be introduced into the semantic mainstream to achieve efficient global modeling. Specifically, this involves applying StarFusion to the preprocessed visible light image within the semantic mainstream. Backbone network features are extracted to obtain basic feature maps. Basic feature map The StarFusion module is introduced to achieve long-range dependency modeling with linear complexity.

[0020] Optionally, semantic arbitration is performed using a frozen VLM to generate a semantic consistency graph as an interpretable adjudication signal for gated fusion, specifically including the following steps: Visual features are extracted from the main semantic features to obtain visual embedding vectors, and text semantic priors are constructed from the list of task-related category words to obtain text embedding matrices. The visual embedding vector and the text embedding matrix are subjected to cross-modal semantic matching to obtain a similarity tensor. A semantic consistency graph is generated based on the similarity tensor, which is the adjudication signal.

[0021] Optionally, by combining the semantic consistency map with the semantic mainstream features and structural cue flow features, an adaptive closed-loop fusion of nighttime recovery and thermal noise suppression can be achieved through a semantically guided denoising fusion module.

[0022] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration, which has the following beneficial effects: 1. By introducing a semantic arbitration flow and a dual-path gating mechanism, rational fusion based on semantic credibility is achieved. The frozen VLM provides cross-modal semantic alignment capabilities, generating a pixel-level semantic consistency map that clearly identifies "which modal information at which locations is more reliable." The nighttime gain recovery path and thermal noise suppression path handle different scenario requirements, ensuring that infrared information is only used as an enhanced signal in areas where semantic verification has passed, and is actively suppressed in low-confidence areas. This interpretable fusion decision-making process not only improves performance but also provides a clear basis for system debugging and failure analysis.

[0023] 2. The StarFusion module in semantic mainstream applications reduces the quadratic complexity of traditional self-attention to linear complexity through a star-shaped interaction of spatial and channel projections. Specifically, spatial projection compresses spatial dimensions through global pooling, while channel projection focuses on the statistical relationships between feature channels. The interaction between the two forms a global context-enhanced feature. This design enables the network to model long-range dependencies while maintaining real-time inference speed (≥15 FPS), playing a crucial role in target discrimination in complex backgrounds and instance separation in dense scenes. Computational analysis shows that at a 1024×1024 input resolution, StarFusion's computational cost is significantly lower than that of traditional self-attention mechanisms.

[0024] 3. By employing gradient isolation loss and semantic anchor constraints, the visible light and infrared branches are guided to learn complementary rather than redundant feature representations. The visible light branch focuses on rich texture, color, and semantic information, forming the mainstream of semantic understanding; the infrared branch specializes in geometric structure and thermal radiation features, serving as a cue stream for structural correction. This asymmetric design avoids the modal competition and gradient conflicts common in symmetric architectures, allowing the network to leverage the advantages of each sensor's physical characteristics. The gradient isolation mechanism during training ensures the independence of the two paths, while the semantic anchor constraints guarantee their alignment at the semantic level, forming a synergistic relationship where the mainstream dominates and the tributaries complement each other. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a diagram of the asymmetric three-flow network structure of the present invention; Figure 3This is a flowchart of the gradient prior extraction, edge gating filtering and learnable cueing steps of the present invention; Figure 4 This is a flowchart of the structure-sensing thermal imaging prompt of the present invention; Figure 5 This is a flowchart of the StarFusion feature enhancement process of the present invention; Figure 6 This is a timing diagram for semantic guidance gating fusion according to the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0028] This invention discloses an RGB-infrared target detection method based on asymmetric dual-path gating and VLM arbitration, such as... Figure 1 As shown, it includes the following steps: Acquire and preprocess multimodal input data in the embodied scenario to generate cross-modal input pairs; An asymmetric topology is constructed based on cross-modal input pairs, and a constraint relationship of "semantic mainstream - structural cue flow - semantic arbitration flow" is established. Infrared modes in multimodal input data are converted into injectable structural cues, and StarFusion star fusion is introduced into semantic mainstream to achieve efficient global modeling; Semantic arbitration is performed using a frozen VLM to generate a semantic consistency graph as an interpretable adjudication signal for gated fusion; By using semantically guided denoising fusion to perform gated fusion, a unified closed loop for nighttime recovery and thermal noise suppression is achieved.

[0029] Specifically, the following steps are included: Step 1: Acquire and preprocess multimodal input data from the embodied scenario to form cross-modal input pairs that can be processed uniformly. During the operation of the embodied intelligent agent (such as a warehouse mobile robot AMR / AGV, inspection robot, etc.), visible light images and thermal infrared images are acquired simultaneously. Based on sensor calibration parameters, time synchronization, geometric registration, and scale alignment are performed to obtain spatially corresponding RGB-IR input pairs. Simultaneously, brightness / dynamic range normalization is performed on the visible light images, and noise suppression and intensity normalization are performed on the thermal infrared images to ensure the stability of subsequent feature extraction and semantic arbitration processes. This step provides a data consistency prerequisite for cross-modal fusion and avoids the amplification of pseudo-structural differences caused by modal misalignment in subsequent fusion.

[0030] Step 2: Construct an asymmetric three-stream topology and establish a constraint relationship of "semantic mainstream - structural cueing stream - semantic arbitration stream". The RGB branch is defined as the semantic mainstream, used to carry the main semantic representation and maintain stable evolution on the RGB-centric pre-trained semantic manifold as much as possible; the IR branch is defined as the structural cueing stream, used to provide controllable structural compensation rather than competing with RGB on an equal footing; and a semantic arbitration stream is introduced, using a frozen VLM as a semantic anchor point to generate pixel-level / region-level semantic consistency constraint signals during the fusion process, used to decide "where infrared structures need to be introduced, how much to introduce, and how to suppress thermal noise". This asymmetric organization aims to avoid gradient conflicts and manifold mismatch risks caused by symmetric fusion, and limits the contribution of IR to "semantically verified structural correction terms".

[0031] Step 3: The infrared modality is converted into an injectable structural cue through a structure-aware thermal imaging cue, reducing cross-modal manifold mismatch. In the structural cue stream, a gradient prior is introduced into the thermal infrared image, and operators such as Sobel are used to extract edge gradients to obtain structural information related to the target contour. Furthermore, edge-gated convolution (EGC) is used to learn gating weights to suppress background thermal clutter and enhance effective edges, thus forming a structural edge map. Subsequently, the structural edge map and the intensity information of the thermal infrared image are synthesized with learnable weights to construct a three-channel pseudo-visual cue (Pt). This allows infrared information to enter the subsequent network in the form of "shape / boundary cue," and the input distribution is closer to the RGB semantic mainstream prior space. The core of this step is to first "decouple structure - suppress thermal noise - re-cue," reducing feature pollution caused by IR directly participating in fusion from the source.

[0032] Step 4: Introduce StarFusion star-shaped fusion into the semantic mainstream to achieve efficient global modeling and enhance long-range semantic stability in embodied scenarios. After extracting the backbone features from the RGB input Iv in SMS, to compensate for the insufficient characterization of long-range dependencies by local convolutions and avoid the high computational overhead caused by self-attention, the StarFusion module is introduced: First, local context is extracted through multi-scale depthwise separable convolutions, and then spatial projection and channel projection are interactively combined in a "star shape" to obtain global semantic enhancement features with linear complexity. This step ensures that the semantic mainstream maintains strong discriminativeness under complex conditions (low light, occlusion, cluttered background), providing a reliable main semantic foundation for subsequent semantic arbitration and gating fusion.

[0033] Step 5: Perform semantic arbitration using a frozen VLM to generate a semantic consistency map as an interpretable adjudication signal for gated fusion. In SAS, the VLM is kept frozen to avoid catastrophic forgetting, and its cross-modal semantic alignment capability is used to assess the semantic credibility of the current scene: visual features are matched with text-based semantic priors to obtain pixel-level or region-level semantic consistency maps (or multi-scale maps), which characterize whether the visual response at this location is consistent with the target semantics. This consistency map does not directly replace the detection features but is injected as an adjudication signal into subsequent fusion decisions, thereby elevating the determination of "whether fusion occurs and its intensity" from empirical local statistical rules to "interpretable arbitration constrained by semantic anchors."

[0034] Step 6: Perform gated fusion through semantically guided denoising fusion to achieve a unified closed loop for nighttime recovery and thermal noise suppression. During the fusion stage, the semantic mainstream feature Fv and the structural cueing flow feature F... t (by P) t (Provide) an input SGDF, and construct a "dual-path gating" mechanism with the semantic consistency map M as the core gating condition: The first is a nighttime recovery path, where when insufficient RGB brightness leads to degradation of the main semantics while M displays high target semantic consistency, an enhancement coefficient α is generated to conditionally gain the structural cues to compensate for target contour and boundary information; the second is a thermal noise suppression path, where when there is "hot but non-target" background interference in the infrared and M displays inconsistent semantics, a suppression coefficient β is generated to attenuate the infrared components, blocking thermal clutter from propagating to the detection features. Finally, F is gating according to the gating rules. v With F t The fusion feature F is obtained by weighted combination of rows. fused The results are then fed into the detection head to output the category and bounding box; thus, "inspectable in low light (structural compensation)" and "controllable thermal interference (noise suppression)" can be achieved simultaneously within the same framework, adapting to the stable detection needs of embodied robots in typical working conditions such as nighttime aisles, shaded areas of shelves, and heated areas of equipment.

[0035] Step 7, Training and Deployment: While freezing the VLM (Visual Model), optimize the trainable modules end-to-end, validating and deploying them in standard benchmarks and embodied scenarios. During training, keep the VLM frozen and only optimize the trainable modules such as STP (Structured Context Optimization), StarFusion, SGDF (Structured Context Optimization), and the detection head. This allows the model to learn a closed-loop strategy of "structured prompting - semantic arbitration - gated fusion" under semantic anchor constraints. Multi-scale feature supervision, detection loss, and consistency constraints can be combined to enable adaptive convergence of the fusion behavior in different scenarios. After training, the model can be evaluated on public RGB-T benchmark datasets such as LLVIP and DroneVehicle using metrics such as mAP, mAP50, parameter count, and inference overhead to verify its robustness and generalization ability in nighttime recovery and complex backgrounds. The model is then integrated into the perception module of the embodied intelligence platform, linking with upper-layer planning and safety strategies to achieve reliable identification and risk warning of pedestrians, forklifts, pallets, and dynamic obstacles. Example 2

[0036] The only difference between this embodiment and Implementation 1 is the following steps: Step 1 involves acquiring visible light images synchronized with the embodied intelligent agent. With thermal infrared images Based on sensor calibration parameters, rigorous spatiotemporal alignment and normalization are performed to provide a geometric and radiometric consistency foundation for subsequent asymmetric fusion. The spatial transformation matrix is ​​calculated according to the calibration parameters, and geometric correction is performed on the thermal infrared image to ensure that for any 3D scene point... Its projection satisfies ,in and These are the homogeneous coordinates of the point in the visible light and infrared images, respectively, to eliminate modal misalignment caused by viewing angle differences. Radiometric normalization is then performed on the registered image pair. Perform adaptive histogram equalization and dynamic range stretching to the [0,1] interval to enhance texture visibility in dark areas; for Then a noise suppression operator based on local statistics is adopted. ,in For Gaussian kernel, For local variance, The suppression coefficient is used to filter out thermal noise and preserve the target thermal radiation structure, ultimately yielding spatially aligned and radiation-normalized cross-modal input pairs. ; Furthermore, in step 2, as Figure 2As shown, the construction of the asymmetric three-stream topology establishes the constraint relationship between the semantic mainstream, the structural cueing stream, and the semantic arbitration stream. This relationship is achieved by introducing intermodal gradient isolation loss and a semantic anchor constraint function. The semantic mainstream feature is defined as extracted from the RGB branch; the structural cueing stream feature is generated from the infrared cueing processed in step 3; and the semantic arbitration stream outputs a semantic consistency map. To avoid gradient conflicts in symmetric fusion, an intermodal gradient isolation loss is designed. : in This represents the layer index in the network. This represents the total number of floors. To detect loss, element-wise multiplication is represented. The standard detection loss includes bounding box regression loss and classification loss; Is the detection loss for the first Mainstream features of layer semantics The partial derivatives, Is the detection loss for the first Layer structure suggests flow characteristics (The H×W×C dimension features generated after step 3 of the IR branch are in the first...) The partial derivatives of the layer output are used to first solve the gradient of the detection loss with respect to the RGB branch features and IR branch features of each layer. The L1 norm is calculated after multiplying the two sets of gradients element by element. Then, the calculation results of all network layers are summed. Finally, the dot product of the gradient vectors of the two modes is minimized through this loss, which suppresses the consistency of gradient direction and forces the RGB and IR branches to learn complementary rather than competitive feature representations during the optimization process, thereby avoiding the gradient conflict problem that is prone to occur in symmetric fusion.

[0037] This loss aims to minimize the dot product of the gradient vectors of the two modes, thereby suppressing the uniformity of gradient directions and forcing the RGB and IR branches to learn complementary rather than competitive feature representations during the optimization process. To constrain the infrared contribution as a semantically validated structural correction term, a semantic anchor constraint function is defined. : in For spatial location index, This is a lightweight linear transformation used to map semantically dominant features to the feature space of structural cueing flows. This constraint ensures that only semantically consistent graphs... The indicated high-confidence region and structural indications suggest flow characteristics. Only then are they allowed to be associated with mainstream semantic features. This generates differences, while in low-confidence regions it forces the characteristics to approximate the transformed mainstream features, thereby suppressing the ineffective injection of thermal noise through joint optimization. and Asymmetric topology can naturally form a division of labor during training, with RGB dominating semantics and IR providing structural compensation.

[0038] In step 3, such as Figure 3 and Figure 4 As shown, the structure-aware thermal imaging cue system uses three consecutive steps—gradient prior extraction, edge-gated filtering, and learnable cue synthesis—to synthesize the original infrared image. Convert to structural hints First, the gradient magnitude map of the infrared image is calculated based on the Sobel operator. : in and Gradient operators are used in the horizontal and vertical directions, respectively. An edge-gated convolution module is introduced into the structure-aware thermal imaging cue. This module uses... For gating signals, Perform feature selection filtering: in This represents the purified structure edge map of the output. In spatial location Pixel value at; middle It is the neighborhood window of the convolution operation. The index for the relative position within the window; These are the learnable convolutional kernel weights of this convolutional module; It is the original thermal infrared image. In the neighborhood Pixel value at; This represents the Sigmoid activation function, which is used to map the input to the [0,1] interval to generate gated weights; and These are learnable scale and bias parameters used to dynamically adjust the gating threshold; Gradient magnitude map calculated by the Sobel operator In position The value at that location is used as a gating signal for the structural edge map. Each spatial location In the corresponding neighborhood window Inside, the convolution kernel weights Pixel values ​​at corresponding locations in thermal infrared images With scriptures After adjustment, the gated values ​​generated by the Sigmoid function are multiplied element-wise, and then all multiplication results within the neighborhood window are summed to obtain the pixel value at that location. This enhances the target edge structure in high-gradient regions while suppressing background thermal clutter in low-gradient regions. This operation enhances the edge structure corresponding to high-gradient regions while suppressing thermal clutter in low-gradient regions, resulting in a purified structure edge map as the output. Finally The original infrared intensity information is adaptively synthesized using learnable weights to generate a three-channel pseudo-visual cue. : Where is the channel-dimensional learnable weight vector. Representation layer normalization, The single-channel infrared intensity is replicated into three channels. This synthesis process approximates the statistical characteristics of RGB images in terms of channel distribution, thereby reducing modal manifold differences at the input level. The entire process achieves a deterministic mapping from raw thermal radiation to structural cues, providing complementary inputs that are geometrically enhanced and noise-suppressed for subsequent semantic mainstreaming. like Figure 5 As shown in step 4, the preprocessed visible light image is processed in the semantic mainstream. Backbone network features are extracted to obtain basic feature maps. Basic feature map The StarFusion module is introduced to achieve long-range dependency modeling with linear complexity. This module first... Multi-scale depthwise separable convolutions are performed to extract local multi-scale contextual features, resulting in... A set of scale features Then, global information is integrated through a star-shaped interaction between spatial projection and channel projection: spatial projection compresses the feature vector of each position into spatial weights along the channel dimension, while channel projection diffuses global channel statistics into the spatial domain. The two are connected through a star-shaped interaction function. To merge: in For spatial projection, Indicates global average pooling. For channel projection, This is achieved by adding element-wise data followed by a gated linear unit. This setup allows each location to aggregate global spatial context and channel attention, while maintaining a computational complexity of O(log n). Far lower than self-attention Output features It not only preserves local details, but also enhances the semantic discriminative power between the target and the complex background; Step 5 utilizes the cross-modal semantic alignment capability of the frozen VLM to complete the matching between visual features and textual semantic priors, generating an interpretable semantic consistency map as the decision signal for subsequent fusion decisions. The specific execution logic of this step proceeds hierarchically according to the input, processing, and output chain, and the detailed correspondence of each step is shown in Table 1: Table 1 like Figure 6 As shown, step 6 is based on the pixel-level semantic consistency map generated in step 5. Combining mainstream semantic features With structural hints flow characteristics The adaptive closed-loop fusion of nighttime restoration and thermal noise suppression is achieved through a semantically guided denoising fusion (SGDF) module. This process is carried out in three progressively and systematically phased stages, including: Step 6-1, Generation of gating coefficients guided by semantic consistency graph, based on semantic consistency graph. With semantic mainstream features The brightness statistics are used to generate the nighttime recovery gain coefficient. With thermal noise suppression coefficient Define the luminance mask ,but: in For learnable weights, Here, represents the semantic consistency threshold, and represents element-wise multiplication. In regions with low brightness and high semantic consistency, the value approaches 1. It approaches 1 in regions of low semantic consistency; Step 6-2: Dual-path gating fusion execution, respectively processing the structural prompt flow characteristics. Perform gain and suppression: Combine both with mainstream semantic features Merge according to gating rules: in For adaptive balancing weights; Step 6-3: Fuse feature normalization and output, for After performing layer normalization and residual connection to ensure numerical stability, the sample is fed into the detection head.

[0039] Step 7, the training phase, employs an end-to-end optimization strategy, with the total loss function defined as: in The standard detection loss includes bounding box regression and classification losses; and The gradient isolation loss and semantic anchor constraint loss defined in step 2 are used respectively to ensure stable learning of asymmetric topology; The semantic consistency constraint loss is calculated using the following formula: where This is a lightweight projection function that encourages semantically dominant features corresponding to regions with high semantic consistency to have higher response strength, thereby strengthening the synergy between semantic arbitration flow and dominant features. All loss weights during the optimization process... It was determined through grid search that the frozen VLM parameters do not participate in gradient updates; After training, the model was rigorously evaluated on benchmarks such as LLVIP and DroneVehicle. In addition to mAP and mAP50, a cross-modal fusion quality metric was introduced: L This metric measures both the accuracy of the detection box and the confidence level of semantic consistency to comprehensively evaluate the effectiveness of the fusion. The quantized model is deployed to the embedded platform of the embodied agent via TensorRT or ONNX format. It runs at an inference speed of no less than 15 FPS in the real-time perception pipeline of the inspection robot or warehouse AMR. It communicates with the global planner and safety monitoring module in ROS2 via Topic to achieve continuous identification and risk level classification of dynamic obstacles.

[0040] To verify the adaptability and performance gain of the dual-path gating mechanism in the Semantic Guided Denoising and Fusion Module (SGDF), a targeted experiment was designed: using the confidence level of the semantic consistency graph ( Value) and semantic mainstream feature brightness mask ( Using the value as the core criterion, typical scenario intervals were selected, and the nighttime recovery gain coefficient within each interval was statistically analyzed. With thermal noise suppression coefficient The mean value was used to clarify the activation logic and parameter adaptation rules of dual-path gating in different scenarios. At the same time, the contribution of the corresponding fusion decision type to the model detection mAP was quantified, and the effectiveness of the strong infrared enhancement strategy in high semantic confidence and low brightness scenarios was intuitively verified, providing experimental support for the rationality of the dual-path gating mechanism. The specific experimental data are shown in Table 2. Table 2 The data in Table 2 show that in high-confidence infrared region scenarios where the semantic consistency map confidence M>0.8 and the semantic mainstream feature luminance mask B<0.3, the mean night recovery gain coefficient α of the semantically guided denoising fusion module (SGDF) reaches 0.85±0.10, and the mean thermal noise suppression coefficient β is only 0.12±0.08 (at a low suppression level). This corresponds to activating the strong infrared enhancement fusion decision, which ultimately brings a significant performance contribution of +0.24 to the model detection mAP. This intuitively verifies the effectiveness of compensating for the RGB semantic degradation of the target contour under low illumination by infrared structural information in this scenario, and confirms the adaptability and performance gain value of the dual-path gating mechanism in low-illumination, high-semantic-confidence scenarios.

[0041] To verify the impact of different VLM parameter configurations on the key performance of the RGB-infrared cross-modal target detection model and address the engineering constraints in existing technologies, such as catastrophic forgetting, increased computational overhead, and decreased generalization ability across datasets, a comparative experiment of frozen and fine-tuned VLM parameters was designed. Four typical configurations were selected: frozen VLM (this scheme), full parameter fine-tuning, adapter fine-tuning, and projection layer-only training. A quantitative comparison was conducted from four core dimensions: semantic consistency accuracy, cross-dataset generalization ability, computational overhead, and adaptability to downstream tasks. Simultaneously, multimodal feature space alignment verification was added to clarify the optimal VLM configuration scheme, providing experimental support for the model to balance performance, efficiency, and generalization ability while maintaining semantic prior stability. Specific experimental data are shown in Table 3. Table 3 In the comparison of VLM parameter configurations, although the semantic consistency accuracy of the frozen VLM configuration of this invention is slightly lower than that of full parameter fine-tuning, it has the smallest decrease in cross-dataset generalization ability, the lowest computational cost, and good adaptability to downstream tasks. It effectively avoids the overfitting and high computational cost problems of full parameter fine-tuning and the generalization decay of adapter fine-tuning. In the multimodal feature space alignment verification, the arbitration fusion of this scheme is in the theoretical optimal range in terms of feature cosine similarity between modalities and feature space manifold distance. The gradient conflict coefficient is lower than the training stability threshold, and the semantic consistency contribution meets the standard. It is significantly better than the feature redundancy / severe conflict of early fusion and the feature separation / moderate conflict of late fusion. Overall, it confirms that the design of frozen VLM + arbitration fusion of this scheme takes into account semantic reliability, generalization ability and deployment efficiency, and achieves efficient collaboration and stable training of modal features.

[0042] To verify the core advantages of the asymmetric three-flow topology and semantic arbitration mechanism of this invention at the multimodal feature collaboration level, and to solve the problem of modal feature alignment imbalance caused by redundancy due to excessive feature proximity in traditional early fusion and conflict caused by excessive feature separation in late fusion, a multimodal feature space alignment verification experiment was designed. Four key indicators were selected: intermodal feature cosine similarity, feature space manifold distance, gradient conflict coefficient, and semantic consistency contribution. Intermodal feature cosine similarity is used to measure feature complementarity, feature space manifold distance is used to evaluate modal adaptability, gradient conflict coefficient is used to reflect training stability, and semantic consistency contribution is used to reflect semantic collaboration value. The arbitration fusion of this scheme is compared horizontally with traditional early and late fusion, and the theoretical optimal value is introduced as a reference benchmark. The aim is to quantitatively verify whether this scheme can guide RGB and IR modal features to be in the optimal collaboration range of moderate difference and effective complementarity through asymmetric topological constraints and semantic arbitration, alleviate gradient conflict and manifold mismatch, ensure the rationality and stability of feature space alignment, and lay the underlying feature foundation for the improvement of subsequent fusion performance. The specific experimental data are shown in Table 4. Table 4 As shown in the table above, the intermodal feature cosine similarity of traditional early-stage fusion (0.85±0.08) exceeds the theoretical optimal range (0.70-0.75), resulting in redundancy due to excessive feature similarity. Simultaneously, the feature space manifold distance is excessively close (1.2±0.3), the gradient conflict coefficient reaches 0.45±0.10, and there is no contribution to semantic consistency. Conversely, the intermodal feature cosine similarity of traditional late-stage fusion (0.62±0.12) is below the theoretical optimal range, causing conflict due to excessive feature separation. The feature space manifold distance is excessively dispersed (3.8±0.7), and the gradient conflict coefficient is 0.25±0.08 (moderate conflict), also lacking semantic consistency. The traditional fusion method lacks semantic consistency contribution; however, the arbitration fusion method of this scheme is close to the theoretical optimal value in all indicators: the cosine similarity of features between modalities (0.73±0.09) is in the optimal cooperative range of moderate difference and complementarity, the manifold distance in the feature space (2.1±0.4) maintains a reasonable fusion distance, the gradient conflict coefficient (0.15±0.05) is lower than the training stability threshold (<0.20) to achieve low conflict, and the semantic consistency contribution (0.68±0.07) also exceeds the significant improvement threshold (>0.65). This fully verifies that the asymmetric three-stream topology and semantic arbitration mechanism can effectively alleviate the feature redundancy or separation problem of traditional fusion, so that the RGB and IR modal features are in a cooperative state of moderate difference and effective complementarity, laying a solid foundation of underlying features for improving fusion performance.

[0043] Furthermore, this embodiment addresses common issues in embodied interactive environments such as low illumination, strong occlusion, thermal noise, and dynamic structural changes. Existing symmetric fusion paradigms often assume that visible light (RGB) and thermal infrared (IR) are equally complementary in the feature space, leading to gradient conflicts and representation contamination during end-to-end training. This causes detection features to deviate from the semantic manifold represented by large-scale pre-trained models. Especially when a visual language model is introduced as a high-level semantic prior, there is a significant "manifold mismatch" and semantic gap between the RGB-centric pre-trained semantic space and the IR modality. Without effective arbitration and constraint mechanisms, cross-modal alignment failures or noise propagation can easily occur. To address this, this invention proposes an asymmetric three-stream topology fusion framework: RGB is used as the semantic main stream (Semantic Main Stream, SMS) to maintain the stability of the main semantics; IR is used as the structural prompt stream (Structural Prompt Stream, SPS) for controllable compensation; and a semantic arbitration stream (SAS) based on a frozen VLM is introduced as a semantic anchor. The fusion chain explicitly constrains "where to fuse, how much to fuse, and how to fuse," thereby mitigating cross-modal mismatch and feature contamination problems. Regarding the fusion strategy, this invention adheres to the basic principle of "structural prompting rather than peer-to-peer fusion," constructing a Structure-aware Thermal Prompter (STP) and a Semantic-Guided Denoising Fusion (SGDF) module to achieve interpretable injection of IR information and noise suppression. Specifically, STP introduces gradient priors, extracting edge gradients from thermal infrared images using operators such as Sobel, and combining this with edge-gated convolution (EGC) to form learnable gating weights. This suppresses background thermal noise and highlights structural components related to the target contour during preprocessing. Furthermore, it resynthesizes the structural edges with the original thermal intensity to generate pseudo-visual structural cues that more closely approximate the RGB input distribution, reducing the mismatch risk caused by IR directly entering the RGB-centric semantic manifold. Building on this, SGDF utilizes the pixel-level semantic consistency of the frozen VLM output as a fusion arbitration signal to construct a gating fusion mechanism: when the scene is in low illumination and IR and semantic consistency are high, a "night recovery" gain path is triggered to enhance the structural cues; when "hot but not target" interference areas appear and semantic consistency is low, a "thermal noise suppression" suppression path is triggered to attenuate IR components that may introduce false detections.Therefore, this invention achieves conditional injection of IR structural information and controllable noise propagation while maintaining the stability of the RGB semantic backbone, thus meeting the patent's requirements of "controllable, interpretable, and deployable" fusion.

[0044] To enhance cross-scale and long-range dependency modeling capabilities without significantly increasing computational complexity, this invention further introduces the efficient global modeling component StarFusion (Star-shapedFusion) for global context enhancement of mainstream semantic features, forming a collaborative constraint with VLM semantic arbitration. StarFusion extracts local context through multi-scale depthwise separable convolutions, then interactively combines spatial and channel projections in a "star-shaped" pattern to approximate the broadcasting and aggregation of higher-order global dependencies, achieving low-complexity global semantic enhancement. Under the constraint of Semantic Arbitration Flow (SAS), this mechanism can selectively suppress cross-modal redundancy and conflicting information: on the one hand, it avoids IR noise being amplified and interfering with the main semantic representation during global modeling; on the other hand, it injects structural cues only in the "semantically trustworthy region" through consistency signals, thus forming a closed loop of "semantic backbone steady state + structural cue gain + semantic arbitration decision". Therefore, this invention achieves an efficient, robust, and scalable cross-modal fusion architecture for embodied awareness, balancing accuracy improvement and engineering deployability.

[0045] This invention is particularly applicable to safety and task-oriented detection scenarios in embodied perception, such as pedestrian / forklift / obstacle recognition in warehouse mobile robots (AMR / AGV) during night shifts or in low-light aisles, and false detection suppression and risk warning in shaded areas of shelves or under heat source interference at loading docks. In the above scenarios, the "night recovery" path can compensate for the target contour using IR structural cues when RGB semantics degrade, thereby improving low-light detectability; the "thermal noise suppression" path can suppress non-target thermal anomalies under interference conditions caused by equipment heating, residual sunlight, or local hot spots, enhancing system safety and stability. To verify the effectiveness and generalization ability of this invention, it can be evaluated on publicly available RGB-T target detection benchmark datasets, such as LLVIP and DroneVehicle, using detection metrics such as mAP and mAP50, as well as engineering metrics such as parameter quantity and inference latency as evaluation criteria. The experimental results can demonstrate the advantages of this invention in cross-modal manifold consistency, noise robustness, and multi-scenario generalization ability, thereby providing a reusable technical foundation for upgrading the perception module of the embodied intelligence platform, ensuring safety during nighttime operations, and ensuring reliable system-level operation.

[0046] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for RGB-infrared target detection using asymmetric dual-path gating and VLM arbitration, characterized in that, Includes the following steps: Acquire and preprocess multimodal input data in the embodied scenario to generate cross-modal input pairs; An asymmetric topology is constructed based on cross-modal input pairs, and a constraint relationship of "semantic mainstream - structural cue flow - semantic arbitration flow" is established. Infrared modalities in multimodal input data are transformed into injectable structural cues. StarFusion is introduced into the semantic mainstream to achieve efficient global modeling. Specifically, this involves: converting infrared modalities into injectable structural cues using a structure-aware thermal imaging cue to reduce cross-modal manifold mismatch; introducing gradient priors into the thermal infrared image within the structural cue stream; extracting edge gradients using operators such as Sobel to obtain structural information related to the target contour; and further learning gating weights through edge-gated convolution to suppress background thermal clutter and enhance effective edges, thus forming a structural edge map; and synthesizing the structural edge map and the intensity information of the thermal infrared image using learnable weights to construct a three-channel pseudo-visual cue. This allows infrared information to enter the subsequent network in the form of "shape / boundary cues" and to be closer to the prior space of the mainstream RGB semantics at the input distribution level; Semantic arbitration is performed using a frozen VLM to generate a semantic consistency graph as an interpretable adjudication signal for gated fusion; By using semantically guided denoising fusion to perform gated fusion, a unified closed loop for nighttime recovery and thermal noise suppression is achieved.

2. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 1, characterized in that, Acquire and preprocess multimodal input data in the embodied scenario to generate cross-modal input pairs, specifically including the following steps: Acquiring visible light images With thermal infrared images Based on the sensor calibration parameters, spatiotemporal alignment and normalization are performed, and the spatial transformation matrix is ​​calculated according to the calibration parameters. Geometric correction of thermal infrared images enables the processing of any three-dimensional scene point. The projection satisfies ,in and These are the homogeneous coordinates of the point in the visible light and infrared images, respectively; right Perform adaptive histogram equalization and dynamic range stretching; Noise suppression and intensity normalization are performed to obtain spatially aligned and radiation-normalized cross-modal input pairs. .

3. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 2, characterized in that, right Noise suppression is performed using a noise suppression operator based on local statistics: ； in For Gaussian kernel, For local variance, This is the inhibition coefficient.

4. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 1, characterized in that, The constraint relationship of "semantic mainstream - structural cue flow - semantic arbitration flow" is achieved by introducing intermodal gradient isolation loss and semantic anchor constraint function, where the semantic mainstream feature is... The structural hints at the flow characteristics as follows: Semantic arbitration flow outputs a semantic consistency graph. .

5. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 4, characterized in that, Semantic anchor constraint function : ； in For spatial location index, For a lightweight linear transformation, Represents a semantic consistency graph. The representative structure indicates the flow characteristics. This represents a difference in the mainstream semantic features.

6. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 1, characterized in that, Introducing StarFusion into the semantic mainstream to achieve efficient global modeling, specifically: In the semantic mainstream, preprocessed visible light images... Backbone network features are extracted to obtain basic feature maps. Basic feature map The StarFusion module is introduced to achieve long-range dependency modeling with linear complexity.

7. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 1, characterized in that, Semantic arbitration is performed using a frozen VLM to generate a semantic consistency graph as an interpretable adjudication signal for gated fusion. This process includes the following steps: Visual features are extracted from the main semantic features to obtain visual embedding vectors, and text semantic priors are constructed from the list of task-related category words to obtain text embedding matrices. The visual embedding vector and the text embedding matrix are subjected to cross-modal semantic matching to obtain a similarity tensor. A semantic consistency graph is generated based on the similarity tensor, which is the adjudication signal.

8. The RGB-infrared target detection method with asymmetric dual-path gating and VLM arbitration according to claim 1, characterized in that, By combining semantic consistency graphs, semantic mainstream features, and structural cue flow features, an adaptive closed-loop fusion of nighttime recovery and thermal noise suppression is achieved through a semantically guided denoising fusion module.

Images