An Infrared-Visible Image Fusion Method and System with Embodied Interactive Intent Gating
By generating structured embodied intent states and adaptive gating weights, the lag problem of fusion region and weight adjustment in existing infrared and visible light image fusion methods is solved, realizing operator intent-driven dynamic fusion and improving the stability and effect of fused images.
Patent Information
- Application Number
- CN202611043815.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing infrared and visible light image fusion methods lack a direct response mechanism to the operator's real-time focus objects, focus areas, and task intentions, making it difficult to dynamically adjust the fusion area and weights. Furthermore, erroneous intent-driven and target area mismatches are prone to occur during multimodal interaction.
By acquiring infrared and visible light images and operator embodied interaction data in the same scene, modal features and confidence levels are generated, structured embodied intent states are updated, and regional selective fusion of target energy features and background texture features is achieved by utilizing dense spatial intent priors and adaptive gating weights.
It enables dynamic adjustment of the fusion result according to the operator's real-time task requirements, improves the stability and usability of the fused image, avoids abrupt changes in fusion weights and region mismatch, and enhances the saliency of the target and the preservation of background texture.
Smart Images

Figure CN123114247A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of infrared and visible light image fusion technology, specifically to an infrared and visible light image fusion method and system with embodied interactive intent gating, which can be used in transparent smart cockpits, augmented reality, remote operation, security monitoring, and complex environment situational awareness scenarios. Background Technology
[0002] With the development of intelligent sensing devices and human-computer interaction technologies, complex task scenarios often require the simultaneous processing of multiple modal data, including images, voice, text, target detection results, target tracking results, and environmental status information. Multimodal data fusion can leverage the complementary advantages of different information sources to improve target perception, environmental understanding, risk identification, and situational awareness capabilities.
[0003] Existing infrared and visible light image fusion methods typically rely on the complementarity of sensor data itself, generating fused images through pixel weighting, feature stitching, multi-scale transformation, attention mechanisms, cross-modal alignment, or deep network reconstruction. These methods can, to some extent, balance thermally salient targets in infrared images with textural details in visible light images. However, most fusion rules are relatively fixed during the inference phase, usually automatically determining fusion weights based on image intensity, gradient, saliency, or semantic probability, lacking a direct response mechanism to the operator's real-time focus on objects, regions of interest, and task intent.
[0004] In human-machine collaborative scenarios such as transparent intelligent cockpits, remote operation of unmanned systems, and emergency search and rescue, operators often do not need to perform balanced fusion of the entire image. Instead, they expect the system to enhance specific targets, preserve background textures, or highlight risk areas based on their current gaze, voice commands, or corrective feedback. For example, when an operator gazes at a suspected target in the distance and issues the command "enhance this target," the fusion system should enhance the infrared thermal saliency of the target's core area, preserve the visible light texture in the background area, and maintain a continuous transition in the target's boundary area. When the operator provides negative or corrective feedback, the fusion system should also update the focus and fusion area in a timely manner.
[0005] While existing multimodal human-computer interaction methods can infer operator intent using information such as voice, gaze, and images, most methods remain at the level of target selection, command recognition, or interface interaction, failing to further transform the operator's structured intent state into semantic priors, region segmentation, and gating weights in the infrared and visible light image fusion process. Consequently, even when existing image fusion methods introduce semantic masks or attention mechanisms, they typically lack an intent state update mechanism driven by operator feedback loops, making it difficult to dynamically change the fusion strategy for target regions, boundary transition regions, and background regions during interaction.
[0006] Furthermore, during actual interaction, interaction modalities such as voice, gaze, and image may experience missing elements, noise, occlusion, delays, or decreased confidence. If the fusion system cannot determine the availability and assess the confidence of the currently available modalities, it is prone to problems such as erroneous intent-driven behavior, target region mismatch, or abrupt changes in fusion weights, thereby reducing the stability and usability of the fused image.
[0007] Therefore, there is an urgent need for an infrared and visible light image fusion method and system with embodied interactive intent gating, which can transform the operator's multimodal interactive behavior into a computable and updatable structured embodied intent state, and further apply this intent state to semantic prior generation, multidimensional physical prior tensor construction, core domain / transition ring / background region division, and spatial adaptive gating weight generation, so that the fusion result can be dynamically adjusted according to the operator's real-time task requirements. Summary of the Invention
[0008] In view of the problems existing in the prior art, the present invention discloses an infrared and visible light image fusion method with embodied interactive intent gating, comprising the following steps:
[0009] Step 1: Acquire infrared images of the same scene that have undergone temporal synchronization and spatial registration. With visible light images And perform normalization processing; at the same time, acquire the concrete interaction data generated by the operator during the interaction process;
[0010] Step 2: Process the embodied interactive data and infrared images. Visible light images Modality encoding is performed on the image-associated perception results to generate modality features, modality availability identifiers, and modality confidence scores. Based on these modality features, modality availability identifiers, and modality confidence scores, the current fusion intent features are generated. ;
[0011] Step 3: Based on the structured embodied intention state of the previous moment Current fusion intent characteristics Modal availability identifier Modal confidence and operator feedback Generate the current structured embodied intent state ; wherein, the current structured embodied intent state Generate using the following state update function: ;
[0012] Step 4: Based on the current structured embodied intent state Generate semantic cues to describe the current task objective. and spatial cues for constraining regions to be enhanced, maintained, or suppressed. and the semantic prompt and space prompts Transformation into dense space intention prior ;
[0013] Step 5: Process the infrared images respectively. With visible light images Multi-scale feature encoding is performed to obtain infrared coded features and visible light coded features. Shared feature aggregation is then performed on the infrared coded features and visible light coded features, based on the dense spatial intent prior. Semantic injection is performed on the aggregated shared fusion features to obtain potential features for intent enhancement.
[0014] Step 6: Input the intended enhancement latent features into the dual-head orthogonal decoupling module to obtain target energy features for characterizing the thermal saliency of infrared targets and background texture features for characterizing visible light texture details, respectively;
[0015] Step 7: Based on the aforementioned dense spatial intent prior. Generate uncertainty graph and boundary transition clues According to the normalized infrared intensity map With normalized visible light intensity map Intensity differences and gradient differences generate modal difference cues The dense spatial intention is a priori. Uncertainty diagram Boundary transition clues Gradient difference generates modal difference cues Normalized infrared intensity map Normalized visible light intensity map Concatenation forms a gated input tensor The spatial adaptive gating weights are generated by the gating network in conjunction with temperature regulation parameters. ;
[0016] Step 8: Based on the spatial adaptive gating weights The target energy features and background texture features are selectively fused to generate target enhancement results. Preserve the result with texture And based on the spatial adaptive gating weights Enhancement results for the target Preserve the result with texture Regional selection fusion is performed to obtain a fused image that matches the operator's current intention. The fused image is output through a transparent display screen, AR-HUD, augmented reality display terminal, human-machine collaborative control console, unmanned system remote operation terminal, vehicle cockpit, robot inspection terminal, security monitoring terminal, or emergency search and rescue terminal, and the operator's subsequent feedback is used for the next round of structured embodied intent state update.
[0017] As a preferred embodiment of the present invention, the embodied interaction data includes at least one of voice commands, task text, operator feedback, gaze information, gesture area, touch position, and head posture information; the image association perception result includes at least one of target detection result, target segmentation result, target tracking result, risk area result, target category probability map, and target candidate box; the modality availability identifier is used to indicate whether the corresponding interaction modality exists, is valid, or participates in the current intent understanding. When any interaction modality is missing, occluded, has excessive noise, excessive latency, fails to acquire, fails to recognize, or has a confidence level lower than a preset threshold, the interaction modality is marked as unavailable or low-weight available.
[0018] As a preferred embodiment of the present invention, the current structured embodied intent state It should at least include the object of interest, the region of interest, the target mask, the fusion mode, and the intent confidence; the operator feedback The system confirms, denies, or corrects feedback for the operator. When the confirmed feedback is valid, it increases the confidence of the intent of the current object of interest, the area of interest, and the fusion mode. When the denial or corrected feedback is valid, it reduces the matching probability of the original object of interest and updates the area of interest and the target mask.
[0019] As a preferred embodiment of the present invention, when the embodied interaction data includes gaze information, the operator's gaze is mapped to the display coordinate system, image coordinate system, spatial coordinate system or sensor coordinate system to obtain the gaze point or gaze area, and the object of attention or the area of attention is determined based on the spatial overlap, center distance, target detection confidence and historical attention consistency between the gaze point or gaze area and the target candidate box, target segmentation area, target tracking area or risk area.
[0020] The matching probability between the i-th candidate target and the current operator's intention is expressed as:
[0021] ;
[0022] in, Indicates the area of focus. Indicates the first One candidate target region Indicates the point of fixation. Indicates the candidate target center, This represents the confidence level of object detection. This indicates a focus on consistency in historical perspectives. , , , and These are preset or learnable parameters.
[0023] As a preferred embodiment of the present invention, the dense space intention is priori. The method used to distinguish between the target core area, background area, and boundary transition area is as follows:
[0024] ;
[0025] in, Indicates semantic hints, Indicates a spatial hint, Indicates the intent parsing operator, This indicates that the parsed initial intent region will be scaled and aligned to the fusion feature scale. The semantic prompts The spatial prompts are obtained by parsing voice commands, task text, or operator feedback. It is generated from at least one of the following: gaze point, gaze region, gesture region, touch location, target candidate box, target segmentation region, or historical tracking region.
[0026] As a preferred embodiment of the present invention, the uncertainty graph and boundary transition clues They are represented as follows:
[0027] ;
[0028] ;
[0029] in, To prevent abnormally positive numbers in logarithmic calculations, This represents the spatial gradient operator.
[0030] As a preferred embodiment of the present invention, the modal difference clue Gated input tensors Spatial adaptive gating weights and fused images They are represented as follows:
[0031] ;
[0032] ;
[0033] ;
[0034] ;
[0035] in, This represents the normalized infrared intensity map. This represents a normalized visible light intensity map. This represents the gradient difference weight coefficient. This indicates a channel splicing operation. Indicates a gating network. This represents the activation function. Indicates temperature control parameters. This represents element-wise multiplication. The spatial gating constraint diagram is represented as follows:
[0036] ;
[0037] in, Indicates the spatial gating constraint coefficient. This indicates a morphological dilation operation.
[0038] As a preferred embodiment of the present invention, the target enhancement result The texture is reconstructed from the target energy features and constrained by a normalized infrared intensity map or an infrared target saliency map; the texture preservation result... It is reconstructed from background texture features and is constrained by visible light brightness, gradient, or texture structure.
[0039] An infrared-visible image fusion system with embodied interactive intent gating includes: a multimodal data acquisition module, an embodied interactive data acquisition module, a variable modal intent understanding module, a structured embodied intent state update module, an intent-driven multimodal fusion module, an interactive feedback module, and an interactive terminal;
[0040] The multimodal data acquisition module includes an image data acquisition unit, an acoustic data acquisition unit, a text data access unit, a structured perception result access unit, and a positioning and environmental perception data acquisition unit. The image data acquisition unit is used to acquire infrared images and visible light images that have been time-synchronized and spatially registered in the same scene, and to normalize the infrared images and visible light images to obtain normalized infrared intensity maps and normalized visible light brightness maps.
[0041] The embodied interaction data acquisition module is used to acquire at least one of semantic interaction data and spatial interaction data generated by the operator during the interaction process;
[0042] The variable modal intent understanding module is used to perform modal encoding on the currently available embodied interaction data and image association perception results to generate modal features, modal availability identifiers, modal confidence, and current fused intent features;
[0043] The structured embodied intent state update module is used to generate or update the current structured embodied intent state based on the previous structured embodied intent state, the current fused intent features, modal availability identifier, modal confidence, and operator feedback information; the structured embodied intent state includes at least one of intent type, object of interest, region of interest, target mask, fusion mode, task priority, output method, and intent confidence.
[0044] The intent-driven multimodal fusion module is used to generate semantic prior information, fusion mode control quantity and multidimensional physical prior tensor according to the structured embodied intent state. It further reorganizes the target energy features and background texture features through the regional gating mechanism of core domain, transition ring and background area to obtain fused image, enhanced display result, target prompt information, risk prompt information or control command.
[0045] The interactive feedback module is used to output the fused image or enhanced display result, and to provide the modality coding module or the structured embodied intent state update module with the subsequent embodied interaction data generated by the operator based on the fused image or enhanced display result.
[0046] An electronic device according to the present invention includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described embodied interactive intent-gated infrared and visible light image fusion method.
[0047] The beneficial effects of this invention are as follows: This invention unifies currently available semantic interaction data, spatial interaction data, image association perception results, modal availability identifiers, modal confidence, and operator feedback information into a structured embodied intent state, and further parses it into semantic cues, spatial cues, and dense spatial intent priors. This allows changes in the operator's focus object or region of interest to be converted into spatially computable fusion control quantities. Therefore, the spatially adaptive gating weights can be updated according to changes in the focus object, region of interest, and fusion mode, solving the problem in existing infrared and visible light image fusion methods where fusion weights cannot be adjusted in real time or are prone to region mismatch when the operator's focus target changes.
[0048] This method uses a dense spatial intent prior, uncertainty map, boundary transition cues, modal difference cues, normalized infrared intensity map, and normalized visible light brightness map to construct a gated input tensor. Temperature-adjusted parameters control the steepness of the spatial adaptive gating weights and the smoothness of the transition. This increases the contribution of target energy features in the target core domain, increases the contribution of background texture features in the background region, and performs continuous weighted fusion based on uncertainty and modal differences in the transition zone. This addresses the problems of infrared thermal response diffusion into the background at the target boundary, visible light texture overlaying the target's thermal saliency, and hard boundary switching.
[0049] This method uses modal availability identifiers and modal confidence scores to determine the validity of speech, gaze, gesture, touch, task text, infrared images, visible light images, and image-related perception results. When some interaction modalities are missing or have low confidence, it retains the high-confidence fields from the previous time step's structured embodied intent state, only making local corrections to the low-confidence fields. This avoids abrupt changes in fusion patterns and gating weights due to missing single interaction modalities, occlusion, excessive noise, excessive latency, or recognition failure, thus improving the fusion stability during continuous interactions.
[0050] This method transforms multimodal embodied interaction data into a spatial intent prior that can be computed, updated, and participated in the gating control of image fusion networks. It combines uncertainty, boundary transition, and modal differences to achieve regionalized selection fusion of the target core domain, transition ring, and background area, thereby enhancing the thermal saliency of infrared targets while maintaining the continuity of visible light background texture and target boundaries. Attached Figure Description
[0051] Figure 1 This is a schematic diagram of the overall architecture of the infrared and visible light image fusion system with embodied interactive intent gating according to the present invention.
[0052] Figure 2 This is a schematic diagram of the hardware composition of the infrared and visible light image fusion system of the present invention;
[0053] Figure 3 This is a flowchart of the infrared and visible light image fusion method with embodied interactive intent gating according to the present invention;
[0054] Figure 4 This is a schematic diagram of the variable modality embodied intent understanding and intent state update process of the present invention;
[0055] Figure 5 This is a schematic diagram illustrating the operator-focused region mapping process of this invention;
[0056] Figure 6 This is a schematic diagram illustrating the process by which the present invention drives the fusion of infrared and visible light images;
[0057] Figure 7 This is a schematic diagram of the embodied interactive closed-loop process of the present invention;
[0058] Figure 8 This is a schematic diagram of the spatial adaptive gating and regional fusion mechanism of the present invention;
[0059] Figure 9 This is a schematic diagram illustrating the qualitative fusion results of the method of the present invention and several comparative infrared and visible light image fusion methods in representative scenarios.
[0060] Figure 10 This is a schematic diagram of the fusion results of the method of the present invention under different interactive intent conditions;
[0061] Figure 11 This is a schematic diagram illustrating the verification results of the fusion results generated by the method of the present invention for downstream target detection tasks.
[0062] In the diagram: 100 - Multimodal data acquisition module, 110 - Image data acquisition unit, 120 - Acoustic data acquisition unit, 130 - Text / task data access unit, 140 - Structured perception result access unit, 150 - Localization and environmental perception data acquisition unit, 200 - Variable modal intent understanding module, 210 - Modal-specific encoding unit, 220 - Modal type identifier generation unit, 230 - Modal availability judgment unit, 240 - Modal confidence assessment unit, 250 - Variable modal fusion unit, 260 - Intent decoding unit, 300 - Structured embodied intent state update module, 400 - Intent-driven multimodal fusion module, 500 - Interactive feedback module, 510 - Display / enhanced display unit, 520 - Prompt / alarm unit, 530 - Voice / haptic feedback unit, 540 - Control command output unit, 600 - Operator, 700 - Interactive terminal, 710 - Transparent display / AR-HUD 720 - Eye tracking unit, 730 - Voice acquisition unit, 740 - Head posture / IMU unit, 750 - Gesture / touch / button unit, 760 - Cockpit camera / text input unit. Detailed Implementation
[0063] Example 1
[0064] This invention relates to an infrared and visible light image fusion method and system with embodied interactive intent gating. It addresses the problem that existing infrared and visible light image fusion methods struggle to dynamically adjust the fusion region, fusion weights, and output methods based on the operator's real-time focus on the target object, region of interest, and feedback. This invention generates a structured embodied intent state using currently available voice, gaze / attention, and image data. This structured embodied intent state is then transformed into semantic prior information, fusion mode control variables, and multidimensional physical prior tensors. This further controls the contribution ratio of infrared target energy features and visible light background texture features in the core domain, transition zone, and background region, thereby achieving interactive, regionalized, and closed-loop image fusion tailored to the operator's real-time task requirements.
[0065] like Figures 1 to 8 As shown, an infrared-visible image fusion system with embodied interactive intent gating is disclosed. The system includes a multimodal data acquisition module 100, a processing and fusion computing platform, an interactive feedback module 500, an operator 600, and an interactive terminal 700. The processing and fusion computing platform includes a variable modal intent understanding module 200, a structured embodied intent state update module 300, and an intent-driven multimodal fusion module 400. The processing and fusion computing platform can be an in-vehicle computing unit, an edge computing device, an industrial control computer, a server, an embedded computing platform, or a combination of the above devices.
[0066] The multimodal data acquisition module 100 includes an image data acquisition unit 110, an acoustic data acquisition unit 120, a text / task data access unit 130, a structured perception result access unit 140, and a localization and environmental perception data acquisition unit 150. The multimodal data acquisition module 100 is used to acquire infrared images, visible light images, and image association perception results to be fused. The image association perception results can be target detection results, target segmentation results, target tracking results, localization results, pose estimation results, or environmental state results. Infrared images provide structural and thermal radiation information, while visible light images provide color, texture, boundary, and background structural information.
[0067] Image data acquisition unit 110 is used to acquire visible light images, infrared images, depth images or other image information; acoustic data acquisition unit 120 is used to acquire voice or environmental acoustic information; text / task data access unit 130 is used to receive task text, task instructions or operation text; structured perception result access unit 140 is used to receive target detection results, target segmentation results, target tracking results, risk identification results or state identification results; positioning and environmental perception data acquisition unit 150 is used to receive positioning data, attitude data, meteorological data, temperature data, distance data or other environmental state data.
[0068] The variable modal intent understanding module 200 is used to encode, determine availability, assess confidence, and fuse currently available embodied interaction data and image-related perception results to generate fused intent features in a unified intent space and further obtain structured intent information. The embodied interaction data includes at least one of the following: speech generated by the operator 600, gaze / attention information, and images.
[0069] Variable modality embodied intent understanding and intent state update are as follows:
[0070] The variable modality intent understanding module 200 includes a modality-specific encoding unit 210, a modality type identifier generation unit 220, a modality availability judgment unit 230, a modality confidence evaluation unit 240, a variable modality fusion unit 250, and an intent decoding unit 260. It supports any one or more interactive modal inputs.
[0071] The modality-specific coding unit 210 encodes different modalities separately. Speech information can be converted into semantic cues by a speech encoder; gaze / attention information can be converted into spatial cues by a gaze encoder; head pose information can be converted into posture orientation features; gesture / touch information can be converted into operational behavior features; task text can be converted into task semantic features; and image association perception results can be converted into target candidate features, risk region features, or historical tracking features. The interaction intent at time t is represented as:
[0072] ;
[0073] in, This indicates semantic prompts obtained from parsing voice commands, task text, or feedback information. This represents spatial cues derived from the gaze region, gesture region, touch location, target candidate box, or historical tracking results. This representation differs from simply using text as a global conditional vector, allowing interactive intents to directly participate in the information allocation of local regions.
[0074] The structured embodied intent state update module 300 is used to generate or update the structured embodied intent state based on the structured intent information output by the variable modality intent understanding module 200, historical intent states, and feedback information. The structured embodied intent state includes at least one of intent type, object of interest, region of interest, target mask, fusion mode, task priority, output method, and intent confidence.
[0075] The structured embodied intent state update module 300 updates the current intent state based on the previous structured embodied intent state, the current fused intent features, the modal availability identifier, and the operator feedback information. The structured embodied intent state can be represented as:
[0076] ;
[0077] in, Indicates the intent type. Indicates the object of attention. Indicates the area of interest. Indicates the target mask. Indicates the fusion mode. Indicates task priority. Indicates the output method. This represents the confidence level of the intent. The current intent state can be represented by the state update function as follows:
[0078] in, For the structured embodied intention state of the previous moment, For the current fusion intent features, For modal availability identification, For modal confidence, Provide feedback to the operator for confirmation, rejection, or correction.
[0079] The intent-driven multimodal fusion module 400 is used to generate semantic prior information, fusion mode control quantity and multidimensional physical prior tensor according to the structured embodied intent state. It further reorganizes the target energy features and background texture features through the regional gating mechanism of core domain, transition ring and background area to obtain fused image, enhanced display result, target prompt information, risk prompt information or control command.
[0080] The interactive feedback module 500 includes a display / enhanced display unit 510, a prompt / alarm unit 520, a voice / tactile feedback unit 530, and a control command output unit 540. The display / enhanced display unit 510 displays the fused image, enhanced display results, or target prompt information; the prompt / alarm unit 520 outputs risk prompts, target prompts, or abnormal alarms; the voice / tactile feedback unit 530 outputs voice broadcasts or tactile feedback; and the control command output unit 540 outputs control commands related to the fusion results or the operator's intent.
[0081] The interactive terminal 700 includes a transparent display / AR-HUD 710, an eye-tracking unit 720, a voice acquisition unit 730, a head posture / IMU unit 740, a gesture / touch / button unit 750, and a cockpit camera / text input unit 760. The interactive terminal 700 is used to facilitate interaction between the operator 600 and the system. The interactive terminal 700 can be a transparent smart cockpit, a human-machine collaborative control console, an augmented reality display terminal, an unmanned system control terminal, a vehicle cockpit, a robot remote operation terminal, a security monitoring terminal, an emergency search and rescue terminal, or an industrial inspection terminal. In a preferred embodiment, the interactive terminal 700 is a transparent smart cockpit, where the operator 600 is located. The operator observes the fusion result through the transparent display / AR-HUD 710 and generates embodied interactive data through the eye-tracking unit 720, the voice acquisition unit 730, the head posture / IMU unit 740, the gesture / touch / button unit 750, or the cockpit camera / text input unit 760.
[0082] The interactive feedback module 500 is used to feed back the fusion result to the interactive terminal 700 in the form of display, enhanced display, prompts, alarms, voice broadcasts, haptic feedback, or control commands, and then the interactive terminal 700 feeds it back to the operator 600. At the same time, the subsequent interactive information generated by the operator 600 based on the fusion result can be input into the variable modal intent understanding module 200 and / or the structured embodied intent state update module 300 through the interactive terminal 700, thereby forming a closed-loop interactive process.
[0083] Focus on region mapping and spatial cue generation, specifically:
[0084] When the embodied interaction data includes gaze / attention information, the eye-tracking unit 720 acquires the eye movement information of the operator 600 to obtain the gaze point or gaze area in the display coordinate system. The system maps the gaze information to an infrared image, a visible light image, or a unified image coordinate system through calibration transformation, gaze-to-screen mapping, screen-to-image mapping, image-to-sensor mapping, and temporal alignment.
[0085] In one implementation, the gaze point is set as... The area of focus is , No. The candidate target areas are Candidate regional center is The target detection confidence level is Historical attention is consistent The matching probability of the i-th candidate target with the current interaction intent is expressed as:
[0086] ;
[0087] in, For the gaze area, For the i-th candidate target region, For the fixation point, As candidate target centers, For target detection confidence, To maintain consistency in historical focus, , which represents the weighting coefficient. When the gaze modality is unavailable, spatial cues can be supplemented by voice commands, gesture regions, task text, or historical tracking results.
[0088] Intent prior modeling, specifically:
[0089] In order to enable sparse interaction signals to participate in pixel-level or feature-level fusion, this embodiment will include the interaction intent. This is transformed into a dense spatial intent prior. The dense spatial intent prior can be represented as:
[0090] ;
[0091] in, Indicates semantic hints, The system generates dense spatial intent priors based on the semantic and spatial cues, thereby converting sparse interaction signals into spatial constraints that can participate in local feature modulation. Indicates the intent parsing operator, This indicates that the initial intent region is scaled and aligned to the fused feature scale. , This represents a feature-level dense spatial intent prior. In this way, the originally sparse speech, text, gaze, bounding box, or gesture regions are transformed into dense spatial constraints that can participate in local feature modulation, enabling the fusion model to respond to the current task intent at the region level. The uncertainty graph can be represented as:
[0092] ;
[0093] in, To prevent abnormally small positive numbers in logarithmic calculations. The larger the value, the more uncertain the foreground / background assignment is at that location, making it more suitable as a transition ring or smooth blending area. The boundary intensity map, derived from the spatial gradient of the dense spatial intent prior, is used to determine the transition ring between the target core region and the background region. The boundary transition cue can be represented as:
[0094] ;
[0095] In this way, the originally sparse speech, text, gaze, target box or gesture regions are transformed into dense spatial constraints that can participate in local feature modulation, enabling the fusion model to respond to the current task intent at the region level.
[0096] Semantic injection adaptation and intent-enhanced latent feature generation, specifically:
[0097] The intent-driven multimodal fusion module 400 extracts modal features using both an infrared encoder and a visible light encoder.
[0098] ;
[0099] in, and These represent the infrared encoder and the visible light encoder, respectively. Shared fused features are then obtained through a feature aggregation operator, and projected dense semantic features are obtained through a semantic projection adapter.
[0100] ;
[0101] in, For infrared / visible light shared feature aggregation operator, For semantic projection adapter, This refers to the dense visual semantic features output by the image association perception result or the visual prior encoder. The intent-aware attention map can be represented as:
[0102] ;
[0103] And obtain the potential features intended to be enhanced:
[0104] ;
[0105] in, and These are learnable weight coefficients. This indicates element-wise multiplication.
[0106] Two-head orthogonal decoupling and foreground / background reconstruction, specifically:
[0107] To avoid target enhancement and background keeping competing with each other on the same path, this embodiment aims to enhance latent features. Inputting the dual-head orthogonal decoupling module yields foreground target branch features and background texture branch features respectively:
[0108] ;
[0109] in, For the target energy branch, This is a branch for the background texture.
[0110] The foreground target branch reconstruction result and the background texture branch reconstruction result satisfy:
[0111] ;
[0112] in, and These represent the target reconstruction head and the texture reconstruction head, respectively. This indicates the operation of maximizing the value pixel by pixel. This represents the normalized infrared intensity map. (The above...) The operation can be applied to the luminance channel, intensity channel, or target saliency channel, and broadcasts infrared lower bound cues to the output channel when needed to reduce the risk of the infrared thermal response of the target area being covered by visible light texture. To avoid over-enhancement of the boundary caused by foreground enhancement, this invention further introduces uncertainty-aware gating and transition ring band constraints.
[0113] Uncertainty-aware gating and selective fusion, specifically:
[0114] The uncertainty-aware gating unit constructs a gating input tensor based on dense spatial intent priors, uncertainty, boundary transition cues, modal differences, normalized infrared intensity, and normalized visible light brightness.
[0115] ;
[0116] in, Indicates splicing along the channel. and These represent the normalized infrared intensity map and visible light brightness map, respectively. Modal difference cues are:
[0117] ;
[0118] in This represents the gradient difference weighting coefficient. This expression considers both infrared intensity differences and local structural differences, making it more suitable for identifying target boundaries and modal conflict regions.
[0119] The gating weights are generated by the gating network:
[0120] ;
[0121] in, Indicates a gating network. For temperature control parameters, This indicates a morphological expansion operation, used to extend the gating range to the target boundary and transition zone. The smaller the value, the steeper the gating boundary; The larger the value, the smoother the transition zone.
[0122] The fusion result is represented as follows:
[0123] ;
[0124] in, Controlling the contribution of the target energy branch, The contribution of the background texture branch is controlled. This formula ensures that the core domain prioritizes enhancing the thermal saliency of the infrared target, the background region prioritizes preserving visible light texture details, and the transition rings achieve a smooth transition through continuous gating weights.
[0125] The regional adaptive optimization mechanism is as follows:
[0126] During the training or optimization phase, in order to ensure that the fusion result simultaneously satisfies the requirements of region reconstruction consistency and feature decoupling, this embodiment defines the overall optimization objective as follows:
[0127] ;
[0128] in, For regional reconstruction losses, This is the semantic decoupling loss. The region reconstruction loss applies constraints separately for the foreground target region, background region, and transition loop:
[0129] ;
[0130] in, Indicates the region The corresponding binary mask or soft mask. The foreground target region loss is used to maintain the thermal saliency of the target and retain necessary texture details; the background region loss is used to constrain the fusion result to be close to the visible light structure in the background region and suppress the diffusion of infrared thermal noise; the transition ring band loss is used to maintain the gradient continuity at the target boundary and reduce halo artifacts, structural breaks or hard boundaries.
[0131] Semantic decoupling loss is used to reduce redundant coupling between target saliency features and background texture features:
[0132] ;
[0133] in, Constraint target branch features and background branch features The correlation, A contrastive learning mechanism is employed to bring the foreground fusion features closer to the positive sample anchor points. and away from the background negative sample anchor point This mechanism can enhance the discriminative power of foreground region representation and reduce the interference of background texture on the thermal saliency of the target.
[0134] The embodied interaction closed loop is as follows:
[0135] The fusion process of this invention is not a single execution process, but an embodied interaction closed-loop process. The multimodal data acquisition module 100 acquires infrared images, visible light images, and image association perception results in the current scene; the variable modal intent understanding module 200 generates interaction intent and fusion intent features based on the currently available embodied interaction data; the structured embodied intent state update module 300 generates or updates the structured embodied intent state based on the fusion intent features and historical intent states; the intent-driven multimodal fusion module 400 performs selective information allocation fusion based on the interaction intent and structured embodied intent state; and the interaction feedback module 500 feeds back the fusion results to the operator 600.
[0136] After observing the fusion result, the operator (600) can generate new embodied interaction data through eye contact confirmation, voice confirmation, gesture confirmation, touch confirmation, button confirmation, negative feedback, or corrective feedback. This new embodied interaction data re-enters the variable modality intent understanding module (200) and / or the structured embodied intent state update module (300) for the next round of intent understanding, intent state correction, and fusion strategy update. This closed loop enables the system to update dense spatial intent priors and spatial adaptive gating weights based on new fusion intent features and historical structured embodied intent states during continuous interaction. When the operator's focus changes, the system can redetermine the region of interest, target mask, and target core domain, reducing fusion weight update lag and region mismatch. When there is high uncertainty at the target boundary or significant differences between infrared and visible light modalities, the system can smooth the gating weights through boundary transition cues and temperature adjustment parameters, reducing the risk of infrared thermal response diffusion and visible light texture covering the target's thermal saliency. When some interaction modalities are missing or have low confidence, the system can maintain high-confidence intent fields and locally correct low-confidence fields, thereby avoiding abrupt changes in fusion modes and gating weights.
[0137] Example 2:
[0138] This invention provides an infrared-visible image fusion method with embodied interactive intent gating, comprising the following steps:
[0139] Step 1: Acquire infrared images of the same scene that have undergone temporal synchronization and spatial registration. With visible light images The data is then normalized; simultaneously, the embodied interaction data generated by the operator during the interaction process is acquired.
[0140] Step 2: Process the embodied interactive data and infrared images. Visible light images Modality encoding is performed on the image-associated perception results to generate modality features, modality availability identifiers, and modality confidence scores. Based on these modality features, modality availability identifiers, and modality confidence scores, the current fusion intent features are generated. .
[0141] Step 3: Based on the structured embodied intention state of the previous moment Current fusion intent characteristics Modal availability identifier Modal confidence and operator feedback Generate the current structured embodied intent state .
[0142] Step 4: Based on the current structured embodied intent state Generate semantic cues to describe the current task objective. and spatial cues for constraining regions to be enhanced, maintained, or suppressed. and the semantic prompt and space prompts Transformation into dense space intention prior .
[0143] Step 5: Process the infrared images respectively. With visible light images Multi-scale feature encoding is performed to obtain infrared coded features and visible light coded features. Shared feature aggregation is then performed on the infrared coded features and visible light coded features, based on the dense spatial intent prior. Semantic injection is performed on the aggregated shared fusion features to obtain potential features for intent enhancement.
[0144] Step 6: Input the intended enhancement potential features into the dual-head orthogonal decoupling module to obtain target energy features for characterizing the thermal saliency of infrared targets and background texture features for characterizing visible light texture details.
[0145] Step 7: Based on the aforementioned dense spatial intent prior. Generate uncertainty graph and boundary transition clues According to the normalized infrared intensity map With normalized visible light intensity map Intensity differences and gradient differences generate modal difference cues The dense spatial intention is a priori. Uncertainty diagram Boundary transition clues Gradient difference generates modal difference cues Normalized infrared intensity map Normalized visible light intensity map Concatenation forms a gated input tensor The spatial adaptive gating weights are generated by the gating network in conjunction with temperature regulation parameters. .
[0146] Step 8: Based on the spatial adaptive gating weights The target energy features and background texture features are selectively fused to generate target enhancement results. Preserve the result with texture And based on the spatial adaptive gating weights Enhancement results for the target Preserve the result with texture Regional selection fusion is performed to obtain a fused image that matches the operator's current intention. .
[0147] The process involves several steps: Step 1 is executed by the multimodal data acquisition module 100; Step 2 by the variable modal intent understanding module 200; Step 3 by the structured embodied intent state update module 300; Steps 4 through 7 by the intent-driven multimodal fusion module 400. Step 4 generates semantic cues, spatial cues, and dense spatial intent priors based on the structured embodied intent state; Step 5 performs infrared and visible light feature encoding and semantic injection; Step 6 performs dual-head orthogonal decoupling; and Step 7 generates spatially adaptive gating weights. Step 8 is jointly executed by the intent-driven multimodal fusion module 400 and the interactive feedback module 500. The intent-driven multimodal fusion module 400 outputs the fused image or enhanced display result, while the interactive feedback module 500 feeds the fusion result back to the operator and collects confirmation, rejection, or correction feedback. The feedback path after Step 8 indicates that subsequent embodied interaction data from the operator can be fed back to Steps S2 and / or S3 for the next round of intent understanding, intent state correction, and fusion strategy update.
[0148] Example 3
[0149] Applications in transparent smart cockpits
[0150] In a transparent smart cockpit application scenario, the multimodal data acquisition module 100 receives infrared and visible light images from vehicles, drones, robots, or fixed sensing devices, and receives target detection, target segmentation, or risk identification results. The operator 600 observes the scene ahead through a transparent display / AR-HUD 710. When the operator gazes at a suspected pedestrian or thermally anomalous target in the distance and issues a voice command to "enhance this target," the eye-tracking unit 720 acquires the gaze point, the voice acquisition unit 730 acquires the voice command, and the variable modal intent understanding module 200 converts the gaze point into a spatial cue, the voice command into a semantic cue, and generates an interactive intent. .
[0151] The intent-driven multimodal fusion module 400 converts interactive intents into dense spatial intent priors, generating uncertainty maps and boundary transition cues. Then, it injects the semantic priors into shared fusion features of infrared and visible light, generating intent-enhancing latent features. A dual-head decoupling structure is then used to obtain foreground target reconstruction results and background texture reconstruction results. Finally, selective fusion is performed using uncertainty-aware gating weights. For gazed target regions that match speech semantics, the system enhances infrared target thermal saliency; for background regions such as roads, buildings, and the sky, the system preserves visible light texture; and for target contours or modal conflict regions, the system performs smooth transitions.
[0152] If the operator subsequently issues a corrective statement such as "No, it's the target on the right" or shifts their gaze to the target area on the right, the system updates the structured embodied intent state module 300 with the new gaze point and corrected semantic input, reduces the intent confidence of the original object of interest, and recalculates the spatial cues, dense spatial intent priors, and gating weights for the target area on the right. Thus, the fusion result changes in real time according to the operator's intent without requiring manual reconfiguration of the fusion algorithm parameters.
[0153] Example 4
[0154] Fusion effect verification
[0155] To verify the technical effectiveness of the method of this invention in infrared and visible light image fusion tasks, publicly available infrared and visible light image datasets were used to train, test, and validate the method across scenarios. The training set and basic test set can use the LLVIP dataset, which contains rigorously registered infrared and visible light images. The cross-scenario test set can use at least one of the MSRS, TNO, and VIFB datasets. These datasets include typical scenarios such as nighttime roads, pedestrian targets, vehicle targets, building backgrounds, low-light scenes, thermally anomalous targets, and complex background textures, which can be used to verify the technical effectiveness of the method of this invention in enhancing target thermal saliency, preserving background texture, maintaining boundary continuity, and achieving cross-scenario stability.
[0156] In this embodiment, the method of the present invention is compared with several publicly available infrared and visible light image fusion methods. These comparison methods may include at least one of MemoryFusion, SAGE-IVIF, F2Fusion, MaeFuse, Text-IF, DSFusion, SwinFusion, and U2Fusion. To ensure the comparability of the verification results, each method uses the same input image, image resolution, normalization method, and evaluation script. For methods that publicly provide pre-trained models, their publicly available weights are preferred. For methods that do not provide corresponding dataset weights, the recommended configuration in their publicly available implementation is used for testing.
[0157] In this embodiment, at least one of the following is used as an evaluation index for fusion quality: Information Entropy (EN), Spatial Frequency (SF), Standard Deviation (SD), Visual Fidelity (VIF), Average Gradient (AG), Edge Intensity (EI), and Difference Correlation (DF). EN characterizes the amount of information in the fusion result; SF and AG characterize spatial details and gradient changes in the image; SD characterizes image contrast; VIF characterizes visual information fidelity; EI characterizes edge preservation capability; and DF characterizes the degree to which complementary information from the source images is preserved in the fusion result. Since a single index may be affected by noise, contrast, or texture intensity, this embodiment employs a multi-index joint evaluation method to comprehensively analyze the fusion result.
[0158] The input infrared and visible light images are cropped or scaled to a preset size and then input into the fusion network. The training process can use the Adam optimizer, with an initial learning rate of 1×10⁻⁴, a batch size of 4, and 20 training epochs. These parameters are only one optional implementation method. Those skilled in the art can adjust the image size, batch size, learning rate, and number of training epochs according to the computing platform, input image resolution, training data scale, and application scenario without affecting the implementation of the technical solution of this invention.
[0159] Figure 9 The qualitative fusion results of the method of this invention and several comparative methods in representative scenarios are shown. Figure 9As can be seen, while some contrast methods can enhance infrared targets, they are prone to causing loss of background texture details, target boundary distortion, or thermal response diffusion into the background. Other contrast methods, while preserving more visible light texture, weaken the thermal saliency of infrared targets. In contrast, the method of this invention generates dense spatial intent priors through structured embodied intent states and combines uncertainty maps, boundary transition cues, and modal difference cues to generate spatially adaptive gating weights. This prioritizes enhancing the thermal saliency of infrared targets in the target core domain, prioritizes preserving visible light texture details in the background region, and achieves continuous weighted fusion in the boundary transition region, thereby reducing problems such as hard boundary switching, halo artifacts, and texture overlay on the target.
[0160] Tables 1 to 4 show the objective evaluation results of the method of the present invention and several comparative methods on different public test sets. In several test scenarios, the method of the present invention performs better in terms of information preservation, visual fidelity, edge preservation, and modal complementarity utilization, indicating that the method of the present invention can enhance the thermal saliency of infrared targets while maintaining visible light background texture. Furthermore, under cross-scene test conditions, the method of the present invention can still maintain relatively stable fusion quality, indicating that regional fusion through structured embodied intent states, dense spatial intent priors, and spatial adaptive gating weights can improve the adaptability of the fusion results to different scene distributions.
[0161] Table 1
[0162]
[0163] Table 2
[0164]
[0165] Table 3
[0166]
[0167] Table 4
[0168]
[0169] Figure 10 The fusion results under different interactive intent conditions are shown. Under the target enhancement intent, the method of the present invention can improve the fusion weight of target energy features in the target core domain, making infrared thermal targets more prominent; under the background fidelity intent, the method of the present invention can improve the fusion weight of background texture features in the background region, making visible light texture, road structure and building edges better preserved; under the case of target correction or change of the area of interest, the method of the present invention can update the structured embodied intent state according to new semantic cues, spatial cues and operator feedback, and regenerate dense spatial intent priors and spatial adaptive gating weights, thereby reducing region mismatch and lag in fusion weight update when the area of interest changes.
[0170] To further verify the support effect of the method of the present invention on downstream vision tasks, the fusion result is input into the object detection model for detection performance testing, such as... Figure 11 As shown in Table 5, the verification results demonstrate that, compared to partial contrast fusion methods, the fused image generated by the method of this invention provides clearer thermal saliency cues in the target region and retains visible light structure information in the background region. This is beneficial for improving the accuracy, precision, and recall of the target detection model for pedestrians, vehicles, or thermally anomalous targets. These results indicate that the method of this invention not only improves the visual quality of the fused image but also enhances the usability of the fusion result for subsequent target detection, risk alerts, and decision support tasks.
[0171] Table 5
[0172]
[0173] It should be noted that the above-mentioned dataset, comparison method, evaluation metric, training parameters, and downstream target detection model are only used to illustrate one verification method of the present invention and do not constitute a limitation on the scope of protection of the present invention. Those skilled in the art can also use other infrared and visible light registered image data, other publicly available fusion methods, other objective evaluation metrics, or other downstream task models to verify the method of the present invention.
[0174] The parts not described in detail in this article are existing technologies.
[0175] While the specific embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and modifications or variations without creative effort are still within the protection scope of the present invention.
Claims
1. A method for infrared and visible light image fusion with embodied interactive intent gating, characterized in that, The steps include the following: Step 1: Acquire infrared images of the same scene that have undergone temporal synchronization and spatial registration. With visible light images And perform normalization processing; at the same time, acquire the concrete interaction data generated by the operator during the interaction process; Step 2: Process the embodied interactive data and infrared images. Visible light images Modality encoding is performed on the image-associated perception results to generate modality features, modality availability identifiers, and modality confidence scores. Based on these modality features, modality availability identifiers, and modality confidence scores, the current fusion intent features are generated. ; Step 3: Based on the structured embodied intention state of the previous moment Current fusion intent characteristics Modal availability identifier Modal confidence and operator feedback Generate the current structured embodied intent state ; wherein, the current structured embodied intent state Generate using the following state update function: ; Step 4: Based on the current structured embodied intent state Generate semantic cues to describe the current task objective. and spatial cues for constraining regions to be enhanced, maintained, or suppressed. and the semantic prompt and space prompts Transformation into dense space intention prior ; Step 5: Process the infrared images respectively. With visible light images Multi-scale feature encoding is performed to obtain infrared coded features and visible light coded features. Shared feature aggregation is then performed on the infrared coded features and visible light coded features, based on the dense spatial intent prior. Semantic injection is performed on the aggregated shared fusion features to obtain potential features for intent enhancement. Step 6: Input the intended enhancement latent features into the dual-head orthogonal decoupling module to obtain target energy features for characterizing the thermal saliency of infrared targets and background texture features for characterizing visible light texture details, respectively; Step 7: Based on the aforementioned dense spatial intent prior. Generate uncertainty graph and boundary transition clues According to the normalized infrared intensity map With normalized visible light intensity map Intensity differences and gradient differences generate modal difference cues The dense spatial intention is a priori. Uncertainty diagram Boundary transition clues Gradient difference generates modal difference cues Normalized infrared intensity map Normalized visible light intensity map Concatenation forms a gated input tensor The spatial adaptive gating weights are generated by the gating network in conjunction with temperature regulation parameters. ; Step 8: Based on the spatial adaptive gating weights The target energy features and background texture features are selectively fused to generate target enhancement results. Preserve the result with texture And based on the spatial adaptive gating weights Enhancement results for the target Preserve the result with texture Regional selection fusion is performed to obtain a fused image that matches the operator's current intention. .
2. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: The embodied interaction data includes at least one of operator feedback, gaze information, gesture area, touch position, and head posture information; the image association perception results include at least one of target detection results, target segmentation results, target tracking results, risk area results, target category probability map, and target candidate bounding box. The modality availability identifier is used to indicate whether the corresponding interaction modality exists, is valid, or participates in the current intent understanding. When any interaction modality is missing, occluded, has excessive noise, excessive delay, fails to acquire, fails to recognize, or has a confidence level lower than a preset threshold, the interaction modality is marked as unavailable or low-weight available.
3. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: The current structured embodied intent state It should at least include the object of interest, the region of interest, the target mask, the fusion mode, and the intent confidence; the operator feedback Provide operators with confirmation, rejection, or correction feedback; When the confirmed feedback is valid, increase the intent confidence of the current object of interest, the area of interest, and the fusion mode; when the negative or corrective feedback is valid, decrease the matching probability of the original object of interest and update the area of interest and the target mask.
4. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: When the embodied interaction data includes gaze information, the operator's gaze is mapped to the display coordinate system, image coordinate system, spatial coordinate system, or sensor coordinate system to obtain the gaze point or gaze area. The object of attention or the area of attention is determined based on the spatial overlap, center distance, target detection confidence, and historical attention consistency between the gaze point or gaze area and the target candidate box, target segmentation area, target tracking area, or risk area. The matching probability between the i-th candidate target and the current operator's intention is expressed as: ; in, Indicates the area of focus. Indicates the first One candidate target region Indicates the point of fixation. Indicates the candidate target center, This represents the confidence level of object detection. This indicates a focus on consistency in historical perspectives. , , , and These are preset or learnable parameters.
5. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: The dense space intention prior The method used to distinguish between the target core region, background region, and boundary transition region is as follows: ; in, Indicates semantic hints, Indicates a spatial hint, Indicates the intent parsing operator, This indicates that the parsed initial intent region will be scaled and aligned to the fusion feature scale. The semantic hints The spatial prompts are obtained by parsing voice commands, task text, or operator feedback. It is generated from at least one of the following: gaze point, gaze region, gesture region, touch location, target candidate box, target segmentation region, or historical tracking region.
6. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: The uncertainty diagram and boundary transition clues They are represented as follows: ; ; in, To prevent abnormally positive numbers in logarithmic calculations, This represents the spatial gradient operator.
7. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: Modal difference cues Gated input tensors Spatial adaptive gating weights and fused images They are represented as follows: ; ; ; ; in, This represents the normalized infrared intensity map. This represents a normalized visible light intensity map. This represents the gradient difference weight coefficient. This indicates a channel splicing operation. Indicates a gating network. This represents the activation function. Indicates temperature control parameters. This represents element-wise multiplication. The spatial gating constraint diagram is represented as follows: ; in, Indicates the spatial gating constraint coefficient. This indicates a morphological dilation operation.
8. The infrared-visible image fusion method with embodied interactive intent gating according to claim 1, characterized in that: The target enhancement result The texture is reconstructed from the target energy features and constrained by a normalized infrared intensity map or an infrared target saliency map; the texture preservation result... It is reconstructed from background texture features and is constrained by visible light brightness, gradient, or texture structure.
9. An infrared-visible light image fusion system with embodied interactive intent gating, characterized in that, include: Multimodal data acquisition module, embodied interaction data acquisition module, variable modal intent understanding module, structured embodied intent state update module, intent-driven multimodal fusion module, interactive feedback module, interactive terminal; The multimodal data acquisition module includes an image data acquisition unit, an acoustic data acquisition unit, a text data access unit, a structured perception result access unit, and a positioning and environmental perception data acquisition unit. The image data acquisition unit is used to acquire infrared images and visible light images that have been time-synchronized and spatially registered in the same scene, and to normalize the infrared images and visible light images to obtain normalized infrared intensity maps and normalized visible light brightness maps. The embodied interaction data acquisition module is used to acquire at least one of semantic interaction data and spatial interaction data generated by the operator during the interaction process; The variable modal intent understanding module is used to perform modal encoding on the currently available embodied interaction data and image association perception results to generate modal features, modal availability identifiers, modal confidence, and current fused intent features; The structured embodied intent state update module is used to generate or update the current structured embodied intent state based on the previous structured embodied intent state, the current fused intent features, the modal availability identifier, the modal confidence, and the operator feedback information. The structured embodied intent state includes at least one of intent type, object of interest, region of interest, target mask, fusion mode, task priority, output method, and intent confidence. The intent-driven multimodal fusion module is used to generate semantic prior information, fusion mode control quantity and multidimensional physical prior tensor according to the structured embodied intent state. It further reorganizes the target energy features and background texture features through the regional gating mechanism of core domain, transition ring and background area to obtain fused image, enhanced display result, target prompt information, risk prompt information or control command. The interactive feedback module is used to output the fused image or enhanced display result, and to provide the modality coding module or the structured embodied intent state update module with the subsequent embodied interaction data generated by the operator based on the fused image or enhanced display result.
10. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.
