Dual-view decoupled image fusion method based on semantic constraints

Abstract

The application discloses a dual-view decoupled image fusion method based on semantic constraints, comprising: acquiring registered multi-modal images and corresponding text descriptions; decomposing visual representation into modal-specific features and modal-shared features, and applying decoupled constraints; extracting modal-specific semantic representation by using a frozen CLIP model, and obtaining modal-invariant shared semantic representation by element-by-element multiplication; cross-modal modulation is performed on the modal-specific semantic representation and the modal-specific visual features, and cross-modal alignment is performed on the shared semantic representation and the modal-shared visual features, so that semantic-enhanced modal-specific features and semantically-aligned shared features are obtained; and finally, the fusion features are input into a reconstruction decoder to generate a fusion image. The application effectively solves the problems of detail loss and structural distortion caused by feature coupling in multi-modal image fusion, introduces high-level semantic constraints to improve the semantic consistency and detail fidelity of the fusion result, and is widely used in downstream visual tasks such as target detection and image segmentation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal information processing technology, specifically relating to a dual-view decoupled image fusion method based on semantic constraints. Background Technology

[0002] With the development of multi-sensor imaging and information acquisition technologies, multimodal image fusion has been widely used in fields such as intelligent perception, target recognition, and medical image analysis. By fusing images from different modalities, the advantages of each modality can be integrated to obtain a more comprehensive and reliable visual representation.

[0003] Existing multimodal image fusion methods are mainly based on feature extraction and information aggregation. Their development has generally gone through a process from simple fusion to task-driven fusion (as shown in the appendix). Figure 1 (As shown). Simple fusion methods typically mix features directly through concatenation, weighting, or convolution. While simple to implement and computationally efficient, they lack a mechanism to distinguish between modality-shared structure and modality-specific information, easily leading to weakened detailed information and insufficient representation of important targets. Task-driven fusion methods combine the fusion process with downstream tasks (such as object detection or semantic segmentation). While this can improve performance on specific tasks, the fusion results are easily affected by task objective bias. In complex scenarios or with multi-task requirements, it is difficult to balance structural information and modality-specific information.

[0004] Overall, the aforementioned methods generally suffer from feature coupling problems, meaning that shared modality information and modality-specific information are mixed during the fusion process, lacking a clear decoupling mechanism. This leads to phenomena such as loss of detail, blurred structural representation, and excessive smoothing in the fusion results. Furthermore, existing methods largely rely on low- or mid-level visual features, lacking effective constraints from high-level semantic information. In recent years, although some methods have introduced visual-language models to enhance semantic expressive capabilities, they typically treat semantic information as a global condition uniformly applied to the feature space, failing to differentiate between shared and modality-specific information, thus still struggling to fundamentally solve the feature entanglement problem.

[0005] Therefore, the technical problem that existing technologies need to solve is: how to effectively distinguish and decouple modal shared structural information from modal-specific detailed information during multimodal image fusion, and to introduce a reasonable semantic constraint mechanism to avoid the loss of details and structural distortion caused by feature coupling, thereby improving the quality, stability and adaptability of the fusion results. Summary of the Invention

[0006] The purpose of this invention is to provide a semantically constrained dual-view decoupled image fusion method that addresses the problems of modal feature coupling, insufficient semantic constraints, loss of details, and structural distortion in existing multimodal image fusion methods.

[0007] The technical solution provided by this invention is as follows: This invention provides a dual-view decoupled image fusion method based on semantic constraints, comprising:

[0008] S1: Obtain the registered multimodal image and its corresponding text description, and use the image-text pair as the fusion input;

[0009] S2: Extract basic visual features from the input image and generate modality-specific features and shared features respectively; decouple the shared features and modality-specific features;

[0010] S3: Perform semantic-level feature encoding on the text description to extract modality-specific semantic representations and shared semantic representations;

[0011] S4: Based on the modality-specific features, modality-shared features, modality-specific semantic representations, and shared semantic representations, perform cross-modal interaction and fusion between visual features and textual semantics: perform cross-modal modulation on the modality-specific semantic representations and the modality-specific visual features, and perform cross-modal alignment on the shared semantic representations and the modality-shared visual features to obtain semantically enhanced modality-specific features and semantically aligned shared features;

[0012] S5: The semantically enhanced modality-specific features and semantically aligned shared features are fused by element-level addition, and the fused features are input into the reconstruction decoder to generate the final fused image.

[0013] Preferably, the step of extracting basic visual features from the input image and generating modality-specific features and shared features respectively includes:

[0014] Shallow visual features are extracted from images of each modality using a shared backbone encoder B(·). and :

[0015]

[0016] The shallow features are input into the specific encoders of their respective modalities. and Generate modality-specific feature representations:

[0017]

[0018] and These represent specific features in infrared and visible light encoding, respectively.

[0019] Preferably, the specific encoder of the mode and In this study, one approach uses a lightweight Transformer module, while the other employs a hybrid CNN-INN architecture.

[0020] Input shallow features into the modality-sharing encoder Generate shared feature representations:

[0021]

[0022] in, and These represent modal-shared features extracted from infrared and visible light inputs, respectively.

[0023] Preferably, a contrastive alignment-separation constraint is used to adjust the distribution of shared features and modality-specific features in the pixel-level feature space, wherein the alignment operator (AO) encourages shared features to converge to a common subspace; and the separation operator (SO) suppresses unwanted correlations between modality-specific features and shared features, as well as between different modality-specific features.

[0024] The formal representation is as shown in formula (4):

[0025]

[0026] in, Represents the cosine similarity function;

[0027] By jointly optimizing the alignment-separation constraints, the structure decouples pixel-level features.

[0028] Preferably, the semantic-level feature encoding:

[0029] Text prompts for each modality are input into a fixed vision-language model for encoding, generating modality-specific semantic representations. and ;

[0030] By multiplying the specific semantic features of the two modalities element by element, a shared semantic representation is obtained, as shown in formula (5):

[0031]

[0032] Preferably, S4 specifically includes: setting up modality-specific interaction units (SpIUs) to refine modality-specific information, and adaptively adjusting features through semantically driven affine transformations, wherein for visible modalities, the process is as follows:

[0033]

[0034] in, and These are instance normalization and batch normalization, respectively; the same applies to infrared modality.

[0035] Shared Interaction Units (SIUs) are used to align modal shared structures, jointly model shared representations from different modalities, and perform semantic attention alignment guided by shared semantic representations. By calculating the correlation between visual and semantic features, cross-modal structures are consistently fused. The process is as follows:

[0036]

[0037] in, Spatial features are reshaped into a sequence form, with K and V derived from shared semantic embeddings.

[0038] Preferably, the reconstruction decoder is built on the Transformer architecture and is used to recover an image representation with global consistency and local detail.

[0039] Channel compression is performed on the fused features through convolution operations, mapping information from different modalities to a unified feature space;

[0040] Features are modeled by stacking Transformer modules, capturing both long-range dependencies and local structural relationships.

[0041] Image details are gradually restored through convolutional layers, and the final fusion result is output after activation function. The process can be represented as:

[0042]

[0043] in, This indicates a decoder based on Restormer. Indicates passage Convolution performs channel compression.

[0044] Furthermore, this invention also provides a semantically constrained dual-view decoupled image fusion system, comprising:

[0045] The input module is used to acquire the registered multimodal images and their corresponding text descriptions;

[0046] A pixel-level decoupling coding module is used to perform pixel-level decoupling coding on the multimodal image, decompose the visual representation into modality-specific features and modality-shared features, and apply decoupling constraints to the shared features and modality-specific features;

[0047] The semantic-level encoding module is used to perform semantic-level encoding on the text description, extract modality-specific semantic representations using a frozen visual-language model, and obtain modality-invariant shared semantic representations by multiplying them element by element.

[0048] A cross-modal interaction and fusion module is used to perform cross-modal interaction and fusion on the modality-specific features and the modality-shared features based on the modality-specific semantic representation and the shared semantic representation, respectively. The modality-specific semantic representation is used to perform semantic modulation on the modality-specific features, and the shared semantic representation is used to perform semantic alignment on the modality-shared features, to obtain semantically enhanced modality-specific features and semantically aligned shared features. A reconstruction module is used to fuse the semantically enhanced modality-specific features and the semantically aligned shared features and input them into a reconstruction decoder to generate a final fused image.

[0049] Preferably, the pixel-level decoupling coding module includes:

[0050] A shared backbone encoder is used to extract shallow visual features from images of various modalities;

[0051] Modality-specific encoders are used to generate modality-specific features from shallow visual features;

[0052] A modality-sharing encoder is used to generate modality-sharing features from shallow visual features;

[0053] The decoupling constraint unit is used to apply a contrastive alignment-separation constraint, which includes an alignment operator and a separation operator.

[0054] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described above.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] This invention effectively avoids the problems of information redundancy and structural distortion in traditional methods by combining pixel-level decoupling and semantic-level guidance in the fusion strategy of multimodal images. At the same time, the introduction of semantic information to constrain and optimize the feature interaction process improves the semantic consistency and interpretability of the fusion results. In addition, through a structured cross-modal interaction and reconstruction mechanism, the collaborative expression of modal characteristics and shared information is realized, thereby obtaining a high-quality fused image with both saliency and detail fidelity, which can better serve downstream tasks such as object detection and image segmentation. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0058] Figure 1 This diagram illustrates the representative paradigm evolution of existing multimodal image fusion methods, showcasing the development process from simple fusion to task-driven fusion.

[0059] Figure 2 This is a flowchart illustrating the overall process of the semantically constrained dual-view decoupled image fusion method proposed in this invention.

[0060] Figure 3 This is a comparison chart of the qualitative results of the method of this invention (SGDFusion) on the MSRS dataset with other comparative methods.

[0061] Figure 4 This is a comparison chart of the qualitative results of the method of this invention (SGDFusion) with other comparative methods on the TNO dataset.

[0062] Figure 5 This is a comparison chart of the qualitative results of the method of this invention (SGDFusion) with other comparative methods on the RoadScene dataset.

[0063] Figure 6 This is a comparison chart of the qualitative results of the method of this invention (SGDFusion) on the FMB dataset with other comparative methods. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are only for explaining the invention and do not constitute a limitation on the scope of protection of this invention.

[0065] Example 1: Specific implementation of the fusion method:

[0066] This invention provides a semantically constrained dual-view decoupled image fusion method, the steps of which are as follows (e.g.) Figure 2As shown):

[0067] Step 1: Input Preparation. Obtain the registered multimodal images and their corresponding text descriptions, and use the image-text pairs as the fusion input.

[0068] Step 2: Pixel-Level Disentanglement Encoding (PDE). Basic visual features are extracted from the input image, and modality-specific features and shared features are generated separately. Decoupling constraints are applied to the shared features and modality-specific features to maintain the consistency of the shared structure and preserve modality-specific information.

[0069] Based on the registration of the multimodal images and corresponding text descriptions obtained in step 1, step 2 is used to visually distinguish between modality-shared structures and modality-specific details. For this purpose, the input image... and Pixel-level decoupled encoding is performed to decompose the visual representation into modality-specific features and modality-shared features. First, shallow visual features are extracted from each modality using a shared backbone encoder B(·). and (The process is as shown in formula (1)), providing a unified basic representation for subsequent decoupling. The backbone encoder adopts a Transformer-based structure, including overlapping patch embedding layers and stacked Transformer modules, to capture local details and long-range dependencies simultaneously.

[0070]

[0071] Subsequently, to extract modality-specific features, the shallow features are input into their respective modality-specific encoders. and The modality-specific representation is generated (as shown in Equation (2)). One modality-specific encoder uses a lightweight Transformer module to emphasize the global structure and modality-specific distribution features; the other modality-specific encoder adopts a hybrid CNN-INN architecture, which preserves texture details through convolutional layers and reduces information loss by utilizing reversible coupling layers. At the same time, in order to extract cross-modality shared structures, shallow features are also input into the modality-shared encoder. The encoder generates a shared representation (as shown in Equation (3)). This encoder combines a global Transformer and a reversible decomposition module to ensure the consistency of the modally invariant structure.

[0072]

[0073]

[0074] in, and These represent unique features in infrared and visible light coding, respectively. and These represent the mode-sharing structures extracted from infrared and visible light inputs, respectively.

[0075] To regulate the distribution of shared features and modality-specific features in the pixel-level feature space, a contrastive alignment-separation constraint is designed. This constraint consists of an alignment operator (AO) and a separation operator (SO): the alignment operator encourages shared features to converge to a common subspace, thereby achieving cross-modal structural consistency; the separation operator suppresses unwanted correlations between modality-specific features and shared features, as well as between different modality-specific features, to prevent feature confusion and excessive overlap. The formal representation is shown in Equation (4):

[0076]

[0077] in, This represents the cosine similarity function.

[0078] By jointly optimizing this constraint, the structural decoupling of pixel-level features is achieved, providing a clear basic representation for semantic encoding and cross-modal fusion in subsequent steps.

[0079] Step 3: Semantic-Level Encoding and Grounding (SEG). Extract semantic features from the text description to provide high-level semantic constraints for the visual features, enabling the fusion process to focus on semantically relevant regions and structures.

[0080] Specifically, in parallel with the pixel-level encoding in step 2, semantic-level feature encoding is performed on the text description to introduce high-level semantic information and enhance semantic consistency during the fusion process.

[0081] The text prompts for each modality are encoded by a fixed visual-language model to generate modality-specific semantic representations. and These representations preserve the unique semantic characteristics of each modality, such as the thermal saliency of infrared and the texture or color information of visible light, while avoiding semantic ambiguity caused by unifying heterogeneous semantic concepts in the early stages of fusion. In order to extract modality-invariant shared semantic information, the specific semantic features of the two modalities are multiplied element-wise to obtain the shared semantic representation, as shown in formula (5):

[0082]

[0083] This shared semantic representation retains only the dimensions that are consistently activated in both modalities, suppressing unimodal-specific responses, thereby capturing high-level semantic concepts shared across modalities, such as object identity, scene layout, or structural regions of interest. The generated shared semantic representation serves as explicit semantic guidance for subsequent feature interactions, ensuring semantic consistency while preserving modality-specific details.

[0084] In this embodiment, a frozen CLIP model (e.g., the ViT-B / 32 architecture) is used as the text encoder. The CLIP model is a pre-trained contrastive language-image pre-trained model, whose parameters are fixed at this stage and do not participate in gradient updates.

[0085] This invention is not limited to the CLIP model. Any pre-trained vision-language model capable of mapping text prompts to a semantic space aligned with visual features, such as ALIGN, Florence, or a cross-modal encoder of BERT+ViT, can achieve the objectives of this invention. Those skilled in the art can choose other equivalent models based on actual computing resources and performance requirements.

[0086] Step 4: Cross-Modal Interaction and Fusion (CMIF). This involves interacting and fusing pixel-level and semantic-level features to enhance shared structural representations while preserving modality-specific details.

[0087] Specifically, based on the pixel-level decoupled features obtained in step 2 and the semantic representation obtained in step 3, cross-modal feature interaction and fusion are performed to avoid feature re-coupling and achieve effective fusion. This step uses a structured interaction mechanism to enable modality-specific features and modality-shared features to work synergistically under semantic guidance. Specifically, modality-specific interaction units (SpIU) and shared interaction units (SIU) are set up to refine modality-specific information and align modality-shared structures, respectively, thereby taking into account both modality differences and cross-modality consistency during the fusion process.

[0088] For modality-specific features, modulation is performed based on the corresponding semantic representation. The features are adaptively adjusted through semantically driven affine transformations, allowing semantic information to selectively enhance or suppress visual responses. This highlights semantically relevant regions while preserving modality-specific details. For visible modalities, the process can be represented as follows:

[0089]

[0090] in, and These are instance normalization and batch normalization, respectively. Similar operations are also applied to infrared modes to obtain... .

[0091] For modal shared features, shared representations from different modalities are jointly modeled, and semantic attention alignment is performed under the guidance of shared semantic representations. By calculating the correlation between visual features and semantic features, consistent fusion of cross-modal structures is achieved. This process can be represented as follows:

[0092]

[0093] in, Spatial features are reshaped into a sequence form, with K and V derived from shared semantic embeddings.

[0094] Through the above-mentioned interaction mechanism, the semantic enhancement of modality-specific information and the semantic alignment of modality-shared structures are realized, effectively avoiding feature confusion and redundant expression in the fusion process. This improves the structural expression and semantic consistency of the fused features, providing stable and discriminative feature representations for subsequent image reconstruction.

[0095] Step 5: Fusion and Reconstruction (RDB). The fused features are then used by the decoder to generate the final fused image, achieving coordinated preservation of structural details and semantic consistency.

[0096] Step 6: Output. The fused image is obtained, which retains rich structural details and semantic consistency, and can be used for downstream vision tasks or further analysis.

[0097] Specifically, after completing cross-modal feature interaction and fusion, semantically enhanced modality-specific features are obtained. and and shared features of semantic alignment The aforementioned features respectively characterize modality-specific information and cross-modal consistent structure. To fully utilize their complementarity, they are fused by element-wise addition and input into the reconstruction decoder to generate the final fused image.

[0098] The reconstruction decoder is built on a Transformer architecture to recover image representations with both global consistency and local detail. First, the fused features are channel-compressed through convolutional operations, mapping different modal information to a unified feature space to stabilize feature distribution. Then, stacked Transformer modules are used to model the features, simultaneously capturing long-range dependencies and local structural relationships, thereby improving structural integrity and reducing inconsistencies introduced by the fusion process. Finally, convolutional layers progressively recover image details, and the final fusion result is output after activation. .

[0099] This reconstruction process integrates semantic and structural information in the feature domain and then maps it to the pixel space, thereby obtaining a fused image that combines modal saliency and texture detail. The process can be represented as follows:

[0100]

[0101] in, This indicates a decoder based on Restormer. Indicates passage Convolution performs channel compression.

[0102] Example 2: System Construction:

[0103] This invention also provides a semantically constrained dual-view decoupled image fusion system, comprising the following modules:

[0104] Input module: Used to acquire registered multimodal images and corresponding text descriptions. This can be implemented using a stereo camera and a text annotation interface.

[0105] Pixel-level decoupling encoding module: includes a shared backbone encoder, a modality-specific encoder, a modality-shared encoder, and a decoupling constraint unit. The specific implementation is as described in step S2 of Example 1.

[0106] Semantic-level encoding module: includes a frozen CLIP text encoder and an element-wise multiplication unit. The specific implementation is as described in step S3 of Example 1.

[0107] Cross-modal interaction and fusion module: includes modality-specific interaction units and shared interaction units, and the specific implementation method is as described in step S4 of embodiment 1.

[0108] Reconstruction module: contains a Restormer-based decoder, and its specific implementation is as described in step S5 of Example 1.

[0109] The above modules can be integrated into a computing device and perform corresponding functions by running computer programs stored on computer-readable storage media.

[0110] Example 3: Dataset and Experimental Validation

[0111] To verify the performance and applicability of the present invention, four publicly available datasets, including MSRS, TNO, RoadScene, and FMB, were selected to conduct experimental evaluations of the proposed method.

[0112] The MSRS dataset contains 1083 pairs of training images and 361 pairs of test images, with semantic segmentation labels provided for each pair of images, as well as 80 pairs of images with object detection labels.

[0113] The TNO dataset provides enhanced visual (390-700nm), near-infrared (700-1000nm), and long-wave infrared (8-12μm) nighttime images of various military and surveillance scenarios, showing different objects and targets (such as people and vehicles) in different backgrounds (such as rural and urban areas). Due to the disorganized nature of the dataset, we selected 60 pairs of images for testing.

[0114] The RoadScene dataset contains 221 pairs of registered infrared and visible light images, covering a wide range of scenes including roads, vehicles, and pedestrians.

[0115] The FMB dataset includes 1500 pairs of strictly spatiotemporally registered infrared and visible light images, covering various extreme environments such as fog, heavy rain, and low light. The data was acquired using an intelligent binocular imaging system, ensuring high-precision alignment, and provides 14 categories of pixel-level semantic labels (such as roads, pedestrians, vehicles, and traffic signs), with a labeling coverage rate as high as 98.16%, making it particularly suitable for autonomous driving and semantic understanding tasks.

[0116] To objectively evaluate the performance of the model in this invention, we used six performance evaluation metrics, including information entropy (EN), standard deviation (SD), visual fidelity (VIF), sum of differential correlation (SCD), spatial frequency (SF), and average gradient (AG), which are defined as follows:

[0117]

[0118] Where L is the number of gray levels. This is the normalized histogram of the corresponding gray levels in the fused image. The higher the EN, the richer the information contained in the fused image.

[0119]

[0120] in, This represents the mean of the fused image. The human visual system is often more attracted to areas with high contrast, therefore, a higher SD fusion result will have better contrast.

[0121]

[0122] The higher the VIF value, the more the fusion result matches human visual perception.

[0123]

[0124] in, This represents the difference image between the fused image F and the source image X. A higher SCD means that the fused image contains more information from the source image.

[0125]

[0126] in, Indicates row frequency;

[0127] Indicates column frequency; a higher SF means richer edge and texture details.

[0128]

[0129] in, , A higher AG value in the fused image means it contains richer gradient information.

[0130] All implementations were performed on a workstation equipped with an NVIDIA RTX 4090 GPU. For infrared-visible light training, the input pairs were resized and randomly cropped into 128×128 image patches. The Adam optimizer was used with an initial learning rate of 1×10⁻⁻⁻⁶. 4 SGDFusion was trained for 200 epochs with a batch size of 8. The learning rate decayed by 0.5x every 20 epochs. Unless otherwise stated, all results were obtained using the same random seed and preprocessing to ensure fair comparison. Quantitative results are shown in Tables 1-4 below.

[0131] Table 1. Quantitative comparison of SGDFusion with other methods on MSRS

[0132]

[0133] Note: Bold numbers indicate the best performance value, and underlined numbers indicate the second-best performance value.

[0134] Table 2. Quantitative comparison of SGDFusion with other methods on TNO.

[0135]

[0136] Note: Bold numbers indicate the best performance value, and underlined numbers indicate the second-best performance value.

[0137] Table 3. Quantitative comparison of SGDFusion with other methods on Roadscene.

[0138]

[0139] Note: Bold numbers indicate the best performance value, and underlined numbers indicate the second-best performance value.

[0140] Table 4. Quantitative comparison of SGDFusion with other methods in FMB

[0141]

[0142] Note: Bold numbers indicate the best performance value, and underlined numbers indicate the second-best performance value.

[0143] The following conclusions can be drawn from the relevant experimental results:

[0144] (1) Traditional methods are prone to losing detailed information during downsampling, especially in low-contrast areas, and are also prone to introducing irrelevant information, thus affecting the overall performance. In contrast, this method effectively suppresses redundant information expression and enhances the ability to extract key features by introducing an attention mechanism.

[0145] (2) In the process of feature reconstruction, some methods lack effective integration of high-level semantic information and low-level detailed information, resulting in incomplete structural expression; while this method integrates semantic information and detailed information in a coordinated manner through a multi-level feature fusion mechanism, thereby improving the accuracy of structural expression.

[0146] (3) For fine-grained targets or weakly structured regions, existing methods often ignore their contextual information, resulting in discontinuous or incomplete results; while this method effectively enhances the integrity and continuity of the target region through structured feature transfer and fusion mechanism.

[0147] Table 1-4 presents the quantization results for the MSRS, TNO, RoadScene, and FMB datasets, covering evaluation scenarios both within the domain and across datasets. Overall, SGDFusion performs best on most metrics across all datasets, demonstrating strong robustness and generalization ability. Notably, SGDFusion achieves the best results on five out of six evaluation metrics, showcasing excellent overall performance. The only exception is the VIF metric on the TNO and RoadScene datasets, where the method ranks second by a narrow margin, but still significantly outperforms other comparative methods. This result indirectly confirms that the method maintains stable fusion performance under different imaging conditions and scene types, further validating its robustness and effectiveness.

[0148] In summary, this method demonstrates that a fusion strategy combining pixel-level decoupling and semantic guidance can effectively improve the structural consistency, detail fidelity, and semantic expressiveness of multimodal images. Furthermore, it provides general technical support for collaborative modeling of multimodal information in complex scenes and can serve subsequent tasks such as object detection and image segmentation.

[0149] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0150] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A semantically constrained dual-view decoupled image fusion method, characterized in that, include: S1: Obtain the registered multimodal image and its corresponding text description, and use the image-text pair as the fusion input; S2: Extract basic visual features from the input image and generate modality-specific features and shared features respectively; decouple the shared features and modality-specific features; S3: Perform semantic-level feature encoding on the text description, extract text semantic features, and generate a shared semantic representation; S4: Based on the modality-specific features, modality-shared features, modality-specific semantic representations, and shared semantic representations, perform cross-modal interaction and fusion between visual features and textual semantics: perform cross-modal modulation on the modality-specific semantic representations and the modality-specific visual features, and perform cross-modal alignment on the shared semantic representations and the modality-shared visual features to obtain semantically enhanced modality-specific features and semantically aligned shared features; S5: The semantically enhanced modality-specific features and semantically aligned shared features are fused by element-level addition, and the fused features are input into the reconstruction decoder to generate the final fused image.

2. The semantically constrained dual-view decoupled image fusion method according to claim 1, characterized in that, The step of extracting basic visual features from the input image and generating modality-specific features and shared features includes: Shallow visual features are extracted from images of each modality using a shared backbone encoder B(·). and : （1） The shallow features are input into the specific encoders of their respective modalities. and Generate modality-specific feature representations: （2） and These represent specific features in infrared and visible light encoding, respectively.

3. The semantically constrained dual-view decoupled image fusion method according to claim 2, characterized in that, The specific encoder of the mode and In this study, one approach uses a lightweight Transformer module, while the other employs a hybrid CNN-INN architecture. Input shallow features into the modality-sharing encoder Generate shared feature representations: （3） in, and These represent modal-shared features extracted from infrared and visible light inputs, respectively.

4. The semantically constrained dual-view decoupled image fusion method according to claim 2, characterized in that, A contrastive alignment-separation constraint is used to regulate the distribution of shared features and modality-specific features in the pixel-level feature space. The alignment operator (AO) encourages shared features to converge to a common subspace, while the separation operator (SO) suppresses unwanted correlations between modality-specific features and shared features, as well as between different modality-specific features. The formal representation is as shown in formula (4): （4） in, Represents the cosine similarity function; By jointly optimizing the alignment-separation constraints, the structure decouples pixel-level features.

5. The semantically constrained dual-view decoupled image fusion method according to claim 1, characterized in that, The semantic-level feature encoding: Text prompts for each modality are input into a fixed vision-language model for encoding, generating modality-specific semantic representations. and ; By multiplying the specific semantic features of the two modalities element by element, a shared semantic representation is obtained, as shown in formula (5): （5）。 6. The semantically constrained dual-view decoupled image fusion method according to claim 1, characterized in that, S4 specifically includes: setting up modality-specific interaction units (SpIUs) to refine modality-specific information, and adaptively adjusting features through semantically driven affine transformations. For visible modalities, the process is as follows: （6） in, and These are instance normalization and batch normalization, respectively. Shared Interaction Units (SIUs) are used to align modal shared structures, jointly model shared representations from different modalities, and perform semantic attention alignment guided by shared semantic representations. By calculating the correlation between visual and semantic features, cross-modal structures are consistently fused. The process is as follows: （7） in, Spatial features are reshaped into a sequence form, with K and V derived from shared semantic embeddings.

7. The semantically constrained dual-view decoupled image fusion method according to claim 1, characterized in that, The reconstruction decoder is built on the Transformer architecture and is used to recover image representations with global consistency and local detail. Channel compression is performed on the fused features through convolution operations, mapping information from different modalities to a unified feature space; Features are modeled by stacking Transformer modules, capturing both long-range dependencies and local structural relationships. Image details are gradually restored through convolutional layers, and the final fusion result is output after activation function. The process can be represented as: （8） in, This indicates a decoder based on Restormer. Indicates passage Convolution performs channel compression.

8. A semantically constrained dual-view decoupled image fusion system, characterized in that, include: The input module is used to acquire the registered multimodal images and their corresponding text descriptions; A pixel-level decoupling coding module is used to perform pixel-level decoupling coding on the multimodal image, decompose the visual representation into modality-specific features and modality-shared features, and apply decoupling constraints to the shared features and modality-specific features; The semantic-level encoding module is used to perform semantic-level encoding on the text description, extract modality-specific semantic representations using a frozen visual-language model, and obtain modality-invariant shared semantic representations by multiplying them element by element. The cross-modal interaction and fusion module is used to perform cross-modal interaction and fusion with the corresponding modal-specific visual features and modal-shared visual features based on the modal-specific semantic representation and the shared semantic representation, respectively, to obtain semantically enhanced modal-specific features and semantically aligned shared features; The reconstruction module is used to fuse the semantically enhanced modality-specific features with the semantically aligned shared features and input them into the reconstruction decoder to generate the final fused image.

9. The system according to claim 8, characterized in that, The pixel-level decoupling encoding module includes: A shared backbone encoder is used to extract shallow visual features from images of various modalities; Modality-specific encoders are used to generate modality-specific features from shallow visual features; A modality-sharing encoder is used to generate modality-sharing features from shallow visual features; The decoupling constraint unit is used to apply a contrastive alignment-separation constraint, which includes an alignment operator and a separation operator.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

Images