Unified visual and semantic learning based multi-modal image collaborative fusion method

Abstract

The application discloses a kind of unified visual and semantic learning multimodal image collaborative fusion method, it is related to infrared and visible light image fusion field. Including: based on sample scene visible light image and infrared image generate shared semantic representation;Through hierarchical interactive attention module and fusion specific enhancement module respectively to shared semantic representation and the visible light feature of visible light image is processed, generates the spatial domain feature of segmentation task and fusion task;Through frequency perception task router, shared semantic representation is processed to generate the frequency domain feature of segmentation task and fusion task;Based on the spatial domain feature and frequency domain feature corresponding to segmentation task and fusion task respectively, the task feature of segmentation task and fusion task is generated;Based on the task feature of segmentation task, preliminary segmentation image is obtained, based on the task feature of fusion task, preliminary fusion image is obtained, to realize in unified framework, visual fidelity, significant target expression and availability of high-level task are considered.

Description

Technical Field

[0001] This invention relates to the field of Infrared-Visible Image Fusion (IVIF), and more particularly to a multimodal image collaborative fusion method that unifies visual and semantic learning. Background Technology

[0002] Infrared and visible light image fusion is undergoing a significant transformation from visual enhancement tasks geared towards human perception to joint representation learning tasks that simultaneously serve scene understanding and machine perception. Under this shift in focus, the fusion result is no longer merely the final output image, but rather an intermediate information carrier for advanced visual tasks such as semantic segmentation and object detection. Infrared images can capture the salience of thermal radiation of targets under adverse conditions such as low light and smoke obscuration; visible light images provide rich texture, color, and structural details. This complementary advantage makes IVIF a key technology in scenarios such as autonomous driving perception, smart city surveillance, and nighttime security.

[0003] With the development of deep learning, IVIF has made significant breakthroughs in cross-modal interaction and attention modeling. However, when the fusion process is jointly modeled with high-level vision tasks, how to balance visual fidelity, salient object representation, and usability for high-level tasks within a unified framework remains a core challenge in the field. Summary of the Invention

[0004] Based on the above-mentioned technical problems, the present invention provides a multimodal image collaborative fusion method that unifies visual and semantic learning, aiming to overcome the above problems or at least partially solve them.

[0005] The first aspect of this invention provides a multimodal image collaborative fusion method that unifies visual and semantic learning, the method comprising: Visible light and infrared images of the sample scene are input into a dual-stream encoder for multi-scale feature extraction to generate shared semantic representations between the fusion and segmentation tasks. The shared semantic representation and the visible light features of the visible light image are processed by the hierarchical interactive attention module to generate the spatial domain features of the segmentation task. By fusing specific enhancement modules to process the shared semantic representation and the visible light features of the visible light image, spatial domain features of the fusion task are generated; The shared semantic representation is spectrally decomposed by a frequency-aware task router to obtain low-frequency band, mid-frequency band and high-frequency band; based on the low-frequency band and mid-frequency band, the frequency domain features of the segmentation task are generated, and based on the mid-frequency band and high-frequency band, the frequency domain features of the fusion task are generated. Based on the spatial domain and frequency domain features of the segmentation task, the task features of the segmentation task are generated; and based on the spatial domain and frequency domain features of the fusion task, the task features of the fusion task are generated. A preliminary segmented image is obtained by decoding the task features of the segmentation task using a segmentation decoder, and a preliminary fused image is obtained by decoding the task features of the fusion task using a fusion decoder.

[0006] In the multimodal image collaborative fusion method proposing this invention, multi-scale feature extraction is performed on visible light and infrared images. Then, deep cross-modal and cross-task interaction is conducted between the extracted visible light features of the visible light image and the infrared features of the infrared image to generate shared semantic representations between the fusion and segmentation tasks, integrating complementary information from infrared thermal saliency and visible light texture. Secondly, in the spatial domain, the shared semantic representations are guided to the task-specific objectives through two independent parameterized paths (i.e., the shared semantic representations are guided to the segmentation task in the spatial domain through a hierarchical interactive attention module to generate spatial domain features for the segmentation task; and the shared semantic representations are guided to the fusion task in the spatial domain through a fusion-specific enhancement module to generate spatial domain features for the fusion task). A stable structure is provided for these two independent parameterized paths based on visible light features. The system first establishes a structure and texture reference; then, through a frequency-aware task router, it implements explicit decoupling and directional routing in the frequency domain for different frequency bands sharing semantic representations according to task requirements, obtaining the frequency domain features of the segmentation task and the frequency domain features of the fusion task. The frequency domain features of each task are then injected into the spatial domain features of each task in the form of residuals to generate the task features of each task. This forms a complementary decoupling dimension with the spatial domain bias, realizing dual-domain task decoupling and fundamentally alleviating cross-task frequency resource competition. Finally, based on the task features of the fusion task and the task features of the segmentation task, preliminary fused images and preliminary segmented images are obtained. Thus, when jointly modeling with high-level visual tasks (such as semantic segmentation) during the fusion process, it achieves a unified framework that balances visual fidelity, salient target representation, and usability of high-level tasks, while improving the pixel-level reconstruction quality of the fused image and the semantic adaptability of downstream high-level visual tasks. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart illustrating the steps of a multimodal image collaborative fusion method that unifies visual and semantic learning, as shown in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a spatially perceptive semantic interaction module according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the processing of a cross-modal deep interaction module according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the framework of an infrared and visible light image collaborative fusion model that unifies visual and semantic learning, as shown in an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating a qualitative fusion comparison of VSFuse and state-of-the-art infrared-visible light image fusion methods on the MSRS dataset, as shown in an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating a qualitative fusion comparison between VSFuse and state-of-the-art infrared-visible light image fusion methods on the FMB dataset, as shown in an embodiment of the present invention. Figure 7 This is a schematic diagram illustrating a qualitative fusion comparison between VSFuse and state-of-the-art infrared-visible light image fusion methods on the M3FD dataset, as shown in an embodiment of the present invention. Figure 8 This is a visual comparison diagram of the output of VSFuse and nine of the most advanced or representative fusion algorithms, as shown in an embodiment of the present invention. Figure 9 This is a comparison chart of fused images, illustrating the comparison between VSFuse and nine fusion algorithms, using a retrained Vit-adapter to segment and visualize FMB test set images; Figure 10 This is a schematic diagram illustrating an embodiment of the present invention, showing the comparison of target detection performance of VSFuse with nine of the most representative fusion methods using a YOLOv7 network in MSRS selection scenario #855. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] Current fusion-segmentation joint modeling methods generally employ a shared feature space strategy: the extracted infrared and visible light features are integrated by a feature fusion module, and the fused image is output through a fusion decoder, with an additional task branch (segmentation or detection) for joint training. This invention, through careful study, has found that this method suffers from at least three core technical defects: First, the dual coupling of cross-modal and cross-task representations leads to implicit resource competition between tasks: related fusion-segmentation joint modeling methods (such as SegMiF, MRFS, SDCFusion, etc.) generally adopt a shared feature space strategy, but this invention finds that their shared representations lack explicit control mechanisms oriented towards task preferences. Furthermore, the thermal radiation saliency information of infrared images and the texture structure information of visible light images (cross-modal heterogeneous information), as well as the region discrimination features required for semantic segmentation and the visual fidelity features required for image fusion (cross-task heterogeneous requirements), coexist in the same shared representation space and undergo dual coupling, resulting in implicit resource competition between the fusion path and the segmentation path. The optimization of either path often comes at the cost of performance loss of the other path, causing the optimization objectives of the two paths to interfere with each other and making it difficult to improve them synergistically.

[0011] Second, the lack of a collaborative decoupling mechanism between the spatial and frequency domains means that the competition for frequency resources across tasks is not effectively mitigated. This invention finds that related methods mainly construct a bi-branch structure in the spatial domain, neglecting the potential of the frequency domain in cross-task feature allocation, i.e., ignoring the inherent correspondence between different frequency components and different task requirements. Low-frequency components carry large-scale semantic structure information, which is beneficial for segmentation; high-frequency components correspond to fine textures and edge details, which are beneficial for fusion visual quality. In other words, segmentation tasks rely on low- and mid-frequency regional semantic information, while fusion tasks rely on mid- and high-frequency edge texture information. Relying solely on the bi-branch structure in the spatial domain cannot achieve precise resource allocation for different tasks at the frequency level, leading to continuous re-coupling of features during propagation. That is, under pure spatial domain modeling, the two types of requirements are prone to deep entanglement of low-frequency semantics and high-frequency textures, and spatial domain bias cannot fundamentally prevent different frequency components from re-coupling in subsequent feature propagation.

[0012] Third, there is a semantic supervision gap between high-level semantic cognition and low-level fusion generation, lacking continuous feedback to the intermediate fusion process: This invention finds that the semantic guidance mechanisms of related methods mostly remain at the output-end supervision (such as joint loss) or staged shallow injection level, failing to form a position-sensitive and continuous feedback link between high-level segmentation semantics and intermediate fusion representations. This leads to the fact that the improvement of visual reconstruction quality and the improvement of semantic task performance are often asynchronous—the fused image performs well in pixel-level fidelity, but its support for downstream tasks such as semantic segmentation and object detection is still significantly insufficient.

[0013] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, multi-scale feature extraction is performed on visible light and infrared images, and cross-modal, cross-task deep interaction is conducted between the extracted visible light features of the visible light image and the infrared features of the infrared image to generate shared semantic representations between the fusion task and the segmentation task, thereby integrating complementary information from infrared thermal saliency and visible light texture. Secondly, in the spatial domain, the shared semantic representations are guided to the task-specific objectives through two independent parameterized paths (i.e., the shared semantic representations are guided to the segmentation task in the spatial domain through a hierarchical interactive attention module to generate spatial domain features for the segmentation task, and the shared semantic representations are guided to the fusion task in the spatial domain through a fusion-specific enhancement module to generate spatial domain features for the fusion task), and stable structural and texture references are provided for these two independent parameterized paths based on visible light features. Then, through… The frequency-aware task router explicitly decouples and routes different frequency bands with shared semantic representations according to task requirements in the frequency domain, obtaining the frequency domain features of the segmentation task and the frequency domain features of the fusion task. The frequency domain features of each task are then injected into the spatial domain features of each task in the form of residuals to generate the task features of each task. This forms a complementary decoupling dimension with the spatial domain bias, realizing dual-domain task decoupling and fundamentally alleviating cross-task frequency resource competition. Finally, based on the task features of the fusion task and the task features of the segmentation task, preliminary fused images and preliminary segmented images are obtained. Thus, when jointly modeling with high-level visual tasks (such as semantic segmentation) during the fusion process, it achieves a unified framework that balances visual fidelity, salient target representation, and usability of high-level tasks, while improving the pixel-level reconstruction quality of the fused image and the semantic adaptability of downstream high-level visual tasks.

[0014] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal image collaborative fusion method that unifies visual and semantic learning, as shown in an embodiment of the present invention. Figure 1 As shown, the multimodal image collaborative fusion method with unified vision and semantic learning provided in this embodiment includes at least the following steps: Step S11: Input the visible light image and infrared image of the sample scene into the dual-stream encoder for multi-scale feature extraction to generate a shared semantic representation between the fusion task and the segmentation task.

[0015] In this embodiment, image pairs can be acquired for the sample scene, including a visible light image and an infrared image of the sample scene. This embodiment proposes a unified vision and semantic learning-based infrared-visible light image collaborative fusion model. This model jointly performs image fusion and semantic segmentation, and includes at least: a two-stream encoder, a hierarchical interactive attention module, a fusion-specific enhancement module, a frequency-aware task router, a segmentation decoder, and a fusion decoder.

[0016] First, the visible light and infrared images of the sample scene are input into a dual-stream encoder for multi-scale feature extraction, resulting in visible light features of the visible light image and infrared features of the infrared image. Then, the visible light features and infrared features are subjected to deep cross-modal and cross-task interaction to generate a shared semantic representation between the fusion task and the segmentation task. This shared semantic representation integrates complementary information of infrared thermal saliency and visible light texture.

[0017] Furthermore, in another embodiment, the dual-stream encoder of this embodiment can be replaced with other mainstream backbone networks, including but not limited to: Vision Transformer (ViT), Swing Transformer, ResNet series, EfficientNet, etc. These backbone networks have different receptive fields and computational complexities, and can be selected according to the accuracy-speed requirements of the actual application scenario.

[0018] Step S12: Process the shared semantic representation and the visible light features of the visible light image through the hierarchical interactive attention module to generate the spatial domain features of the segmentation task.

[0019] In this embodiment, the hierarchical interactive attention module and the fusion-specific enhancement module together form the task-conditional structure bias layer in the infrared and visible light image collaborative fusion model of unified vision and semantic learning. This task-conditional structure bias layer is used to: guide the shared semantic representation to the task-specific target through two independent parameterized paths in the spatial domain, and provide stable structural and texture references for the two independent parameterized paths through visible light features.

[0020] Specifically, for segmentation tasks, the hierarchical interactive attention module can process the shared semantic representation and visible light features of the visible light image to generate spatial domain features for the segmentation task, thereby providing spatial domain features with regional consistency and boundary recognition capabilities for the segmentation task.

[0021] Step S13: Process the shared semantic representation and the visible light features of the visible light image by fusing specific enhancement modules to generate the spatial domain features of the fusion task.

[0022] In this embodiment, for the fusion task, the visible light features of the shared semantic representation and the visible light image can be processed by fusing specific enhancement modules to generate spatial domain features for the fusion task, thereby providing spatial domain features that preserve high-frequency texture and visual fidelity for the fusion task.

[0023] Step S14: Perform spectral decomposition on the shared semantic representation using a frequency-aware task router to obtain low-frequency band, mid-frequency band, and high-frequency band; generate frequency domain features of the segmentation task based on the low-frequency band and mid-frequency band, and generate frequency domain features of the fusion task based on the mid-frequency band and high-frequency band.

[0024] In this embodiment, for the frequency domain, the shared semantic representation can first be decomposed into low-frequency, mid-frequency, and high-frequency bands using a frequency-aware task router. The low-frequency band carries structural information, the mid-frequency band contains semantic boundary components, and the high-frequency band corresponds to texture and edge details.

[0025] Then, considering that the segmentation task relies on low-to-mid-frequency regional semantic information, frequency domain features for the segmentation task are generated based on at least the low-frequency and mid-frequency bands; and considering that the fusion task relies on mid-to-high-frequency edge texture information, frequency domain features for the fusion task are generated based on at least the mid-frequency and high-frequency bands. Thus, this embodiment reorganizes different frequency bands according to task preferences, achieving explicit decoupling and directional routing of different frequency bands in the frequency domain according to task requirements.

[0026] Step S15: Generate task features for the segmentation task based on the spatial domain features and frequency domain features of the segmentation task, and generate task features for the fusion task based on the spatial domain features and frequency domain features of the fusion task.

[0027] In this embodiment, the frequency domain features of the obtained segmentation task can be used as an auxiliary increment and injected into the spatial domain features of the segmentation task as a residual to obtain the task features of the segmentation task. Similarly, the frequency domain features of the obtained fusion task can be used as an auxiliary increment and injected into the spatial domain features of the fusion task as a residual to obtain the task features of the fusion task. This ensures that frequency domain decoupling provides a frequency dimension supplement to the spatial domain bias, rather than a replacement.

[0028] Step S16: Decode the task features of the segmentation task using a segmentation decoder to obtain a preliminary segmentation image; and decode the task features of the fusion task using a fusion decoder to obtain a preliminary fusion image.

[0029] In this embodiment, the task features of the segmentation task can be input into the segmentation decoder, and the segmentation decoder can decode the task features of the segmentation task to obtain a preliminary segmented image as the preliminary segmentation result of the semantic segmentation task; and the task features of the fusion task can be input into the fusion decoder, and the fusion decoder can decode the task features of the fusion task to obtain a preliminary fused image as the preliminary fusion result of the image fusion task.

[0030] In this embodiment, multi-scale feature extraction is performed on visible light and infrared images, and cross-modal and cross-task deep interaction is conducted on the extracted visible light features of the visible light image and the infrared features of the infrared image to generate shared semantic representations between the fusion task and the segmentation task, thereby integrating complementary information of infrared thermal saliency and visible light texture. Secondly, in the spatial domain, the shared semantic representations are guided to the task-specific targets through two independent parameterization paths (i.e., the shared semantic representations are guided to the segmentation task in the spatial domain through a hierarchical interactive attention module to generate spatial domain features for the segmentation task, and the shared semantic representations are guided to the fusion task in the spatial domain through a fusion-specific enhancement module to generate spatial domain features for the fusion task), and stable structural and texture references are provided for these two independent parameterization paths based on visible light features. Then, through... The frequency-aware task router explicitly decouples and routes different frequency bands with shared semantic representations according to task requirements in the frequency domain, obtaining the frequency domain features of the segmentation task and the frequency domain features of the fusion task. The frequency domain features of each task are then injected into the spatial domain features of each task in the form of residuals to generate the task features of each task. This forms a complementary decoupling dimension with the spatial domain bias, realizing dual-domain task decoupling and fundamentally alleviating cross-task frequency resource competition. Finally, based on the task features of the fusion task and the task features of the segmentation task, preliminary fused images and preliminary segmented images are obtained. Thus, when jointly modeling with high-level visual tasks (such as semantic segmentation) during the fusion process, it achieves a unified framework that balances visual fidelity, salient target representation, and usability of high-level tasks, while improving the pixel-level reconstruction quality of the fused image and the semantic adaptability of downstream high-level visual tasks.

[0031] In one embodiment, based on the above embodiments, the dual-stream encoder performs multi-scale feature extraction (such as four-scale feature extraction) on the visible light image and the infrared image respectively to obtain multi-scale visible light features. and multi-scale infrared features Then, for the visible light characteristics at each scale... and infrared features Generate shared semantic representations at this scale. And through the hierarchical interactive attention module, a shared semantic representation of this scale is obtained. and visible light characteristics at this scale Processing is performed to generate spatial domain features for the segmentation task at this scale. By fusing shared semantic representations of this scale using specific enhancement modules. and visible light characteristics at this scale Processing is performed to generate spatial domain features for the fusion task at this scale. Shared semantic representation of this scale via frequency-aware task router Spectral decomposition is performed to obtain the low-frequency, mid-frequency, and high-frequency bands at this scale. Based on the low-frequency and mid-frequency bands at this scale, frequency domain features for the segmentation task at this scale are generated; and based on the mid-frequency and high-frequency bands at this scale, frequency domain features for the fusion task at this scale are generated. Finally, based on the spatial domain and frequency domain features of the segmentation task at this scale, task features for the segmentation task at this scale are generated. Furthermore, based on the spatial and frequency domain features of the fusion task at this scale, task features for the fusion task at this scale are generated. Finally, the segmentation decoder is based on the task features of the multi-scale segmentation task. This generates an initial segmented image Z, and a fusion decoder based on task features from a multi-scale fusion task. Generate preliminary segmented image .

[0032] In conjunction with the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, in addition to the steps described above, steps S21 to S26 may also be included: Step S21: Process the read-only prior corresponding to the preliminary segmentation image through the semantic prior path in the spatially aware semantic interaction module to obtain the semantic boundary; generate the fusion-segmentation injection strength and the segmentation-fusion injection strength based on the task features of the segmentation task, the task features of the fusion task, and the semantic boundary.

[0033] In this embodiment, the infrared and visible light image collaborative fusion model that unifies visual and semantic learning further includes a spatially perceptual semantic interaction module. This module is used to establish bidirectional knowledge interaction between segmentation semantics and fusion representation, and continuously injects high-level semantic boundary priors into the intermediate fusion path.

[0034] In this embodiment, after the segmentation decoder outputs the preliminary segmentation image, a stop-gradient process can be performed on the preliminary segmentation image to obtain a read-only prior. This prevents unstable segmentation noise in the early training stages from perturbing the parameter updates of the main decoder through the backflow link. Then, the read-only prior corresponding to the preliminary segmentation image is input into the semantic prior path in the spatially aware semantic interaction module. The read-only prior is processed through the semantic prior path to obtain the semantic boundary (i.e., the semantic boundary prior). This semantic boundary is used to quantify the boundary strength distribution of the semantic region. In an optional example, the semantic prior path can extract a pixel-wise maximum class probability map P from the read-only prior, and then extract a normalized semantic boundary B from the pixel-wise maximum class probability map P using the Sobel operator.

[0035] Then, based on the task features of the segmentation task, the task features of the fusion task, and the semantic boundary, the fusion-segmentation injection strength and the segmentation-fusion injection strength are generated. In an optional embodiment, the task features of the segmentation task, the task features of the fusion task, and the semantic boundary can be concatenated and fed into a modulation network M consisting of three 3×3 convolutional layers to predict the pixel-by-pixel cross-task injection strength: fusion-segmentation injection strength. (Injection intensity in the fusion-segmentation direction) and segmentation-fusion injection intensity (Injection intensity in the control segmentation → fusion direction).

[0036] In this embodiment, the introduction of semantic boundary B enables the modulation network to perceive the spatial difference between the semantic target region and the background region, thereby adaptively enhancing the cross-task injection intensity in the semantically significant target region and maintaining a low injection amplitude in the flat background region.

[0037] Step S22: Process the read-only prior corresponding to the preliminary segmented image through the segmentation branch in the spatially aware semantic interaction module to obtain segmentation knowledge.

[0038] In this embodiment, the segmentation branch in the spatially perceptual semantic interaction module, which corresponds to the initial segmented image, can be processed by the segmentation task-specific projection function in the segmentation branch to obtain segmentation knowledge.

[0039] Step S23: Process the task features of the fusion task through the fusion branch in the spatially aware semantic interaction module to obtain fusion knowledge.

[0040] In this embodiment, the task features of the fusion task can be input into the fusion branch of the spatially aware semantic interaction module. The task features of the fusion task can be processed by the fusion task-specific projection function in the fusion branch to obtain fusion knowledge.

[0041] Step S24: Supplement the segmentation knowledge with the fusion knowledge to obtain fusion-segmentation knowledge. Based on the fusion-segmentation knowledge, the fusion-segmentation injection strength, and the task characteristics of the segmentation task, obtain the enhanced task characteristics of the segmentation task.

[0042] Step S25: Use the segmentation knowledge to supplement the fusion knowledge to obtain segmentation-fusion knowledge. Based on the segmentation-fusion knowledge, the segmentation-fusion injection strength, and the task characteristics of the fusion task, obtain the enhanced task characteristics of the fusion task.

[0043] In this embodiment, bidirectional injection supplements the segmentation knowledge with fusion knowledge to obtain fusion-segmentation knowledge, and segmentation knowledge supplements the fusion knowledge to obtain segmentation-fusion knowledge. Then, based on the fusion-segmentation knowledge, the fusion-segmentation injection strength, and the task features of the segmentation task, enhanced task features for the segmentation task are obtained. Thus, the segmentation path receives fusion texture supplementation, providing richer discriminative support for semantic target regions. Furthermore, based on the segmentation-fusion knowledge, the segmentation-fusion injection strength, and the task features of the fusion task, enhanced task features for the fusion task are obtained. Thus, the fusion path receives semantic region structural constraints, enhancing the representational integrity and information fidelity of salient target regions.

[0044] Step S26: Decode the enhanced task features of the segmentation task using the segmentation decoder to obtain the final segmented image; and decode the enhanced task features of the fusion task using the fusion decoder to obtain the final fused image.

[0045] In this embodiment, the enhanced task features of the segmentation task can be input into the segmentation decoder, and the segmentation decoder can decode the enhanced task features of the segmentation task to obtain the final segmented image as the final segmentation result of the semantic segmentation task; and the enhanced task features of the fusion task can be input into the fusion decoder, and the fusion decoder can decode the enhanced task features of the fusion task to obtain the final fused image as the final fusion result of the image fusion task.

[0046] In this embodiment, a bidirectional continuous semantic feedback mechanism based on semantic boundary prior is proposed: a position-sensitive bidirectional knowledge interaction is designed, with the segmentation boundary prior (i.e., semantic boundary) as spatial guidance, to establish a continuous feedback link between segmentation semantics and intermediate fusion representation. At the same time, the quality of semantic prior is improved by reverse regularization of fusion texture, which alleviates the generation-cognition semantic supervision gap, thereby significantly improving the asynchronous phenomenon between the improvement of visual reconstruction quality and the improvement of semantic task performance.

[0047] Specifically, in one optional example, the spatially aware semantic interaction module is only used at the full-resolution scale ( This involves performing cross-task interaction, injecting semantic boundary priors into the segmentation path to constrain the target region representation, and simultaneously feeding back fused textures into the segmentation path to supplement boundary discrimination evidence. For example... Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the structure of a spatially aware semantic interaction module according to an embodiment of the present invention. Figure 2 In this module, the spatially aware semantic interaction module constructs semantic boundary priors using segmentation prediction and achieves bidirectional interaction between segmented semantics and fused textures at the full-resolution layer (l=1) through spatial modulation. Figure 2 The bottom part is the semantic prior path, the middle part is the segmentation branch, and the top part is the fusion branch.

[0048] In the semantic prior path, from the read-only prior... Extract the pixel-wise maximum class probability map P: Then, the normalized semantic boundary B∈R(1×H×W) is extracted from the pixel-by-pixel maximum class probability map P using the Sobel operator: ; Where C is the number of semantic categories, c∈[1,C], It uses a 3×3 Sobel convolution kernel. To normalize to [0,1], This is the numerical stability constant, with a value of 1 × 10⁻⁶. -6 This is used to prevent division by zero errors when the gradient magnitude is zero.

[0049] Then, the task features of the segmentation task will be... Task characteristics of fusion tasks After concatenating the semantic boundary B, the signal is fed into the modulation network M (three layers of 3×3 convolution + BN + ReLU, with a final output of 2 channels + Sigmoid) to predict the fusion-segmentation injection intensity. and segmentation-fusion injection intensity : .

[0050] In the split branch, the task-specific projection function is used for splitting. (3×3 convolution + BN + ReLU + 1×1 convolution, input C channels, output d channels, mapping segmentation semantics to the fused feature space) based on this read-only prior. Processing is performed to obtain segmentation knowledge. ; Reutilize integrated knowledge Knowledge of segmentation To supplement this, we obtain fusion-segmentation knowledge. Finally, based on fusion-segmentation knowledge Fusion-segmentation injection intensity and the task characteristics of segmentation tasks To obtain enhanced task features for the segmentation task. : .

[0051] In the fusion branch, through the fusion task-specific projection function (and With the same structure, both input and output are d-channel, and the fused texture is projected onto the segmented feature space, this describes the task features of the fusion task. Processing yields integrated knowledge. ; and then utilize segmentation knowledge Knowledge integration To supplement this, we obtain segmentation-fusion knowledge. Finally, based on segmentation-fusion knowledge Segmentation-fusion injection intensity and the task characteristics of fusion tasks To obtain enhanced task features for the fusion task : .

[0052] In conjunction with any of the above embodiments, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, step S11 may specifically include steps S31 to S36: Step S31: Based on the visible light characteristics of the visible light image, obtain the visible light Q value, visible light K value, and visible light V value; based on the infrared characteristics of the infrared image, obtain the infrared Q value, infrared K value, and infrared V value; based on the visible light V value and the infrared V value, determine the shared V value.

[0053] In this embodiment, a dual-stream encoder can be used to extract multi-scale features from the visible light image and infrared image of the sample scene, respectively, to obtain the visible light features of the visible light image and the infrared features of the infrared image. In this embodiment, the infrared-visible light image collaborative fusion model that unifies visual and semantic learning also includes a cross-modal deep interaction module. Considering that the aggregation methods of symmetrical stitching and element-wise addition assume equal contributions from both modalities, which cannot characterize the directional complementary relationship between infrared saliency and visible light texture, this embodiment designs a cross-modal deep interaction module. This module is used to explicitly distinguish between intra-modal enhancement and cross-modal directional transfer in a unified interaction space through four-way directed attention and learnable gating.

[0054] In this embodiment, the cross-modal depth interaction module can obtain the visible light Q-value, visible light K-value, and visible light V-value based on the visible light features of the visible light image; and the infrared Q-value, infrared K-value, and infrared V-value based on the infrared features of the infrared image. In a specific example, the visible light features and infrared features can be bicubic resampled to a 32×32 low resolution, and the query (Q), key (K), and value (V) can be generated through three sets of independent 3×3 convolutional projections, and finally flattened into... : ; in, The three projections are independent convolutions.

[0055] Then, a shared V value is generated based on the visible light V value and the infrared V value. In an optional instance, the shared V value... Visible light V value With infrared V value The sum of .

[0056] Step S32: Based on the infrared intramodal enhancement branch, process the infrared Q value, infrared K value, and shared V value to obtain the infrared attention result; and based on the visible light intramodal enhancement branch, process the visible light Q value, visible light K value, and shared V value to obtain the visible light attention result.

[0057] In this embodiment, the cross-modal deep interaction module constructs four directed attention branches, including: intra-modal enhancement branches (infrared intra-modal enhancement branch and visible light intra-modal enhancement branch, to achieve self-enhancement of each modality in a shared context) and cross-modal transfer branches (visible light-infrared transfer branch and infrared-visible light transfer branch, to achieve directional cross-modal transfer of infrared saliency and visible light texture).

[0058] Among these, the infrared Q-value can be improved based on the intramodal enhancement branch of the infrared mode. Infrared K-value And shared V value The infrared attention result is obtained through processing. ;For example, Furthermore, the visible light Q-value can be improved based on the enhancement branch within the visible light mode. Visible light K value and shared V value The process is performed to obtain the visible light attention result. ;For example, Among them, a low-resolution spatial attention operator is introduced. , Normalize along the token dimension.

[0059] In another alternative example, the four-way directed attention can be replaced with: cross-attention mechanism, sparse attention, linear attention, or a Mamba-based state-space model (SSM). The computational resolution of the attention (32×32 in the example above) can also be adjusted.

[0060] Step S33: Based on the visible light-infrared transmission branch, process the infrared Q value, visible light K value, and visible light V value to obtain the visible light-infrared attention result; and, based on the infrared-visible light transmission branch, process the visible light Q value, infrared K value, and infrared V value to obtain the infrared-visible light attention result.

[0061] In this embodiment, the infrared Q value can be determined based on the visible light-infrared transmission branch. Visible light K value and visible light V value The process is performed to obtain the visible light-infrared attention result. ;For example, Furthermore, the Q-value of visible light can be determined based on the infrared-visible light transmission branch. Infrared K-value and infrared V value The infrared-visible light attention results were obtained through processing. ;For example, .in, A learnable scalar initialized to zero is used to ensure stability during the initial training phase.

[0062] Step S34: Based on the infrared attention results, visible light attention results, visible light-infrared attention results, and infrared-visible light attention results, obtain the shared attention features.

[0063] In this embodiment, shared attention features can be obtained based on infrared attention results, visible light attention results, visible light-infrared attention results, and infrared-visible light attention results. In an optional example, the infrared attention results, visible light attention results, visible light-infrared attention results, and infrared-visible light attention results can be concatenated to obtain shared attention features. .

[0064] Step S35: Obtain infrared local features based on the infrared features of the infrared image; and obtain visible light local features based on the visible light features of the visible light image.

[0065] In this embodiment, the cross-modal depth interaction module can perform spatial compensation based on the infrared features of the infrared image to obtain infrared local features; and perform spatial compensation based on the visible light features of the visible light image to obtain visible light local features.

[0066] Step S36: Based on infrared local features, visible light local features, and shared attention features, obtain the shared semantic representation.

[0067] In this embodiment, the cross-modal deep interaction module can concatenate infrared local features, visible light local features, and shared attention features, and then perform fusion convolution to obtain the shared semantic representation output by the cross-modal deep interaction module.

[0068] In this embodiment, a cross-modal deep interaction mechanism with four-way directed attention is proposed: four-way directed attention with intramodal enhancement and cross-modal directional transfer is constructed. Through learnable gating, the precise complementary aggregation of infrared thermal saliency and visible light texture is achieved in a unified interaction space, overcoming the limitation of traditional symmetric aggregation methods that cannot characterize directional complementary relationships.

[0069] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the processing of a cross-modal deep interaction module according to an embodiment of the present invention. Figure 3 In this module, CDIM (Cross-Modal Deep Interaction Module) is used to perform intra-modal enhancement and cross-modal directional transfer of visible light and infrared features at a unified scale. It also combines gating and spatial compensation to generate shared semantic representations. .

[0070] exist Figure 3 In terms of scale The dual-stream encoder outputs visible light and infrared features. . Figure 3 The CAM on the right represents the four directed attention branches, ultimately yielding the shared attention features of the CAM output. .

[0071] Shared attention features plastic surgery After feature map, splicing convolution Generate a shared gated graph with Sigmoid : After gated modulation and bicubic upsampling, the original resolution is restored, and the outputs of each branch are obtained in the form of residuals, preserving the local structure of the original modal features.

[0072]

[0073] in, Indicates bicubic upsampling to the original resolution , When it is vi, For ir; When it is ir, For vi.

[0074] Spatial compensation and shared representation output: To recover lost local details at a fixed resolution, a method is introduced... convolution Perform original resolution space compensation: Global interaction results are processed by channel-compressed convolution. After integration, it is concatenated with local compensation features and then subjected to fusion convolution. Obtain shared semantic representation : The resulting shared semantic representation, while preserving the local structure, integrates intramodal context and cross-modal directional complementary information, serving as a unified input for subsequent task differentiation.

[0075] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, step S12 specifically includes steps S41 to S43: Step S41: Process the shared semantic representation and the visible light features of the visible light image through the cross-feature semantic mapping submodule to generate a structure-guided response.

[0076] In this embodiment, the hierarchical interactive attention module includes a cross-feature semantic mapping submodule and a channel-space joint attention submodule. Segmentation tasks require features to possess regional consistency and boundary recognition capabilities. The hierarchical interactive attention module applies structural bias to shared semantic representations through cross-feature semantic mapping and channel-space joint attention.

[0077] After the shared semantic representation and visible light features of the visible light image are input into the hierarchical interactive attention module, the shared semantic representation and visible light features of the visible light image can be processed through the cross-feature semantic mapping submodule to generate a structure-guided response.

[0078] In one specific embodiment, the cross-feature semantic mapping submodule can share semantic representations. and visible light characteristics After 1x1 convolution Each channel is compressed to d / 2 channels and then concatenated using two 1×1 convolutional layers. Generate structured guidance response : Where S is a learnable scaling factor, initialized to 0.3, which controls the injection strength guided by the structure.

[0079] Step S42: Apply channel attention and spatial attention to the shared semantic representation sequentially through the channel-space joint attention submodule to obtain channel attention weights and spatial attention weights in sequence, and process the shared semantic representation based on the channel attention weights and spatial attention weights to obtain the channel-space joint attention result.

[0080] In this embodiment, after the shared semantic representation and the visible light features of the visible light image are input into the hierarchical interactive attention module, channel attention and spatial attention can be applied to the shared semantic representation in sequence through the channel-space joint attention submodule to obtain the channel attention weight and spatial attention weight in sequence. Based on the channel attention weight and the spatial attention weight, the shared semantic representation is processed to obtain the channel-space joint attention result.

[0081] In an optional example, the channel-space joint attention submodule can address this shared semantic representation. First, apply SE channel attention (bottleneck dimensionality reduction ratio r=4) to obtain the channel attention weights. : Then based on channel attention weights This shared semantic representation Apply 7×7 spatial attention and generate spatial attention weights. : Finally, based on the channel attention weights... and the attention weight of this space Shared semantic representation The process is performed to obtain the channel-space joint attention results. : .

[0082] Step S43: Based on the channel-space joint attention results, structure-guided response, and the shared semantic representation, generate the spatial domain features of the segmentation task.

[0083] In this embodiment, the hierarchical interactive attention module can apply structural bias to the shared semantic representation based on the channel-space joint attention result and the structure-guided response, thereby generating spatial domain features for the segmentation task.

[0084] In an alternative implementation, the results can be based on channel-space joint attention. and structure-guided response For shared semantic representations Apply structural bias to generate spatial domain features for segmentation tasks. That is, the results of channel-space joint attention. and structure-guided response By fusing the residuals, the spatial domain features of the segmentation task are obtained. : .in, It is a scale-dependent learnable scalar, initialized to 0.2 and constrained to [0, 0.8], which controls the contribution weights of the two sub-modules to the residual output.

[0085] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, step S13 may specifically include steps S51 to S53: Step S51: Extract the visible light features of the visible light image using the texture extraction submodule to obtain the texture extraction result.

[0086] In this embodiment, the fusion task requires the preservation of high-frequency texture and visual fidelity. A specific enhancement module is used to selectively compensate for high-frequency details that are easily smoothed during cross-modal interactions. This specific enhancement module includes a texture extraction submodule and a global average pooling channel gating submodule.

[0087] In this process, after the shared semantic representation and visible light features of the visible light image are fused with a specific enhancement module, the visible light features of the visible light image are extracted using a texture extraction submodule to obtain the texture extraction result. In an optional example, the texture extraction submodule... (3×3 convolution + PreLU) from visible light features The texture extraction result is obtained from the extraction. : .

[0088] Step S52: Perform channel filtering on the shared semantic representation through the global average pooling channel gating submodule to obtain the channel filtering result.

[0089] In this embodiment, after the shared semantic representation and the visible light features of the visible light image are fused with a specific enhancement module, the shared semantic representation is then subjected to channel filtering by a global average pooling channel gating submodule to obtain the channel filtering result. In an optional example, the global average pooling channel gating submodule can sequentially apply GAP, 1X1 convolution, and Sigmoid to the shared semantic representation. Perform channel filtering and obtain the channel filtering results. : .

[0090] Step S53: Based on the shared semantic representation, texture extraction results, and channel filtering results, the spatial domain features of the fusion task are obtained.

[0091] In this embodiment, fusing specific enhancement modules can obtain the spatial domain features of the fusion task based on shared semantic representations, texture extraction results, and channel filtering results. In a specific optional example, the texture extraction results... and channel filtering results Both can modulate shared semantic representations in the form of residuals. The spatial domain features of the fusion task are obtained. : .in, Both settings are empirically configured to control the contribution ratios of texture compensation and channel filtering, respectively. This design in this embodiment selectively recovers texture and detail information that significantly impacts visual quality while preserving existing task responses in the shared semantic representation.

[0092] In other words, in one embodiment, the hierarchical interaction attention module and the fusion-specific enhancement module will share semantic representations. These are mapped to task-specific representations for segmentation and fusion, respectively: Among them, visible light characteristics This provides stable structural and texture references for these two paths; HIA is a hierarchical interactive attention module, and FSE is a fusion-specific enhancement module. After processing by HIA and FSE, and In the spatial domain, structural biases are formed to cater to different tasks, providing task-specific input representations for subsequent frequency domain routing.

[0093] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, step S14 may specifically include steps S61 to S64: Step S61: Through the frequency band decomposition submodule, the shared semantic representation is converted to the frequency domain to obtain the spectrum, and the spectrum is decomposed using a mask to obtain the low frequency band, mid frequency band and high frequency band.

[0094] However, spatial domain bias cannot prevent different frequency components from recoupling during subsequent feature propagation. This embodiment considers the need for further explicit decoupling in the frequency domain. Specifically, in this embodiment, a frequency-aware task router performs spectral decomposition on the shared semantic representation, recombining different frequency bands according to task preferences and injecting them into the differentiated task features as residuals, thus complementing the spatial domain bias. The frequency-aware task router includes at least a frequency band decomposition submodule, which is used to perform spectral decomposition on the shared semantic features.

[0095] This embodiment can use a frequency band decomposition submodule to convert the shared semantic representation to the frequency domain, obtaining the spectrum corresponding to the shared semantic representation. Then, the spectrum corresponding to the shared semantic representation is decomposed using a mask to obtain the low-frequency band, mid-frequency band, and high-frequency band.

[0096] In an optional specific example, shared semantic representations can be used. Perform a two-dimensional FFT and center the representation to transform the shared semantic representation to the frequency domain, thus obtaining the spectrum corresponding to the shared semantic representation. : Then, define the normalized radial frequency ρ: Where u and v are the row and column coordinates of a pixel in the spectrum, respectively. , These represent the height and width of the shared semantic representation at scale l, respectively. The numerical stability constant (with a value of 1 × 10⁻⁶) -6 (This is used to prevent the denominator from being zero). Next, a continuously differentiable soft bandpass mask is constructed based on the normalized radial frequency ρ. :

[0097] in, Control the steepness of the transition. Based on this, the spectrum corresponding to the shared semantic representation will be... Divided into three functional frequency bands: low frequency band It carries and integrates large-scale structural information for fusion tasks; mid-frequency band It includes semantic boundary components; high-frequency band This corresponds to texture and edge details. The above division is based on the functional attribution of frequency components in dense prediction; The extremely high frequency components do not participate in routing because they contain a lot of noise.

[0098] Furthermore, in another example, the two-dimensional FFT spectral decomposition can be replaced by other frequency analysis tools such as Discrete Cosine Transform (DCT), Wavelet Transform (e.g., DWT), and Learnable Filter Banks. (Frequency band delineation boundaries (in the above example)) , , It can also be adjusted according to the specific task.

[0099] Step S62: Convert the low-frequency band, mid-frequency band, and high-frequency band to the spatial domain to obtain low-frequency band features, mid-frequency band features, and high-frequency band features; based on the low-frequency band features, obtain the shared low-frequency features.

[0100] In this embodiment, after obtaining the low-frequency band, mid-frequency band, and high-frequency band, these bands are transformed to the spatial domain to obtain low-frequency band features, mid-frequency band features, and high-frequency band features. In an optional example, each frequency band can be inversely transformed to recover the spatial domain to obtain the features of each frequency band. : .

[0101] Then, based on the low-frequency band features, shared low-frequency features are obtained. These shared low-frequency features are used to simultaneously inject into both task paths, preserving a consistent low-frequency structural reference base for segmentation and fusion. For example, it can be... Mapping function for low-frequency band features Processing is performed to obtain shared low-frequency features. : ;in, It is a 1×1 convolution + BN + ReLU.

[0102] Step S63: Based on the low-frequency band features and the mid-frequency band features, obtain the intermediate frequency domain features of the segmentation task; based on the shared low-frequency features and the intermediate frequency domain features of the segmentation task, obtain the frequency domain features of the segmentation task.

[0103] In this embodiment, frequency bands are combined according to task preferences, and the segmentation path uses low frequency + mid frequency: based on the low frequency band features and mid frequency band features, the intermediate frequency domain features of the segmentation task are obtained, and then the shared low frequency features are injected into the intermediate frequency domain features of the segmentation task to obtain the frequency domain features of the segmentation task.

[0104] In an optional example, this can be achieved using a lightweight mapping function. For low frequency band characteristics and mid-frequency characteristics Depthwise separable convolution is performed to obtain intermediate frequency domain features for the segmentation task. : .in, It consists of channel-wise 3×3 convolution + BN + ReLU + 1×1 pointwise convolution.

[0105] Furthermore, in another alternative example, the lightweight mapping function of the depthwise separable convolution can be replaced with standard convolution, MLP, or attention aggregation. The spatial modulation network (three-layer convolution) in the spatially aware semantic interaction module can be replaced with a lightweight MLP or a single-layer convolution. The projection function in bidirectional injection can be replaced with a linear transformation or other learnable mapping.

[0106] Step S64: Based on the mid-frequency band features and high-frequency band features, obtain the intermediate frequency domain features of the fusion task; based on the shared low-frequency features and the intermediate frequency domain features of the fusion task, obtain the frequency domain features of the fusion task.

[0107] In this embodiment, the fusion path uses mid-frequency + high-frequency: based on the mid-frequency band features and high-frequency band features, the intermediate frequency domain features of the fusion task are obtained, and then the shared low-frequency features are injected into the intermediate frequency domain features of the fusion task to obtain the frequency domain features of the fusion task.

[0108] In an optional example, this can be achieved using a lightweight mapping function. For high frequency band characteristics and mid-frequency characteristics Depthwise separable convolution is performed to obtain intermediate frequency domain features for the fusion task. : .in, The process consists of channel-wise 3×3 convolution + BN + ReLU + 1×1 pointwise convolution. Empirical settings are used to suppress high-frequency overcompensation. Simultaneously injecting two paths preserves a consistent low-frequency structural reference substrate for the segmentation and fusion branches.

[0109] Furthermore, in one embodiment, the frequency mapping result (i.e., the frequency domain characteristics of the fusion task) Frequency domain features of segmentation tasks ) with residual coefficient (Based on empirical settings, frequency domain information is used as an auxiliary increment) Injected into task features (i.e., the spatial domain features of the fused task). Spatial domain features of segmentation tasks ), to obtain the task characteristics of the segmentation task. and the task characteristics of fusion tasks : , .

[0110] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In addition to the steps described above, this method may further include steps S71 to S73: Step S71: Determine the fusion loss function value based on the final fused image, the visible light image, and the infrared image.

[0111] In this embodiment, the fusion loss function value can be determined based on the final fused image, visible light image, and infrared image to constrain the modal dimension (visual fidelity). In an optional example, the fusion loss function value... : ;in, To preserve the strength fidelity, adaptive weights are used to balance the strength constraints of the two modes. It is obtained based on the brightness map corresponding to the final fused image, the visible light brightness component corresponding to the visible light image, and the infrared image; For edge preservation terms, the optimal edge structure between the two modes is preserved using a maximum gradient reference strategy. It is obtained based on the brightness map corresponding to the final fused image, the visible light brightness component corresponding to the visible light image, and the gradient corresponding to the infrared image; To ensure high-frequency detail fidelity, Laplacian high-frequency residuals are used to compensate for the insufficient texture coverage of the first two items. It is obtained based on the brightness map corresponding to the final fused image, the visible light brightness component corresponding to the visible light image, and the infrared image; and... .

[0112] Step S72: Determine the segmentation loss function value based on the final segmented image and the semantically labeled image.

[0113] In this embodiment, the semantically labeled image (e.g., a pixel-by-pixel semantically labeled image) is the semantic segmentation label corresponding to the visible light image and infrared image of the sample scene. Based on the final segmented image and the semantically labeled image, a segmentation loss function value can be determined to constrain the task dimension (semantic discrimination). Constrain the semantic segmentation accuracy of the main path.

[0114] Step S73: Update the model parameters based at least on the fusion loss function value and the segmentation loss function value to obtain a multimodal image collaborative fusion model.

[0115] In this embodiment, the total loss function value can be obtained based at least on the fusion loss function value and the segmentation loss function value. Then, based on the total loss function value, the model parameters of the unified vision and semantic learning infrared and visible light image collaborative fusion model are updated until the total loss function value converges. The model parameters are then fixed to obtain the trained unified vision and semantic learning infrared and visible light image collaborative fusion model, which serves as a multimodal image collaborative fusion model. The input of this multimodal image collaborative fusion model is a visible light image and an infrared image of a certain scene, and the output is the final segmented image and the final fused image of that scene.

[0116] The multimodal image collaborative fusion model in this embodiment can be used for image fusion and semantic segmentation in multiple fields, such as: Domain 1, Autonomous Driving Perception: Pedestrian and vehicle detection and scene understanding in extreme environments such as nighttime, rain, and smoke; Domain 2, Smart City Surveillance and Nighttime Target Detection: Semantic Analysis and Security Target Recognition in Nighttime Scenes Using Multimodal Cameras; Area 3, UAV Remote Sensing: Land Feature Classification and Target Detection in Infrared and Visible Light Remote Sensing Images; Area 4, Medical Image-Assisted Diagnosis: This can be extended to multimodal medical image fusion scenarios (such as CT-MRI joint analysis).

[0117] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, in addition to the steps described above, steps S81 to S82 may also be included, and step S73 may specifically include step S83: Step S81: Input the final fused image and the infrared image into the dual-stream encoder to obtain the reconstructed segmented image output by the segmentation decoder.

[0118] In this embodiment, after obtaining the final fused image of the sample scene, the final fused image and the infrared image of the sample scene are input into the dual-stream encoder of the unified vision and semantic learning infrared-visible light image collaborative fusion model. After processing by the unified vision and semantic learning infrared-visible light image collaborative fusion model, the reconstructed segmented image output by the segmentation decoder in the unified vision and semantic learning infrared-visible light image collaborative fusion model is obtained, thereby constraining the semantic usability of the fused output (fused image). The model parameters of the unified vision and semantic learning infrared-visible light image collaborative fusion model are frozen during the processing of the final fused image and the infrared image. That is, for the same sample scene, the model parameters used by the unified vision and semantic learning infrared-visible light image collaborative fusion model when processing the visible light image and the infrared image are the same as those used by the unified vision and semantic learning infrared-visible light image collaborative fusion model when processing the final fused image and the infrared image; that is, the parameters are shared. In other words, the reconstructed segmented image is the segmented image obtained by replacing the visible light image with the final fused image as input and then re-inferring through the parameter-shared verification path.

[0119] Step S82: Based on the reconstructed segmented image and the semantically labeled image, obtain the semantic consistency loss function value.

[0120] In this embodiment, a semantic consistency loss function value can be obtained based on the reconstructed segmented image and the semantically labeled image, which can then be used to constrain the hierarchical dimension (generative-cognitive consistency). The method for determining this semantic consistency loss function value is the same as the method for determining the segmentation loss function value; both focus on ambiguous boundaries and regions with difficult-to-classify categories. This semantic consistency loss function value... This constraint on the semantic consistency of the fused image in the isomorphic verification path is an indirect constraint on the semantic availability of the fused output.

[0121] Step S83: Update the model parameters based on the fusion loss function value, the segmentation loss function value, and the semantic consistency loss function value to obtain a multimodal image collaborative fusion model.

[0122] In this embodiment, a total loss function value can be obtained based on the fusion loss function value, the segmentation loss function value, and the semantic consistency loss function value. Then, based on this total loss function value, the model parameters of the unified vision and semantic learning infrared-visible image collaborative fusion model are updated until the total loss function value converges. The model parameters are then fixed, resulting in a trained unified vision and semantic learning infrared-visible image collaborative fusion model, which can be used as a multimodal image collaborative fusion model. It should be noted that the semantic consistency loss path in this embodiment is only enabled during the training phase and removed during model inference, without increasing deployment costs or testing overhead.

[0123] In this embodiment, an isomorphic semantic verification path is proposed. By feeding the final fused image back into the shared module chain to obtain auxiliary segmentation supervision, it provides indirect constraints on the semantic availability of the fused output, and there is no additional computational overhead during inference.

[0124] In an optional example, the total loss function value : ;in, The weight is fixed at 1. and It can adaptively adjust during model training.

[0125] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a multimodal image collaborative fusion method that unifies visual and semantic learning. In this method, in addition to the steps described above, steps S91 to S94 may also be included, and step S83 may specifically include step S95: Step S91: Based on the reconstructed segmented images and semantically labeled images from training round 1 to training round t, determine the cumulative mIoU of the segmentation task in training round t.

[0126] The model training process in this embodiment includes multiple training rounds. Each training round corresponds to the output of a reconstructed segmentation image. In the t-th training round (i.e., the t-th training round), the cumulative mIoU (mean intersection-union ratio) of the segmentation task in the t-th training round can be determined based on the reconstructed segmentation images and semantically labeled images corresponding to the 1st to the t-th training rounds, where t is an integer greater than 1.

[0127] In a specific example, the mIoU of the segmentation task in the first training round can be determined based on the reconstructed segmentation image and semantically labeled image corresponding to the first training round, and this mIoU of the segmentation task in the first training round can be used as the cumulative mIoU of the segmentation task in the first training round. Based on the reconstructed segmentation image and semantically labeled image corresponding to the second training round, the mIoU of the segmentation task in the second training round can be determined, and the cumulative mIoU of the segmentation task in the second training round can be determined based on the mIoU of the segmentation task in the first training round and the mIoU of the segmentation task in the second training round. Based on the reconstructed segmentation image and semantically labeled image corresponding to the third training round, the mIoU of the segmentation task in the third training round can be determined, and the cumulative mIoU of the segmentation task in the third training round can be determined based on the mIoU of the segmentation task in the first training round, the mIoU of the segmentation task in the second training round, and the mIoU of the segmentation task in the third training round. This process continues until the cumulative mIoU of the segmentation task in the t-th training round is obtained.

[0128] Step S92: Determine the adjustment base value for training round t based on the cumulative mIoU of the segmentation task in training round t.

[0129] In this embodiment, to avoid training timing conflicts in joint optimization (i.e., introducing semantic preservation constraints too early will interfere with the establishment of segmentation capabilities), this embodiment uses the cumulative mIoU of the training set as the maturity signal, and determines the adjustment base value for the t-th training round based on the cumulative mIoU of the segmentation task in the t-th training round.

[0130] In an optional example, the cumulative mIoU of the segmentation task in training epoch t is used as the maturity signal, and the adjustment base value in training epoch t is obtained by inverse Sigmoid mapping. : .

[0131] Step S93: Based on the segmentation loss function values ​​of training round t and training round (t-1), and the semantic consistency loss function values ​​of training round t and training round (t-1), determine the loss change rate of training round t, and based on the loss change rate of training round t, determine the convergence factor of training round t.

[0132] In this embodiment, a convergence factor is introduced for smoothing correction to suppress weight oscillations caused by loss fluctuations. Specifically, the loss change rate in training epoch t can be determined based at least on the segmentation loss function values ​​in training epoch t and t-1, as well as the semantic consistency loss function values ​​in training epoch t and t-1. Then, the convergence factor in training epoch t is determined based on the loss change rate in training epoch t.

[0133] In an optional specific example, the rate of change of loss r in training epoch t and the convergence factor in training epoch t can be obtained by the following formula. : ; in, , These are the segmentation loss function values ​​for training round t and training round (t-1), respectively. , Let be the semantic consistency loss function values ​​for training round t and training round (t-1), respectively. This is the numerical stability constant, with a value of 1 × 10⁻⁶. -6 This is used to prevent division by zero errors in the calculation of the rate of change of loss.

[0134] ;in, Appropriately reduce This leaves room for optimization while preserving semantic constraints. When losses fluctuate, the director's task coefficient should be appropriately increased.

[0135] Step S94: Based on the adjustment baseline value of training round t, determine the semantic consistency coefficient of training round t, and based on the adjustment baseline value and the convergence factor of training round t, determine the segmentation coefficient of training round t.

[0136] In this embodiment, after obtaining the convergence factor and the adjustment baseline value, the semantic consistency coefficient of training round t can be determined based on the adjustment baseline value of training round t; and the segmentation coefficient of training round t can be determined based on the adjustment baseline value and the convergence factor of training round t. Furthermore, the segmentation coefficient of training round t is greater than the semantic consistency coefficient of training round t. In this embodiment, the segmentation coefficient of training round t... The semantic consistency coefficient of training round t The model is adaptively adjusted based on the accumulated mIoU during training.

[0137] In an optional example, the split coefficient for the t-th training epoch can be obtained by the following formula. The semantic consistency coefficient of training round t : ; in, Ensure that segmentation remains dominant within the semantic supervision items. To prevent the semantic preservation loss from acquiring weights comparable to the segmentation main loss, the above constraints maintain a stable optimization hierarchy: the fusion and reconstruction weights are fixed, segmentation supervision always dominates the semantic optimization term, and semantic preservation constraints are gradually introduced as training maturity progresses.

[0138] In this embodiment, a maturity-driven adaptive multi-task loss adjustment mechanism is used, with the training set mIoU as the maturity signal. Combined with a convergence factor smoothing mechanism, the training timing conflict between segmentation capability establishment and semantic consistency constraint introduction in joint optimization is resolved, and priority conflicts between tasks are avoided. This enables simultaneous improvement in fusion quality and semantic performance, and achieves stable multi-task collaborative convergence.

[0139] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the framework of an infrared and visible light image collaborative fusion model that unifies visual and semantic learning, according to an embodiment of the present invention. Figure 2 In the upper left section, the implementation box illustrates the processing flow of the infrared and visible light image collaborative fusion model that unifies visual and semantic learning, fusion of visible light images... With infrared images As input, via the cross-modal deep interaction module (i.e. Figure 4 Constructing shared semantic representations in CDIM Then, the task condition structure bias layer (including Figure 4 HIA and FSE in the frequency-aware task router (i.e. Figure 4 In the FRT (Functional Radio Frequency Interaction) module, the task decoupling is jointly implemented in the spatial and frequency domains, and finally, through the spatially aware semantic interaction module (i.e., ... Figure 4 SASI (Search Engine Integration) is used to establish cross-layer semantic backflow and jointly output segmented images. With fused images Where C represents the number of semantic categories. In this embodiment, a dual-domain collaborative decoupling mechanism of spatial domain and frequency domain is adopted, and frequency domain analysis is reshaped into the core dimension of cross-task feature resource allocation for the first time: the task condition structure bias (HIA / FSE) in the spatial domain provides the location prior of where differentiation occurs, and the perceptual routing (FRT) in the frequency domain provides the content prior of which frequency components to differentiate. The two constitute a complementary decoupling dimension, fundamentally alleviating cross-task frequency resource competition.

[0140] The infrared and visible light image collaborative fusion model of unified vision and semantic learning in this embodiment starts from three complementary dimensions: modal interaction, task decoupling, and hierarchical semantic supervision, and solves three types of problems: cross-modal heterogeneous aggregation, cross-task representation coupling, and generative-cognitive semantic discontinuity.

[0141] in, Figure 4The gray box on the upper right represents the isomorphic semantic verification path (SIV), which is enabled only during training and removed during inference. Also, Figure 4 The lower half, with its three dashed boxes from left to right, illustrates the architecture of the Hierarchical Interaction Attention Module (HIA), the Fusion-Specific Enhancement Module (FSE), and the Frequency-Aware Task Router (FRT), respectively. Their specific internal processing flows can be found in the aforementioned corresponding embodiments. Furthermore, it should be noted that... Figure 4 In HIA, α and β are scale-dependent learnable scalars, respectively. Figure 4 In FSE, α and β are respectively from the aforementioned embodiments. and .

[0142] In conjunction with any of the above embodiments, experimental details of the present invention are shown in the following embodiments.

[0143] Training dataset. This embodiment uses the MSRS dataset [×] for training and validation. The MSRS dataset contains 1083 pairs of registered infrared and visible light image pairs, covering various complex road scenes during both day and night, with an image resolution of [missing information]. The data is accompanied by pixel-by-pixel semantic segmentation annotations (9 classes in total). During the training phase, the input images are uniformly cropped to... And apply random horizontal flipping and random rotation as data augmentation strategies.

[0144] Test Datasets. This embodiment evaluates the model performance on the test sets of the following three publicly available benchmark datasets. The MSRS test set contains 361 pairs of images with semantic segmentation annotations for joint evaluation of fusion quality and semantic segmentation. The M³FD dataset [×] contains various complex scenes such as strong light, low light, and smoke occlusion, and is used to evaluate the model's generalization ability in cross-scene fusion quality. The FMB dataset [×] contains 280 pairs of test images and provides fine-grained segmentation annotations covering 14 semantic classes, used to verify the generalization ability of this method in semantic understanding of complex scenes.

[0145] Training configuration. The infrared-visible image collaborative fusion model with unified visual and semantic learning proposed in this embodiment is trained end-to-end using the SGD optimizer, with an initial learning rate set to... The momentum coefficient is 0.9, and the weight decay is... The learning rate decreases using a polynomial decay strategy with a power of 0.9, and an exponential warmup is used for the first 1000 steps of training. The batch size is set to 10, and the training lasts for 1000 epochs. The hyperparameters of each module are summarized below: Encoder feature channel dimension ; Scale-learnable coefficients of HIA in §3.3 FSE's fixed scaling factor , ; FRT frequency band division intervals: Upper Realm , interval , Upper Realm Soft mask steepness parameters Residual injection weights ; Fusion loss weights , , OHEM-CE loss threshold Minimum number of pixels in the hardest example , This refers to the batch size.

[0146] Evaluation metrics. Regarding fusion quality, six objective metrics are used for quantitative evaluation: standard deviation (SD), mutual information (MI), peak signal-to-noise ratio (PSNR), visual information fidelity (VIFF), and fusion quality (…). The performance of semantic segmentation was evaluated using the mean intersection-union ratio (mIoU), with ViT-Adapter [×] serving as the segmentation backbone. Object detection performance was evaluated using mAP@0.5 and AP@[0.5 :0.95], with YOLOv7 [×] serving as the detection backbone. All comparison methods generated fused images using their respective publicly available pre-trained weights and were evaluated under a unified task backbone network and training configuration to ensure fairness and reproducibility.

[0147] In one embodiment, the proposed comparison method is as follows: a comprehensive comparison is made between the infrared-visible light image collaborative fusion model that unifies visual and semantic learning and the following nine representative state-of-the-art methods, covering two categories: pure visual fusion methods and task-oriented joint modeling methods: SDCFusion [×], DCEvo [×], MRFS [×], PSFusion [×], SAGE [×], SHIP [×], SegMiF [×], SuperFusion [×], and TarDAL [×]. Among them, SDCFusion and DCEvo employ cross-modal feature learning strategies to improve fusion representation capabilities; MRFS enhances structural information extraction through multi-receptive field aggregation; PSFusion and SAGE introduce semantic priors to impose high-level constraints on the fusion process; SHIP models the synergistic and complementary relationship between bimodalities in spatial and channel dimensions through high-order cross-modal interaction; SegMiF adopts a segmentation-driven fusion strategy; and SuperFusion and TarDAL respectively advance task-oriented fusion modeling from the perspectives of unsupervised generation and target-aware adversarial approaches. The methods described above cover the core representatives of the current mainstream paradigm, providing a systematic reference for comprehensive evaluation. All comparative experiments used the authors' publicly available pre-trained weights or retrained according to the original settings to ensure the fairness and reproducibility of the evaluation.

[0148] Specifically, regarding the main results of fusion quality: In order to determine whether the unified vision and semantic learning infrared visible light image collaborative fusion model (VSFuse) is superior to the current state-of-the-art methods in terms of fusion quality, this embodiment conducts a comprehensive fusion quality evaluation of VSFuse on three datasets: MSRS, FMB, and M³FD, covering a dual-track analysis of qualitative visualization and six quantitative indicators.

[0149] Among them, for qualitative comparisons, Figure 5 , Figure 6 and Figure 7 The diagrams show qualitative fusion comparisons of VSFuse and state-of-the-art infrared-visible image fusion methods on the MSRS dataset, the FMB dataset, and the M3FD dataset.

[0150] For MSRS, in MSRS nighttime street scenes, the red-framed area is used to examine the preservation of thermal radiation saliency of pedestrian targets at medium to long distances, and the green-framed area is used to compare the recovery effect of local vehicle edges and dark structures. For example... Figure 5As shown in the magnified area, TarDAL and SAGE exhibit weak thermal target response in the red-boxed pedestrian area, with a decrease in contrast between the human outline and the low-light background; SuperFusion's overall image is dark, with insufficient clarity of pedestrian and vehicle edges; MRFS and SegMiF show overly smooth textures in the green-boxed vehicle area, resulting in some loss of dark structural details; PSFusion exhibits localized brightness unevenness in the green-boxed area, affecting edge continuity. In contrast, VSFuse more completely preserves the pedestrian thermal radiation outline and target contrast in the red-boxed area, and recovers vehicle edge structure and dark layering well in the green-boxed area, without exhibiting the obvious thermal target attenuation or structural blurring seen in the aforementioned methods. This observation, along with the results of VSFuse on the M³FD dataset MI and... The quantitative results are consistent with the CDIM cross-modal information complementarity mechanism and the design goals of FATER frequency-aware routing.

[0151] For Free-Moving Bridge (FMB), in the FMB nighttime intersection scene, the red-framed area reflects the contour separation capabilities of various methods for dense pedestrian targets, while the green-framed area is used to examine the preservation effect of local vehicle textures and boundaries under strong light interference. For example... Figure 6 As shown in the magnified area, PSFusion and SuperFusion exhibit localized overbrightness and unclear target boundaries in the multi-pedestrian area (red box); SAGE and TarDAL show low overall contrast for thermal targets in the same area, resulting in decreased individual contour recognition; SDCFusion and MRFS show both blurred edges and localized overbrightness in the vehicle area (green box), with some loss of detail. In contrast, VSFuse more clearly presents the individual contours of multiple pedestrians in the red box area and better preserves the local edges and brightness transitions of vehicles in the green box area. This aligns with SASI's design goal of guiding the fusion path through semantic boundary priors and enhancing multi-target boundary perception.

[0152] For M³FD, in a densely trafficked daytime rainy scene, the red-framed area is used to compare the structural integrity of key vehicle targets, and the green-framed area is used to observe the naturalness of the wet road surface reflection texture and the edges of adjacent vehicles. For example... Figure 7As shown in the magnified area, MRFS and SegMiF introduce more infrared information in the red-boxed vehicle area, resulting in a decrease in the naturalness of the vehicle body color. Conversely, SAGE and TarDAL show blurred edges on the vehicle body, leading to insufficient target saliency. DCEvo and SuperFusion exhibit smoother reflective textures in the green-boxed wet road surface area, but with compromised local edge continuity. In contrast, VSFuse more clearly preserves the key structures and boundaries of the target vehicle in the red-boxed area and maintains better wet road surface reflective textures and local edge continuity in the green-boxed area. This observation is consistent with the quantitative results of VSFuse on the MI and Qabf metrics of the M³FD dataset in Table 1.

[0153] Based on qualitative observations from the three datasets, VSFuse demonstrates stable performance across scenes in terms of target saliency preservation, local texture restoration, and visual naturalness. These trends will be systematically verified in the quantitative analysis in the next section.

[0154] For quantitative comparison, as shown in Table 1, Table 1 is a quantitative comparison table of different fusion methods for MSRS, FMB and M³FD datasets shown in an embodiment of the present invention.

[0155] Table 1. Quantitative comparison of different fusion methods for MSRS, FMB, and M³FD datasets.

[0156] For the MSRS dataset, VSFuse achieved best results in 4 out of 6 metrics: SD (42.413), MI (3.464), VIFF (1.063), and... VSFuse (0.724) ranks second in PSNR (16.384) and SSIM (1.396). In terms of horizontal comparison of various indicators, VSFuse's advantage in MI is particularly prominent, significantly outperforming the second-ranked DCEvo (3.054) and SuperFusion (2.918) with a score of 3.464, indicating that the fused image generated by VSFuse captures the most complementary information from the dual-source images. VIFF (1.063) is also significantly higher than the second-ranked PSFusion (1.050) and DCEvo (1.034), reflecting VSFuse's comprehensive quality advantage in visual perception. SegMiF achieves the best PSNR (17.024) due to its segmentation prior, while SuperFusion's generative adversarial training strategy results in the highest SSIM (1.432). However, both show significant gaps compared to VSFuse in information fidelity indicators such as MI and VIFF, indicating that local optimization of a single indicator cannot replace the improvement of multi-dimensional comprehensive performance. It is worth noting that while PSFusion performs exceptionally well in SD (46.279), its MI (2.043) and SSIM (1.332) are significantly lower, reflecting that higher contrast difference does not necessarily correspond to better semantic information preservation. Overall, VSFuse has the highest percentage of "best + second best" scores across the six metrics, particularly in Visual Perception Quality (VIFF), Information Integrity (MI), and Gradient Preservation (GRP). It ranks first in all three core dimensions reflecting task adaptability, which is highly consistent with the core design goal of this paper, "unifying visual and semantic learning".

[0157] For the FMB dataset, which covers 14 categories of fine-grained semantic scenes, VSFuse achieves a MI of 3.529 and... SuperFusion achieved the best performance on both metrics (0.716) and ranked second best on PSNR (14.424) and VIFF (0.886), demonstrating stable cross-dataset generalization ability in complex semantic environments. PSFusion had the highest SD (49.881), but its MI (2.031) and SSIM (1.368) were significantly lower, consistent with its performance on MSRS, indicating that the high contrast output of this type of method comes at the cost of information integrity. SegMiF achieved the highest PSNR (15.345) driven by semantic segmentation, and SuperFusion's SSIM (1.480) also performed best on FMB, but both lagged behind in MI and VIFF. In terms of overall information quality metrics, DCEvo is inferior to VSFuse. DCEvo achieves the best VIFF (0.892), slightly higher than VSFuse (0.886), with the difference within 0.006, which is within a reasonable range of magnitude competition. The results on the FMB dataset further demonstrate that VSFuse's multi-dimensional information integration capability has good generalization consistency in fine-grained semantic category scenarios.

[0158] For the M³FD dataset, VSFuse achieves a similar performance at MI(3.587) compared to... TarDAL maintains its best performance at SD (0.689) and ranks second in PSNR (14.625) and VIFF (0.968), demonstrating a stable leading position across datasets. PSFusion again tops the list in SD (50.951), but lags behind in information integrity. SHIP follows closely behind in MI (3.208), ranking second. SegMiF is the best in PSNR (16.072), while TarDAL has the highest SSIM (1.547), but TarDAL lags behind in SD (26.936), MI (2.349), and VIFF (0.968), indicating a consistent leading position across datasets. (0.269) is in the last interval, indicating the worst overall balance. MRFS performs well on SD (36.436), but its cross-dataset metric dispersion is large, resulting in insufficient generalization stability. VSFuse shows MI and The continuous optimality of CDIM and the continuous suboptimal or optimality of VIFF are consistent with the functional expectations of CDIM's bidirectional information exchange mechanism, FATER's frequency-aware routing design, and SASI's semantic boundary prior injection mechanism.

[0159] Based on the results from the three datasets, VSFuse achieved the highest "number of optimal terms" and the best overall ranking among all methods. In the MI dataset, which best reflects the quality of fused image information and semantic adaptability, and... In both metrics, VSFuse ranked first across all three datasets. This consistent performance across datasets supports the robustness and generalization advantage of the proposed collaborative fusion framework under multi-scene perception conditions, thus proving that the unified vision and semantic learning infrared visible light image collaborative fusion model (VSFuse) is superior to the current state-of-the-art methods in terms of fusion quality.

[0160] To address the semantic segmentation performance issue, and to determine whether VSFuse's fusion results can effectively improve the performance of downstream high-level vision tasks in the semantic understanding dimension, this embodiment evaluates the contribution of each method's fused images to the high-level semantic perception task. Qualitative visualization and quantitative comparative analysis are performed on the MSRS and FMB datasets, respectively. ViT-Adapter is used as the unified segmentation backbone, and evaluation is conducted on all 361 pairs of fused images in the MSRS test set, covering nine semantic categories: Unlabeled, Car, Person, Bike, Curve, Car Stop, Guardrail, Color Cone, and Bump.

[0161] Qualitative comparison: Figure 8 This is a visual comparison chart illustrating the output of VSFuse and nine of the most advanced or representative fusion algorithms, as shown in an embodiment of the present invention. Figure 8 As shown, after segmentation using the Vit-adapter, for an image (#00004N) randomly selected from the MSRS test set, the visual comparison between VSFuse's output and nine state-of-the-art or representative fusion algorithms is presented. Figure 8 As shown, this embodiment selects a representative nighttime scene containing small foreground instances at mid-to-long distances from the MSRS test set for semantic segmentation visualization comparison. The red box and the magnified area in the lower left corner are used to observe the segmentation integrity of adjacent small targets: it can be seen from the magnified area that TarDAL and MRFS have obvious small target omissions in this area, often only retaining one of two adjacent small instances; SuperFusion's semantic characterization of both small targets is seriously insufficient, and its overall small target retention ability is the weakest. In contrast, VSFuse can simultaneously retain the complete semantic structure of two adjacent small foreground instances, and its segmentation result is closest to GT. This result is consistent with the design goal of SASI to enhance the semantic discrimination of small target boundaries.

[0162] Table 2 compares the semantic segmentation performance of the comparative experiments on the MSRS dataset.

[0163] Quantitative Comparison: The quantitative results are shown in Table 2. VSFuse achieved the highest mIoU (0.7931) among all the fusion methods compared, surpassing SHIP (0.7910) by an absolute advantage of 0.0021, and maintaining the best overall performance in close competition with PSFusion (0.7898) and DCEvo (0.7896). The single-modal baseline (visible light: 0.7472, infrared: 0.6742) was significantly lower than that of each fusion method, demonstrating that image fusion has a significant positive contribution to semantic segmentation tasks. In category-level analysis, VSFuse achieved 5 first-place rankings (Unlabeled, Bike, Car Stop, Guardrail, Color Cone) and 4 second-place rankings (with Bump tied for second place) across 9 semantic categories, maintaining a more stable leading performance across all categories rather than relying on extreme advantages in individual categories. Among them, the leading performance of Guardrail (0.8907) and CarStop (0.7927) in the two structure-sensitive categories is particularly outstanding. In contrast, SuperFusion lags significantly behind in mIoU (0.6907), with the Color Cone category having the lowest mIoU at only 0.4550, indicating that generative training strategies focused solely on visual quality optimization may not adequately address the semantic discriminability required for downstream segmentation tasks. This embodiment attributes the segmentation adaptation advantage of VSFuse to the "unified visual and semantic learning" framework proposed in this paper—through the bidirectional semantic knowledge interaction mechanism of SASI, the fusion network continuously receives segmentation semantic supervision during training, which helps improve the semantic discriminability of the fusion representation.

[0164] To further verify the semantic generalization ability of various methods in complex road scenes, this embodiment evaluates the support of fused images for downstream semantic segmentation tasks on the FMB dataset for semantic segmentation. Consistent with the previous work, a retrained ViT-Adapter is used as the unified segmentation backbone, and evaluation is performed on all 280 pairs of fused images in the FMB test set, covering 14 semantic categories: Unlabeled, Road, Sidewalk, Building, Lamp, Sign, Vegetation, Sky, Person, Car, Truck, Bus, Motorcycle, and Pole.

[0165] Qualitative comparison: Figure 9 This is a comparison chart of fused images, illustrating how VSFuse is compared to nine fusion algorithms. The comparison uses a retrained Vit-adapter to segment and visualize FMB test set images. (See the image below.) Figure 9As shown, this embodiment selects scene #00043 in the FMB test set for semantic segmentation visualization comparison. The red box and the magnified area in the lower left corner are mainly used to observe the semantic preservation of the light signs, hanging signs, and their thin support structures. From the magnified area, it can be observed that SDCFusion, MRFS, SegMiF, and TarDAL have weak responses to small structures in this area, with varying degrees of omissions or adhesion to the background; PSFusion and SAGE, although preserving some main areas, still have relatively coarse local structures, showing a certain gap compared to GT. In contrast, VSFuse can more completely preserve the main body of the light sign and related support structures in the red box area, and the magnified results are closer to GT; at the same time, it also maintains good continuity in the segmentation boundaries in the areas of people and the intersection of roads and buildings. This result is consistent with the design goal of SASI to strengthen semantic boundary priors and improve the local discriminativeness of fusion representations in complex scenes.

[0166] Table 3 compares the semantic segmentation performance in comparative experiments on the FMB dataset.

[0167] Quantitative Comparison: The quantitative results are shown in Table 3. VSFuse achieved the highest mIoU (0.7645) among all the fusion methods compared, an improvement of 0.0111 over the second-best method, SDCFusion (0.7534). This lead is significantly greater than in the MSRS experiment, indicating that VSFuse has a more prominent generalization advantage in complex scenes covering 14 categories of fine semantics. The single-modal baseline (visible light: 0.6947, infrared: 0.6760) is significantly lower than that of each fusion method, further confirming the positive contribution of image fusion to the segmentation task of complex semantic scenes. In category-level analysis, VSFuse achieved the highest values ​​in all 14 semantic categories among all the fusion methods compared, demonstrating a stable leading position across categories. In the scene structure categories such as Road (0.9185), Building (0.8814), Vegetation (0.9108), and Sky (0.9472), VSFuse ranked first, indicating its good ability to preserve the global scene semantic layout. In the fine-grained object and slender structure categories, Person (0.7695), Motorcycle (0.7104), and Pole (0.5882) improved by 0.0167, 0.0129, and 0.0189 respectively compared to the second-best method, indicating that the method has stronger competitiveness in local semantic recognition ability. It should be noted that the best overall performance in the Unlabeled, Lamp, and Sign categories comes from the single-modal baseline, indicating that a few strong modal bias categories can still directly benefit from the significant cues in the original source image. From the internal comparison of fusion methods, VSFuse still maintains its lead in the above three categories and ultimately shows a more balanced overall segmentation performance with the highest mIoU. Based on the combined results from the MSRS and FMB datasets, VSFuse maintains optimal segmentation adaptability across different semantic systems and scene complexities, thus providing a positive answer to the sub-problems of RQ2 in the semantic understanding dimension.

[0168] To verify the effectiveness of the proposed method for object detection on the MSRS dataset, this embodiment conducts a comparative experiment on the MSRS dataset. YOLOv7 [×] is used as the unified detection backbone. The AP (Achievements Per Target) of pedestrians (P) and vehicles (C) at different IoU thresholds are reported within the range of 0.5–0.9. Furthermore, mAP@[0.5:0.95] is used to evaluate the overall detection performance.

[0169] Qualitative comparison: Figure 10This is a schematic diagram illustrating an embodiment of the present invention, showing the comparison of target detection performance of VSFuse with nine of the most representative fusion methods using a YOLOv7 network in MSRS selection scenario #855. Figure 10 As shown, this embodiment selects a representative nighttime pedestrian scene from the MSRS test set for a visual comparison of target detection. The scene includes a main pedestrian target and a small pedestrian target in the distance on the left. Visible produced almost no effective detection output in this low-light scene, indicating that a single visible light mode is difficult to provide stable target cues for the detector under low-light conditions. Among the contrast fusion methods, MRFS has insufficient response to the small pedestrian target on the left; TarDAL has some response to the main target, but its overall detection stability is still limited; SHIP, although it stably detects the main target, has a loose bounding box in this scene, with some background redundancy response. In contrast, VSFuse has more stable localization of the main target and also has an effective response to the small pedestrian target on the left, with more complete overall detection cues.

[0170] Table 4 compares the object detection performance in comparative experiments on the MSRS dataset.

[0171]

[0172] Quantitative Comparison: The quantitative results are shown in Table 4, where P represents "Person" and C represents "Vehicle". At mAP@0.5, VSFuse achieved the best score among all compared methods with 95.700, surpassing SHIP (95.570), DCEvo (95.370), and PSFusion (95.250). At the category level, VSFuse achieved the best score in pedestrian detection (P@0.5=98.360); and ranked fourth in vehicle detection (C@0.5=93.040), behind PSFusion (94.290), SDCFusion (94.170), and SHIP (94.020). At the more stringent comprehensive index mAP@[0.5:0.95], SHIP (73.640) achieved the best score, followed by VSFuse (72.630), and then PSFusion (72.600). Further observation of the threshold gradients reveals that VSFuse outperforms SHIP at a relatively loose IoU threshold (0.5), but still lags behind at a high IoU threshold (0.9), indicating that VSFuse has stronger target recall capability under loose matching thresholds, while still having room for improvement in high-precision bounding box regression. MRFS (66.700) and TarDAL (59.980) are both at the bottom of the range at all thresholds, consistent with their lagging performance in MI and Qabf in the aforementioned fusion quality assessment. This result is consistent with CDIM's design for transferring infrared thermal saliency information, which helps enhance the perception of pedestrian targets in low-light nighttime scenarios.

[0173] Based on the combined results of semantic segmentation and object detection on the MSRS dataset, VSFuse achieves leading performance in both high-level vision tasks, namely semantic segmentation (mIoU=0.7931) and object detection (mAP@0.5=95.700). This demonstrates that the fusion results generated by VSFuse can simultaneously support both high-level vision tasks, namely semantic segmentation and object detection, and maintain stable performance at the downstream perception level of multiple tasks.

[0174] In one embodiment, ablation experiments were conducted. This embodiment uses systematic ablation experiments to verify the effectiveness of each core design of VSFuse. All variants were trained independently under the same configuration, and fusion quality (SD, MI, PSNR, VIFF) was reported simultaneously on the MSRS test set (361 pairs). The results of the semantic segmentation performance (mIoU) and SSIM are summarized in Table 5.

[0175] Table 5. Schematic diagram of ablation study results based on the MSRS dataset (361 test pairs)

[0176] For progressive module stacking ablation (Group A): 1) Progressive contributions of modules (A1–A7). Group A started with a two-stream baseline (A1) without any modules in this paper and progressively introduced each component to verify the cumulative contributions.

[0177] The effectiveness of CDIM is demonstrated in Table 5. Introducing CDIM (A2) significantly improved MI from 2.673 to 3.108 (+0.435), PSNR from 15.961 to 16.148, and SSIM from 1.360 to 1.372, resulting in comprehensive improvements across all six fusion metrics. mIoU also increased from 0.702 to 0.758 (+0.056). These comprehensive improvements confirm that the capture of cross-modal complementary information by four-way directed attention is the foundation for the synergistic improvement in fusion quality and semantic discriminativeness.

[0178] The functional specificity of HIA and FSE. Independent evaluations of A3 and A4 show that HIA in (+0.003) and SSIM (+0.004) show greater gains, while FSE is more prominent in SD and VIFF, exhibiting clear metric specificity. The A5 combined configuration shows gains in SD (42.385), Both (0.719) and SSIM (1.384) surpass their respective individual best values, and mIoU is further improved to 0.774, indicating that the two paths have a positive synergistic effect.

[0179] The effectiveness of FATER (Frequency Aware Task Router). The introduction of FATER (A6) further improved MI by 0.117, VIFF to 1.058, and mIoU from 0.774 to 0.785. The synchronous improvement in segmentation performance indicates that explicit frequency band routing also makes a positive contribution to the quality of semantic representation.

[0180] The comprehensive advantages of the complete model (A7). The complete model A7 achieved the best performance across the entire group in all fusion quality and segmentation metrics (MI=3.464, mIoU=0.793, MI improved by 0.106 from 3.358, mIoU improved by 0.008 from 0.785), and the monotonically increasing trend at each stage confirms the cumulative and synergistic contribution of each module.

[0181] For loss function ablation (Group B): 2) Loss components and modulation strategies (A8–A12). Group B, while maintaining the complete module structure, systematically evaluates the training contribution of each loss component and the adaptive modulation strategy.

[0182] The effectiveness of SIV (A8). As shown in Table 5, after removing SIV, MI decreased from 3.464 to 3.378, SSIM decreased to 1.391, and mIoU decreased to 0.790. The impact was moderate, consistent with the fact that SIV is only activated during training and path localization is mainly based on indirect semantic consistency constraints.

[0183] Hierarchical contributions to semantic supervision (A9 vs A10). It should be noted that A9, after removing... Under these conditions, the segmentation decoder retains its complete structure and participates in forward computation, with SASI using its output as a read-only prior (stop-gradient); due to the lack of segmentation supervision, This configuration degenerates into weak semantic output, and its performance degradation thus encompasses the combined effects of missing segmentation supervision and decreased prior quality. (A9 (SSIM=1.379, MI=3.168, mIoU=0.768)) A10 (SSIM=1.387, MI=3.285, mIoU=0.781) The monotonic sorting of A7 indicates that It has an independent positive effect on the fusion quality: when segmentation supervision is lacking, the ability of fusion features to constrain the structure of semantically significant regions tends to weaken, which in turn affects the information integrity and structural consistency of the fused image.

[0184] Effectiveness of MDLM adaptive modulation (A11 vs A12). Removing the full MDLM (A12) causes SSIM to drop to 1.383 and mIoU to drop to 0.770, lower than the baseline value for removing maturity modulation alone. The SSIM (1.388) and mIoU (0.777) of (A11) indicate that course-based weighted scheduling is necessary for multi-task collaborative convergence.

[0185] For structural mechanism ablation (Group C): 3) Key structural design selection (A13–A16). Group C conducted detailed isolation verification of the specific structural design of each module.

[0186] The necessity of bidirectional interaction in CDIM (A13). After CDIM degenerates into unidirectional attention (vi→ir only), MI decreases from 3.464 to 3.296 ( The decrease was 0.168, representing the largest reduction in MI across all structural ablation operations in the group, while SSIM decreased from 1.396 to 1.390. This decrease points to the dominant role of bidirectional interaction in capturing cross-modal information integrity.

[0187] FATER's dedicated contribution to frequency routing (A14). After removing FATER, The price dropped from 0.724 to 0.717 (exclusive). 0.007), representing the most specific response among the changes in each individual indicator, indicates the precise contribution of frequency routing to the quality of the fusion edge, suggesting that this contribution has an indicator-specific rather than a generalized gain.

[0188] SASI bidirectional path asymmetric functionality (A15 vs A16). A15 (disabling Seg→Fus) caused the largest MI decrease across the entire group (3.464 → 3.143). 0.321), indicating that this path is the main direct channel for semantic boundary prior injection fusion representation; A16 (disabling Fus→Seg) causes a larger VIFF reduction ( 0.013 vs A15 The score of 0.008 indicates that this path improves semantic prior availability and visual fidelity of the fused image through reverse regularization during training. Regarding mIoU, both A15 (0.788) and A16 (0.789) are lower than the complete model A7 (0.793), and A15's segmentation performance is weaker than A16, consistent with the functional positioning of the Seg→Fus path, which plays a dominant role in semantic prior injection. In summary, Seg→Fus dominates information integrity (MI), while Fus→Seg dominates visual fidelity (VIFF), and their functional roles are distinct and complementary.

[0189] The three sets of ablation results systematically demonstrate that each core module makes an independent and cumulative contribution to performance improvement; the HIA and FSE dual paths possess clear metric specificity; and the SASI bidirectional path assumes differentiated but complementary functional roles. This chain of evidence systematically answers the three types of verification questions raised in this section from three levels: module necessity, training strategy rationality, and structural design effectiveness. It also provides affirmative answers regarding whether each core module of VSFuse has an independent and cumulative performance contribution, and whether the design mechanism of each module of VSFuse can be verified through observable quantitative evidence.

[0190] In summary, this invention proposes a multimodal image collaborative fusion method that unifies visual and semantic learning. Specifically, it proposes a visual and semantic collaborative learning framework for infrared and visible light image fusion, aiming to simultaneously improve the pixel-level reconstruction quality of the fused image and the semantic adaptability to downstream high-level visual tasks. Addressing the shortcomings of related methods in three core areas—cross-modal information interaction, dual-domain feature decoupling, and semantic knowledge feedback—this invention designs four complementary key modules: CDIM models bidirectional cross-modal complementary relationships within a unified representation space through a four-way directed attention mechanism, enhancing the collaborative capture capability of fused features for infrared thermal saliency and visible light structural texture; HIA and FSE decompose spatial domain task adaptation into two independent parameterized paths: boundary-sensitive structure shaping for segmentation and texture channel compensation for fusion; FATER performs spectral decomposition and task-preference-based reorganization of shared semantic representations through a frequency-aware task routing mechanism, alleviating cross-task frequency resource competition; and SASI constructs a bidirectional semantic knowledge interaction closed loop, continuously injecting segmentation boundary priors into the fusion path while improving the usability of semantic priors through reverse regularization of fused features. Extensive experiments on three public benchmark datasets, MSRS, FMB, and M³FD, demonstrate that VSFuse outperforms current mainstream methods in all three evaluation dimensions: fusion quality, semantic segmentation, and object detection. This verifies the robustness and generalization advantage of the proposed collaborative fusion framework under multi-scene perception conditions.

[0191] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0192] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0195] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A multimodal image collaborative fusion method that unifies visual and semantic learning, characterized in that, The method includes: Visible light and infrared images of the sample scene are input into a dual-stream encoder for multi-scale feature extraction to generate shared semantic representations between the fusion and segmentation tasks. The shared semantic representation and the visible light features of the visible light image are processed by the hierarchical interactive attention module to generate the spatial domain features of the segmentation task. By fusing specific enhancement modules to process the shared semantic representation and the visible light features of the visible light image, spatial domain features of the fusion task are generated; The shared semantic representation is spectrally decomposed by a frequency-aware task router to obtain low-frequency band, mid-frequency band and high-frequency band; based on the low-frequency band and mid-frequency band, the frequency domain features of the segmentation task are generated, and based on the mid-frequency band and high-frequency band, the frequency domain features of the fusion task are generated. Based on the spatial domain and frequency domain features of the segmentation task, the task features of the segmentation task are generated; and based on the spatial domain and frequency domain features of the fusion task, the task features of the fusion task are generated. A preliminary segmented image is obtained by decoding the task features of the segmentation task using a segmentation decoder, and a preliminary fused image is obtained by decoding the task features of the fusion task using a fusion decoder.

2. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 1, characterized in that, The method further includes: The semantic prior path in the spatially aware semantic interaction module is used to process the read-only prior corresponding to the preliminary segmentation image to obtain the semantic boundary; based on the task features of the segmentation task, the task features of the fusion task, and the semantic boundary, the fusion-segmentation injection strength and the segmentation-fusion injection strength are generated. The segmentation knowledge is obtained by processing the read-only prior corresponding to the preliminary segmented image through the segmentation branch in the spatially perceptual semantic interaction module. The task features of the fusion task are processed through the fusion branch in the spatially aware semantic interaction module to obtain fusion knowledge; The fusion knowledge is used to supplement the segmentation knowledge to obtain fusion-segmentation knowledge. Based on the fusion-segmentation knowledge, the fusion-segmentation injection strength, and the task characteristics of the segmentation task, the enhanced task characteristics of the segmentation task are obtained. The segmentation knowledge is used to supplement the fusion knowledge to obtain segmentation-fusion knowledge. Based on the segmentation-fusion knowledge, the segmentation-fusion injection strength, and the task characteristics of the fusion task, the enhanced task characteristics of the fusion task are obtained. The segmentation decoder decodes the enhanced task features of the segmentation task to obtain the final segmented image, and the fusion decoder decodes the enhanced task features of the fusion task to obtain the final fused image.

3. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 1, characterized in that, Visible light and infrared images of the sample scene are input into a dual-stream encoder for multi-scale feature extraction to generate shared semantic representations between the fusion and segmentation tasks, including: Based on the visible light characteristics of the visible light image, the visible light Q value, visible light K value, and visible light V value are obtained; based on the infrared characteristics of the infrared image, the infrared Q value, infrared K value, and infrared V value are obtained; based on the visible light V value and the infrared V value, a shared V value is determined. Based on the infrared intramodal enhancement branch, the infrared Q value, infrared K value, and shared V value are processed to obtain the infrared attention result; and based on the visible light intramodal enhancement branch, the visible light Q value, visible light K value, and shared V value are processed to obtain the visible light attention result. Based on the visible-infrared transmission branch, the infrared Q value, visible K value, and visible V value are processed to obtain the visible-infrared attention result; and based on the infrared-visible light transmission branch, the visible Q value, infrared K value, and infrared V value are processed to obtain the infrared-visible light attention result. Based on the infrared attention results, visible light attention results, visible light-infrared attention results, and infrared-visible light attention results, shared attention features are obtained. Based on the infrared features of the infrared image, infrared local features are obtained; and based on the visible light features of the visible light image, visible light local features are obtained. Based on infrared local features, visible light local features, and shared attention features, a shared semantic representation is obtained.

4. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 1, characterized in that, The shared semantic representation and the visible light features of the visible light image are processed by a hierarchical interactive attention module to generate the spatial domain features for the segmentation task, including: The shared semantic representation and the visible light features of the visible light image are processed by the cross-feature semantic mapping submodule to generate a structure-guided response; The shared semantic representation is sequentially subjected to channel attention and spatial attention through the channel-space joint attention submodule, thereby obtaining channel attention weights and spatial attention weights. The shared semantic representation is then processed based on the channel attention weights and spatial attention weights to obtain the channel-space joint attention result. Based on the channel-space joint attention results, structure-guided response, and the shared semantic representation, spatial domain features for the segmentation task are generated.

5. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 1, characterized in that, By fusing specific enhancement modules to process the shared semantic representation and the visible light features of the visible light image, spatial domain features for the fusion task are generated, including: The visible light features of the visible light image are extracted using the texture extraction submodule to obtain the texture extraction result; The shared semantic representation is filtered by the global average pooling channel gating submodule to obtain the channel filtering result; Based on the shared semantic representation, texture extraction results, and channel filtering results, the spatial domain features of the fusion task are obtained.

6. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 1, characterized in that, The shared semantic representation is spectrally decomposed using a frequency-aware task router to obtain low-frequency, mid-frequency, and high-frequency bands. Frequency domain features for the segmentation task are generated based on the low-frequency and mid-frequency bands, and frequency domain features for the fusion task are generated based on the mid-frequency and high-frequency bands, including: The shared semantic representation is transformed into the frequency domain through the frequency band decomposition submodule to obtain the spectrum, and the spectrum is decomposed using a mask to obtain the low frequency band, mid frequency band and high frequency band. The low-frequency band, mid-frequency band, and high-frequency band are transformed into the spatial domain to obtain low-frequency band features, mid-frequency band features, and high-frequency band features; based on the low-frequency band features, shared low-frequency features are obtained. Based on the low-frequency band features and mid-frequency band features, the intermediate frequency domain features of the segmentation task are obtained. Based on the shared low-frequency features and the intermediate frequency domain features of the segmentation task, the frequency domain features of the segmentation task are obtained. Based on the mid-frequency band features and high-frequency band features, the intermediate frequency domain features of the fusion task are obtained. Based on the shared low-frequency features and the intermediate frequency domain features of the fusion task, the frequency domain features of the fusion task are obtained.

7. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 2, characterized in that, The method further includes: Based on the final fused image, the visible light image, and the infrared image, determine the fusion loss function value; Based on the final segmented image and the semantically labeled image, determine the segmentation loss function value; Based at least on the fusion loss function value and the segmentation loss function value, the model parameters are updated to obtain a multimodal image collaborative fusion model.

8. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 7, characterized in that, The method further includes: The final fused image and the infrared image are input into the dual-stream encoder to obtain the reconstructed segmented image output by the segmentation decoder; Based on the reconstructed segmented image and the semantically labeled image, the semantic consistency loss function value is obtained; Based at least on the fusion loss function value and the segmentation loss function value, the model parameters are updated to obtain a multimodal image collaborative fusion model, including: The model parameters are updated based on the fusion loss function value, the segmentation loss function value, and the semantic consistency loss function value to obtain a multimodal image collaborative fusion model.

9. The multimodal image collaborative fusion method for unified visual and semantic learning according to claim 8, characterized in that, The method further includes: Based on the reconstructed segmented images and semantically labeled images from the 1st training round to the tth training round, the cumulative mIoU of the segmentation task in the tth training round is determined, where t is an integer greater than 1; Based on the cumulative mIoU of the segmentation task in training round t, determine the adjustment baseline value for training round t; Based on the segmentation loss function values ​​of training round t and training round (t-1), and the semantic consistency loss function values ​​of training round t and training round (t-1), the loss change rate of training round t is determined, and based on the loss change rate of training round t, the convergence factor of training round t is determined. Based on the adjustment baseline value of training round t, determine the semantic consistency coefficient of training round t; and based on the adjustment baseline value and the convergence factor of training round t, determine the segmentation coefficient of training round t; the segmentation coefficient of training round t is greater than the semantic consistency coefficient of training round t. The model parameters are updated based on the fusion loss function value, the segmentation loss function value, and the semantic consistency loss function value to obtain a multimodal image collaborative fusion model, including: Based on the fusion loss function value, the segmentation loss function value and the segmentation coefficient of the t-th training round, as well as the semantic consistency loss function value and the semantic consistency coefficient of the t-th training round, the model parameters are updated to obtain a multimodal image collaborative fusion model.

Images