A Multimodal Image Fusion Method and System Based on Dynamic Frequency Domain Fusion

Abstract

This invention relates to the field of image fusion technology, specifically disclosing a multimodal image fusion method and system based on dynamic frequency domain fusion. The method includes forming a dual-channel input tensor based on a first modality medical image and a second modality medical image; inputting the dual-channel input tensor into an encoder path, where each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module; inputting the final output of the encoder path into a bottleneck layer, where the bottleneck layer is composed of at least one multi-frequency multi-scale attention enhancement module to obtain a deep enhanced feature representation; inputting the deep enhanced feature representation into a decoder path, mapping the high-resolution reconstructed feature map into a single-channel image to obtain the final fused image. This invention employs a hierarchical reconstruction structure with five levels of encoding, five levels of decoding, and cross-layer skip connections, which is beneficial for balancing deep semantic recovery and shallow detail reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, specifically to a multimodal image fusion method and system based on dynamic frequency domain fusion. Background Technology

[0002] The purpose of multimodal medical image fusion is to effectively integrate complementary information contained in different imaging modalities into the same image, so that the fusion result can simultaneously retain anatomical structural details as well as functional or metabolic response information, thereby providing more comprehensive and reliable image evidence for lesion localization, clinical auxiliary diagnosis and subsequent treatment effect evaluation.

[0003] Existing medical image fusion methods mainly fall into two categories: traditional fusion methods and deep learning-based fusion methods. Traditional fusion methods typically rely on mathematical transformation tools such as wavelet transform, Laplacian pyramid, principal component analysis, and sparse representation to integrate information from different modalities through pre-defined fusion rules. While these methods are characterized by clear principles and relatively simple implementation, their heavy reliance on manual design of feature representations and fusion criteria makes it difficult to adequately characterize the complex nonlinear coupling relationships between different modalities, potentially leading to weakening of key structural information or loss of significant functional information.

[0004] With the development of deep learning technology, medical image fusion methods based on convolutional neural networks, autoencoders, and generative models have been extensively studied and have improved fusion performance to some extent. However, existing solutions still have the following shortcomings in practical applications: First, most methods mainly complete feature extraction and information fusion in the spatial domain, with relatively insufficient mining and utilization of frequency domain information, making it difficult to effectively adaptively adjust high-frequency texture information, low-frequency structural information, and mid-frequency transition information in different modal images; Second, the receptive field of conventional convolution operations is limited, resulting in insufficient ability to jointly model the relationship between local details and large-scale structures, making it difficult to maintain the consistency of the overall image structure while preserving edge details; Third, the information interaction methods between different branches or modalities in existing methods are usually relatively fixed, lacking the ability to dynamically adjust based on the input feature state, making it difficult to reasonably allocate the contributions of different modalities in different regions; In addition, in the process of deep feature extraction and subsequent reconstruction, multi-scale information and frequency response information are often not fully mined and effectively utilized, resulting in room for further improvement in terms of structural clarity, texture detail preservation, and target saliency expression of the fusion results.

[0005] Therefore, how to propose a multimodal medical image fusion method that can take into account multi-scale spatial structure modeling, dynamic frequency domain selection, and multi-frequency enhancement expression, so as to improve the overall quality and clinical application value of fused images, remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal image fusion method and system based on dynamic frequency domain fusion to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A multimodal image fusion method based on dynamic frequency domain fusion, the method comprising: Acquire the first modality medical image and the second modality medical image, and stitch the two images together in the channel dimension to form a dual-channel input tensor; The dual-channel input tensor is input into the encoder path. The encoder includes a multi-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The basic convolutional feature extraction module is used to extract local structural and texture information at the current scale; the split multi-scale feature module is used to perform multi-branch partitioning and adaptive fusion of the input features, and output multi-scale enhanced features; the dynamic frequency domain selection fusion module is used to receive two features from the basic convolutional feature extraction module and the split multi-scale feature module, adaptively generate inter-modal fusion weights based on the content of the input features, and selectively fuse the two features through dynamic frequency domain filtering to obtain the fused features at the current scale. The final output of the encoder path is input into the bottleneck layer, which consists of at least one multi-frequency multi-scale attention enhancement module, used to enhance the key spectral components and semantic representation capabilities in the deep features, to obtain the deep enhanced feature representation. The deep enhancement feature representation is input into the decoder path. The decoder includes a multi-level layer-by-layer upsampling reconstruction structure. In each level of decoding, the current low-resolution feature is upsampled and then fused with the encoder output of the corresponding scale via a skip connection. After convolutional reconstruction, the spatial resolution and texture details are restored step by step to obtain a high-resolution reconstructed feature map. The high-resolution reconstructed feature map is mapped to a single-channel image to obtain the final fused image.

[0008] As a further improvement of the present invention, the split multi-scale feature module performs the following operations: The input features are divided into local branches, context branches, and lightweight branches along the channel dimension; The local branch uses a combination of depthwise convolution and pointwise convolution to extract edge and texture details; the context branch uses large kernel decomposition convolution and dilated convolution to extract structural information under a large receptive field; and the lightweight branch uses pointwise convolution to maintain the basic intensity response. Global average pooling is performed on the original input to generate three-way gated weights. Based on the three-way gated weights, the output features of the three branches are concatenated by channels, and then the fusion result is obtained by channel shuffling and pointwise convolution. The fusion result is added to the shortcut mapping of the input features through residual connections and then output through an activation function.

[0009] As a further improvement of the present invention, the dynamic frequency domain selection fusion module performs the following operations: The system receives the first and second features and preprocesses them using lightweight convolution. The preprocessed features are then summed to generate conditional features, which are then mapped to adaptive weights corresponding to the two features. For any feature, input it into the dynamic frequency domain filtering submodule to obtain the dynamic frequency domain filtering result of that feature; The dynamic frequency domain filtering results of the two features are added together to obtain the fused features at the current scale.

[0010] As a further improvement of the present invention, the step of inputting any feature into the dynamic frequency domain filtering submodule to obtain the dynamic frequency domain filtering result of the feature includes: mapping the spatial domain feature to the frequency domain through Fourier transform. Filter combination coefficients are generated based on the aforementioned conditional features, and the coefficients are used to perform weighted combination on a pre-set complex-valued basic filter bank to obtain a sample-adaptive and channel-adaptive frequency domain kernel. Perform element-wise multiplication between the frequency domain kernel and the spectral features; The result is then restored to the spatial domain by inverse Fourier transform and added to the residual term to obtain the dynamic frequency domain filtering result of this path feature.

[0011] As a further improvement of the present invention, the dynamic frequency domain filtering submodule further includes: performing reflection filling before performing Fourier transform on the input features; After the inverse Fourier transform, pruning is performed to restore the original size; the complex-valued fundamental filter bank includes real and imaginary parts, which are adapted to the frequency domain size of the current scale after interpolation transformation.

[0012] As a further improvement of the present invention, each level of coding structure also includes a spatial-frequency joint enhancement module. The spatial-frequency joint enhancement module is used to receive the fusion feature output by the dynamic frequency domain selection fusion module, and enhance the feature from the spatial domain and the frequency domain respectively. Then, the spatial enhancement result is fused with the frequency enhancement result, and residual superimposed with the input feature to output the enhanced skip connection feature.

[0013] As a further improvement of the present invention, the space-frequency joint enhancement module performs the following operations: Perform layer normalization on the input features, and then input the normalized features into both the spatial enhancement branch and the frequency enhancement branch. The spatial augmentation branch includes: extracting local spatial context through convolution to obtain intermediate features; dividing the intermediate features into a first subset and a second subset along the channel dimension; applying global average pooling to the first subset to generate a first gate weight; applying global max pooling to the second subset to generate a second gate weight; multiplying the two subsets by their corresponding gate weights and concatenating them; and adding them to the local spatial context to obtain the spatial augmentation result. The frequency enhancement branch includes: performing a Fourier transform on the normalized features to obtain a complex spectrum, decomposing the complex spectrum into an amplitude spectrum and a phase spectrum, modulating the amplitude spectrum and phase spectrum respectively, reconstructing the complex spectrum and restoring it to the spatial domain through an inverse Fourier transform, and then adding it to the normalized feature residual to obtain the frequency enhancement result. The spatial augmentation result is added to the frequency augmentation result, and then superimposed with the input feature residual to output the joint augmentation result.

[0014] As a further improvement of the present invention, the multi-frequency multi-scale attention enhancement module performs the following operations: Input features are processed in parallel at multiple scales. For each scale, average pooling is performed first, and then multi-scale context is extracted by dilated depthwise convolution and pointwise convolution to obtain the intermediate features of that scale. Construct multiple fixed discrete cosine filters based on predefined discrete cosine transform frequency coordinates; The intermediate features are adaptively pooled to the target discrete cosine size, multiplied pointwise with each discrete cosine filter, and the average, maximum and minimum statistics are extracted. The statistics of multiple frequency bands are then averaged and channel attention weights are generated through shared pointwise convolution mapping. Spatial attention maps are generated for each scale, and intermediate features at that scale are differentially enhanced by combining background suppression coefficients and salient region enhancement coefficients. The enhancement results at each scale are upsampled to the original spatial scale and averaged, then superimposed with the input feature residuals to output multi-frequency, multi-scale enhanced features.

[0015] This invention also provides a multimodal image fusion system based on dynamic frequency domain fusion, the system comprising: The image stitching unit is used to acquire a first modality medical image and a second modality medical image, and stitch the two images together in the channel dimension to form a dual-channel input tensor; A step-by-step encoding unit is used to input the dual-channel input tensor into the encoder path. The encoder includes a multi-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The basic convolutional feature extraction module is used to extract local structural and texture information at the current scale; the split multi-scale feature module is used to perform multi-branch partitioning and adaptive fusion of the input features, and output multi-scale enhanced features; the dynamic frequency domain selection fusion module is used to receive two features from the basic convolutional feature extraction module and the split multi-scale feature module, adaptively generate inter-modal fusion weights based on the content of the input features, and selectively fuse the two features through dynamic frequency domain filtering to obtain the fused features at the current scale. An enhancement unit is used to input the final output of the encoder path into a bottleneck layer. The bottleneck layer consists of at least one multi-frequency multi-scale attention enhancement module, which is used to enhance the key spectral components and semantic representation capabilities in deep features to obtain deep enhanced feature representations. The decoding unit is used to input the deep enhanced feature representation into the decoder path. The decoder includes a multi-level layer-by-layer upsampling reconstruction structure. In each level of decoding, the current low-resolution feature is upsampled and then fused with the encoder output of the corresponding scale through a skip connection. After convolutional reconstruction, the spatial resolution and texture details are restored step by step to obtain a high-resolution reconstructed feature map. The fusion unit is used to map the high-resolution reconstructed feature map into a single-channel image to obtain the final fused image.

[0016] Compared with the prior art, the beneficial effects of the present invention are: the present invention adopts a unified backbone processing method after stitching together dual-modal image channels, so that information of different modalities enters the same feature space at the shallow stage, which is conducive to improving the interaction efficiency of cross-modal information; through the split multi-scale feature modeling process, local details, contextual structure and stable response information are extracted at the same time, which improves the ability to represent lesions, edges and tissue structures at different scales. By using a dynamic frequency domain selection fusion module, different frequency responses are adaptively selected in the frequency domain based on the input feature state, achieving more refined modal information control than the fixed convolution fusion method. By using a spatial-frequency joint enhancement module, both the spatial salient region response and the frequency domain structural response are strengthened, which helps to improve the texture clarity and overall contrast of the fused image. By introducing a multi-frequency, multi-scale attention enhancement structure in the bottleneck layer, the expressive power of key spectral information in deep semantic features is further improved, thereby enhancing the target saliency and structural integrity in the fused image.

[0017] The hierarchical reconstruction structure, which employs five-level encoding, five-level decoding, and cross-layer skip connections, is beneficial for balancing deep semantic recovery and shallow detail reconstruction, thereby improving the visual naturalness of the fusion results and their clinical auxiliary application value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0019] Figure 1 This is a diagram illustrating the overall architecture of the method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the structure of the split multi-scale feature modeling module provided in an embodiment of the present invention.

[0021] Figure 3 This is a structural diagram of the dynamic frequency domain selection fusion module provided in an embodiment of the present invention.

[0022] Figure 4 The dynamic frequency structure diagram provided for the embodiments of the present invention.

[0023] Figure 5 This is a structural diagram of a multi-frequency, multi-scale attention enhancement module provided in an embodiment of the present invention.

[0024] Figure 6 The diagram shows the structure of the space-frequency joint enhancement module provided in this embodiment of the invention. Detailed Implementation

[0025] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0026] like Figure 1 As shown in the embodiment of the present invention, the multimodal image fusion method based on dynamic frequency domain fusion includes: Acquire the first modality medical image and the second modality medical image, and stitch the two images together in the channel dimension to form a dual-channel input tensor; The dual-channel input tensor is input into the encoder path. The encoder includes a multi-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The basic convolutional feature extraction module is used to extract local structural and texture information at the current scale; the split multi-scale feature module is used to perform multi-branch partitioning and adaptive fusion of the input features, and output multi-scale enhanced features; the dynamic frequency domain selection fusion module is used to receive two features from the basic convolutional feature extraction module and the split multi-scale feature module, adaptively generate inter-modal fusion weights based on the content of the input features, and selectively fuse the two features through dynamic frequency domain filtering to obtain the fused features at the current scale. The final output of the encoder path is input into the bottleneck layer, which consists of at least one multi-frequency multi-scale attention enhancement module, used to enhance the key spectral components and semantic representation capabilities in the deep features, to obtain the deep enhanced feature representation. The deep enhancement feature representation is input into the decoder path. The decoder includes a multi-level layer-by-layer upsampling reconstruction structure. In each level of decoding, the current low-resolution feature is upsampled and then fused with the encoder output of the corresponding scale via a skip connection. After convolutional reconstruction, the spatial resolution and texture details are restored step by step to obtain a high-resolution reconstructed feature map. The high-resolution reconstructed feature map is mapped to a single-channel image to obtain the final fused image.

[0027] In this embodiment, during the encoding stage, a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module are introduced to enable the input multimodal medical image to complete the collaborative representation of local detail information, regional structural information, and frequency response information at different levels. During the bottleneck stage, a multi-scale multi-frequency attention enhancement module is further introduced to strengthen the expression ability of key spectral components and significant structural information in deep features. During the decoding stage, through stepwise feature recovery and cross-layer information fusion, the effective reconstruction of deep semantic information and shallow detail information is achieved, thereby realizing the efficient fusion of structural information, texture information, and significant target information in multimodal medical images.

[0028] Preliminary data acquisition: Acquire the first modality MRI medical image and the second modality PET medical image to be fused. Spatial registration, size unification, and grayscale normalization are performed on the two images to ensure consistent spatial resolution and pixel scale. Subsequently, the first modality MRI medical image and the second modality PET medical image are stitched together along the channel dimension to form a dual-channel input tensor. This input tensor is then input into the shallow feature mapping module, where initial edge information and low-level texture features are extracted through continuous convolution operations to obtain a shallow basic feature map.

[0029] The shallow basic feature map is input into the encoder path. The encoder includes a five-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The encoding output of each level is downsampled and then sent to the next level of the encoding structure to progressively expand the receptive field, increase the channel dimension, and extract higher-level semantic information.

[0030] The basic convolutional feature extraction module is used to obtain local structural and texture information at the current scale; the split multi-scale feature modeling is used to divide the input features into multiple branches and represent local details, contextual structure, and lightweight stable response respectively; the dynamic frequency domain selection fusion is used to adaptively generate inter-modal fusion weights based on the input feature content and achieve selective fusion of the two features through dynamic frequency domain filtering.

[0031] After the five-level encoding is completed, the deep features enter the bottleneck enhancement stage. This bottleneck enhancement stage consists of multiple cascaded multi-frequency, multi-scale attention enhancement modules, aiming to further enhance the multi-scale contextual representation ability and key spectral component characterization ability of the deep features. The bottleneck-enhanced deep features are then input into the five-level decoding structure. The decoding stage employs a step-by-step upsampling and skip connection fusion approach for feature recovery. In each level of decoding, the current low-resolution features are first upsampled, then concatenated and fused with the skip connection enhanced features of the corresponding scale. Subsequently, spatial resolution and texture details are restored step-by-step through a convolutional reconstruction process, ultimately obtaining a high-resolution reconstructed feature map. Finally, the input-output mapping process of the reconstructed feature map is compressed into a single-channel output through pointwise convolution, and then processed by a hyperbolic tangent activation function to generate the final fused image. This fused image can simultaneously retain the structural boundary information in the first modality MRI medical image and the significant functional information in the second modality PET medical image.

[0032] like Figure 2 As shown, in a preferred embodiment of the present invention, the split multi-scale feature module performs the following operations: The input features are divided into local branches, context branches, and lightweight branches along the channel dimension; The local branch uses a combination of depthwise convolution and pointwise convolution to extract edge and texture details; the context branch uses large kernel decomposition convolution and dilated convolution to extract structural information under a large receptive field; and the lightweight branch uses pointwise convolution to maintain the basic intensity response. Global average pooling is performed on the original input to generate three-way gated weights. Based on the three-way gated weights, the output features of the three branches are concatenated by channels, and then the fusion result is obtained by channel shuffling and pointwise convolution. The fusion result is added to the shortcut mapping of the input features through residual connections and then output through an activation function.

[0033] In this embodiment, during each level of encoding, the split-scale multi-scale feature module first divides the current scale input feature into multiple branches along the channel direction, including local branches, context branches, and lightweight branches. The local branches use a combination of depthwise convolution and pointwise convolution to extract edge and texture details, the context branches use large kernel decomposition convolution and dilated convolution to extract structural information under a large receptive field, and the lightweight branches are used to maintain the basic intensity response and feature stability. Based on this, the features of each branch are adaptively weighted, channel rearranged, and fused, and multi-scale enhanced features are output through residual connections.

[0034] Furthermore, the split-scale feature module is used to perform domain decomposition and heterogeneous representation of the input features, assuming the input features are: ; It is divided into three parts along the channel dimension: ; in, This indicates a channel partitioning operation. Let each represent the number of channels in the three subspaces, satisfying... ; in, Corresponding to the local detail subspace, Corresponding context structure subspace, Corresponding to a lightweight, stable subspace.

[0035] Local branches extract high-frequency textures and edge details through depthwise convolution and pointwise convolution: ; in, Indicates the output characteristics of local branches; Indicates the kernel size as Channel-wise convolution operator; express Point-to-convolution mapping operator; Indicates group normalization, This represents a smooth, nonlinear activation function.

[0036] Context branches extract long-range structural dependencies through large-kernel horizontal decomposition convolution and dilated convolution. The computation process is represented as follows: ; ; ; ; in, Indicates input features The second sub-feature obtained after channel segmentation is used as the input for the context branch; express through The first intermediate feature obtained after channel-wise large kernel convolution processing; express through The second intermediate feature obtained after channel-wise large kernel convolution processing; express The expansion rate is of The third intermediate feature obtained after channel-wise dilated convolution processing; This indicates the output features of the context branch; Indicates the kernel size as Channel-wise convolution operator; Indicates the kernel size as Channel-wise convolution operator; Indicates the kernel size as Expansion rate Channel-wise dilated convolution operator; Indicates large core size, preferably selected ; The dilation rate, representing the dilation rate of dilated convolution, is taken as in this embodiment. .

[0037] Lightweight branches use only pointwise convolutions to maintain low-frequency intensity and overall stability: ; in, This indicates the output characteristics of lightweight branches; Indicates input features The third sub-feature obtained after channel partitioning is used as the input for the lightweight branch.

[0038] After obtaining the three features, global average pooling is performed on the original input to generate three gated weights: ; in, This indicates a global average pooling operation; Indicates two continuous layers The gated mapping constructed by convolution has a final output channel count of 3; This represents a function that performs normalization along the three branch dimensions. Indicates the weights of the three paths. These correspond to local branches, context branches, and lightweight branches, respectively, and satisfy the following conditions: ;in, When multiplied with the corresponding branch feature, it is broadcast along the channel dimension and the spatial dimension.

[0039] Then, the three features are weighted and concatenated: ; in, This represents the feature obtained by concatenating the three weighted features along the channel dimension; This indicates a channel dimension splicing operation.

[0040] Then, the fusion result is obtained through channel shuffling and pointwise convolution: ; in, This indicates channel rearrangement, used to enhance cross-branch information exchange. Indicates the use of the fusion phase Convolution mapping; This indicates the characteristics after channel shuffling and point-by-point fusion.

[0041] If spatial gating is further introduced, then... ; Finally, the output is presented as a residual reconstruction with a learnable scaling factor: ; in This represents the output feature of the split multi-scale feature module; Representing input features Shortcut mapping, when the number of input and output channels is the same. When the number of input and output channels is inconsistent, Depend on Obtained by convolution mapping; This represents the learnable scaling factor, used to control the incremental contribution of the augmented branch to the main path. This residual reconstruction method allows the augmented features to be superimposed on the input features in a controlled incremental manner, rather than directly replacing the original input.

[0042] like Figure 3 and Figure 4 As shown, in a preferred embodiment of the present invention, the dynamic frequency domain selection fusion module performs the following operations: The system receives the first and second features and preprocesses them using lightweight convolution. The preprocessed features are then summed to generate conditional features, which are then mapped to adaptive weights corresponding to the two features. For any feature, input it into the dynamic frequency domain filtering submodule to obtain the dynamic frequency domain filtering result of that feature; The dynamic frequency domain filtering results of the two features are added together to obtain the fused features at the current scale.

[0043] In this embodiment, the dynamic frequency domain selection and fusion module receives two sets of features from the basic convolution, the convolutional feature extraction module, and the split multi-scale feature module. First, the two sets of features are preprocessed using lightweight convolution. Then, conditional information is extracted based on the sum of the two sets of features, and two sets of corresponding modal weights are generated. Subsequently, the two sets of features are input into the dynamic frequency domain filtering submodule, and different frequency responses are adaptively selected and modulated in the frequency domain according to the modal weights. Finally, the two results after dynamic frequency domain filtering are fused to obtain the fused features at the current scale.

[0044] Preferably, the dynamic frequency domain filtering submodule includes processing steps such as linear projection, dynamic routing, two-dimensional Fourier transform, inverse Fourier transform, and residual connection. Specifically, linear projection maps input features to the intermediate channel space to improve the representation capability of subsequent frequency domain modulation; dynamic routing generates combination coefficients for multiple complex-valued frequency domain filters based on conditional information, thereby achieving adaptive selection for different frequency responses; the two-dimensional Fourier transform maps spatial domain features to the frequency domain; the inverse Fourier transform restores the frequency-enhanced result to the spatial domain; and the residual connection preserves the original structural information and improves the stability of the frequency-enhanced process.

[0045] The dynamic frequency domain selection fusion module is used to achieve selective modulation and effective fusion of two complementary features at different frequency components through condition-guided frequency domain adaptive filtering. Let the two input features be: Let the first-path features and the second-path features at the l-th scale be respectively: ; in, Indicates the first The first feature obtained from the basic convolutional feature extraction path at each scale; Indicates the first The second feature path is obtained from the split multi-scale feature path at each scale; Indicates batch size; Indicates the first Number of channels at each scale; and They represent the first The height and width of the feature map at each scale.

[0046] First, perform lightweight convolutional preprocessing on both feature paths: First, lightweight convolutional preprocessing is performed to enhance local expressiveness while preserving channel interactions: ; ; in, Indicates the first-way feature The result after light convolution preprocessing; Indicates the second-path feature The result after light convolution preprocessing; Represents linear combination of channels Point-to-point mapping; Represents local spatial response modeling Channel-wise convolution; This represents the linear rectification activation function.

[0047] Subsequently, the two preprocessed features are summed to generate conditional features: ; in, Indicates the first Fusion conditional features at various scales; This represents the joint response of the two preprocessing features; This represents the normalization operator; the meanings of the other symbols are consistent with the aforementioned definitions.

[0048] Then, two-way competitive selection weights are generated from the conditional features: ; And satisfy: ; in, Indicates the weights used to generate selection weights Point-to-mapping operator; This indicates a tensor rearrangement operation, used to rearrange the mapping result into a two-way weight tensor; This indicates that normalization is performed along the path dimensions of the two features; This represents the adaptive selection weights corresponding to the first feature path; This represents the adaptive selection weights corresponding to the second-path features; Indicates and and A single sheet of uniform size. This constraint indicates that the initial selection responses of the two features in terms of channel location and spatial location are competitive; after global statistics, this competitive relationship is further transformed into sample-level and channel-level frequency domain filter combination conditions, which are used to guide the subsequent dynamic frequency domain modulation process.

[0049] like Figure 3 and Figure 4 As shown, in a preferred embodiment of the present invention, the step of inputting any feature into the dynamic frequency domain filtering submodule to obtain the dynamic frequency domain filtering result of the feature includes: mapping the spatial domain feature to the frequency domain via Fourier transform; Filter combination coefficients are generated based on the aforementioned conditional features, and the coefficients are used to perform weighted combination on a pre-set complex-valued basic filter bank to obtain a sample-adaptive and channel-adaptive frequency domain kernel. Perform element-wise multiplication between the frequency domain kernel and the spectral features; The result is then restored to the spatial domain by inverse Fourier transform and added to the residual term to obtain the dynamic frequency domain filtering result of this path feature.

[0050] The dynamic frequency domain filtering submodule further includes: performing reflection filling before performing Fourier transform on the input features; After the inverse Fourier transform, pruning is performed to restore the original size; the complex-valued fundamental filter bank includes real and imaginary parts, which are adapted to the frequency domain size of the current scale after interpolation transformation.

[0051] In this embodiment, for any dynamic frequency domain filter, let its input be... The corresponding conditional guiding weight is ,in: ; in, Indicates the first Input features of any path to be dynamically filtered in the frequency domain at any scale; Indicates and Corresponding conditional guided weights; when hour, ;when hour, .

[0052] To facilitate frequency domain mapping, Convert from channel-priority form to space-priority form: ; in, This indicates a dimension replacement operation; This represents the feature to be filtered after conversion to space-priority form.

[0053] right Perform a linear projection to obtain the representation in the intermediate channel space: ; in, Represents the intermediate features after projection; Indicates the input linear projection operator; Represents a nonlinear mapping function; Indicates the number of intermediate channels.

[0054] Before performing frequency domain transformation, Perform boundary reflection filling: ; in, This indicates the features after being filled by boundary reflection; This indicates a reflection fill operation, used to reduce boundary artifacts during frequency domain transformation.

[0055] Then, a two-dimensional real-valued fast Fourier transform is performed on the filled features: ; in, Represents the spectral characteristics of complex numbers; This represents a two-dimensional real-valued fast Fourier transform. Indicates the height after reflection filling; This represents the dimension in the frequency width direction after the real-valued Fourier transform; Represents the complex field.

[0056] The dynamic frequency domain filtering submodule is internally configured with a complex-valued fundamental filter bank: ; in, This represents the set of parameters of a complex-valued fundamental filter bank; Indicates the reference frequency domain height; Indicates the reference frequency bandwidth; The number of fundamental filters is indicated; the last dimension, 2, represents the real and imaginary parts of the complex filter, respectively.

[0057] Based on the spectral size at the current scale, the complex-valued fundamental filter bank is interpolated and adapted to obtain the complex-valued frequency domain filter bank at the current scale: ; in, Indicates the first Complex-valued frequency domain filter banks adapted at each scale; This indicates an interpolation adaptation operation; and These represent the height and frequency bandwidth dimensions of the spectral feature at the current scale, respectively. Indicates the number of basic filters.

[0058] At the same time, weights are guided by conditions. Global statistics for generating filter combination coefficients: ; in, Represents the filter combination coefficients; This indicates a global average pooling operation; This represents a lightweight mapping function used to generate filter combination weights from conditional statistics; This indicates normalization along the dimension of the fundamental filter. Indicates batch size; Indicates the number of basic filters; Indicates the number of intermediate channels.

[0059] Based on the filter combination coefficients and the complex-valued frequency domain filter bank, construct a sample-adaptive and channel-adaptive dynamic frequency domain kernel: ; in, Indicates the first The first scale The sample, the first The intermediate channels are on the frequency coordinates Dynamic frequency domain kernel response at the location; Indicates the sample sequence number; and These represent the height and width coordinates in the frequency domain, respectively. Indicates the intermediate channel number; Indicates the basic filter number; Indicates the number of basic filters; Indicates the first The sample, the first The intermediate channel corresponds to the first... The combination coefficients of the basic filters; Indicates the first The fundamental filters on the frequency coordinates Complex value response at the location.

[0060] Applying dynamic frequency domain kernels to spectral features: ; in, This represents the complex spectral characteristics after frequency domain modulation; Representing input features Complex spectral characteristics obtained after projection, filling, and two-dimensional real-valued fast Fourier transform; Represents a dynamic frequency domain kernel; This indicates element-wise multiplication.

[0061] Then, the system is restored to the spatial domain through two-dimensional real-valued inverse Fourier transform, output projection, and clipping operations, and residual terms are added: ; in, This represents the output result of any feature after processing by the dynamic frequency domain filtering submodule; Represents the two-dimensional real-valued inverse Fourier transform; This indicates the output linear projection operator, used to convert the number of intermediate channels... Restore to the original number of channels ; This indicates a trimming operation, used to restore the space to its original size before filling. This represents the input residual term of the dynamic frequency domain filtering submodule.

[0062] Therefore, the dynamic frequency domain filtering results of the first feature and the second feature are expressed as follows: ; ; in, This represents the dynamic frequency domain filtering result of the first feature path; This represents the dynamic frequency domain filtering result of the second feature; Indicates the first Dynamic frequency domain filtering operators at various scales; This indicates the inverse dimension permutation operation, used to restore the space-priority form to the channel-priority form.

[0063] Finally, the results of the two dynamic frequency domain filtering paths are added together to obtain the fused features at the current scale: ; in, Indicates the first The fusion features output by the dynamic frequency domain selection fusion module at each scale; and These represent the dynamic frequency domain filtering results of the first feature and the second feature, respectively.

[0064] like Figure 6 As shown, in a preferred embodiment of the present invention, each level of coding structure further includes a spatial-frequency joint enhancement module. The spatial-frequency joint enhancement module is used to receive the fusion feature output by the dynamic frequency domain selection fusion module, and enhance the feature from the spatial domain and the frequency domain respectively. Then, the spatial enhancement result is fused with the frequency enhancement result, and residual superimposed with the input feature to output the enhanced skip connection feature.

[0065] In this embodiment, shallow basic features are sequentially input into a five-level coding structure. First, the local texture response and structural boundary information at the current scale are obtained through a basic convolutional feature extraction process. Simultaneously, the same input feature is sent to a split multi-scale feature module to perform multi-branch partitioning and heterogeneous scale representation on the input feature. Subsequently, the results from the above two paths are input into a dynamic frequency domain selection and fusion process to complete adaptive selection and complementary fusion in the frequency domain. Finally, the key spatial regions and frequency response information are further enhanced by a spatial-frequency joint enhancement module to obtain enhanced features at the current scale. These enhanced features are downsampled and input into the next coding level, and are also retained as skip connection features at the corresponding scale for feature recovery and detail reconstruction in the subsequent decoding stage.

[0066] The spatial-frequency joint enhancement module comprises two parallel parts: spatial enhancement and frequency enhancement. Spatial enhancement uses convolutional transformation and dual-channel partitioning to generate gated weights for different channel subsets using average pooling and max pooling statistical methods, thereby enhancing the response of key spatial structures and salient regions. Frequency enhancement maps input features to the frequency domain using Fourier transform, performs convolutional modulation on the amplitude and phase spectra respectively, and then restores them to the spatial domain using inverse Fourier transform to enhance frequency expressive power. Subsequently, the spatial enhancement result and the frequency enhancement result are fused and residually superimposed with the input features to obtain the enhanced skip connection features.

[0067] like Figure 6 As shown, in a preferred embodiment of the present invention, the space-frequency joint enhancement module performs the following operations: Perform layer normalization on the input features, and then input the normalized features into both the spatial enhancement branch and the frequency enhancement branch. The spatial augmentation branch includes: extracting local spatial context through convolution to obtain intermediate features; dividing the intermediate features into a first subset and a second subset along the channel dimension; applying global average pooling to the first subset to generate a first gate weight; applying global max pooling to the second subset to generate a second gate weight; multiplying the two subsets by their corresponding gate weights and concatenating them; and adding them to the local spatial context to obtain the spatial augmentation result. The frequency enhancement branch includes: performing a Fourier transform on the normalized features to obtain a complex spectrum, decomposing the complex spectrum into an amplitude spectrum and a phase spectrum, modulating the amplitude spectrum and phase spectrum respectively, reconstructing the complex spectrum and restoring it to the spatial domain through an inverse Fourier transform, and then adding it to the normalized feature residual to obtain the frequency enhancement result. The spatial augmentation result is added to the frequency augmentation result, and then superimposed with the input feature residual to output the joint augmentation result.

[0068] In this embodiment, the spatial-frequency joint enhancement module is further used to jointly enhance the input features at both the spatial and frequency domain levels to improve the comprehensive representation ability of the features for local structural details and global frequency response. Further, in the spatial-frequency joint enhancement module, let the first... The fusion features output by the dynamic frequency domain selection fusion module at each scale are as follows: ; in, Indicates the first Fusion characteristics of input space-frequency joint enhancement modules at various scales; Indicates batch size; Indicates the first The number of channels for fused features at each scale; and They represent the first The height and width of fused features at each scale; Represents the real number field.

[0069] First, a two-dimensional layer normalization transformation suitable for the convolutional layout is performed on the input features.

[0070] This transformation involves rearranging the feature tensor into a channel-wise form, normalizing it along the channel dimension, and then restoring the original dimensions. Its mathematical form can be expressed as: ; ; in, This represents the fusion characteristics after normalization; Representation layer normalization operation; This represents the eigenvalue at the corresponding position before normalization; Indicates the first The spatial location of each sample The mean value calculated along the channel dimension; Indicates the first The spatial location of each sample The variance calculated along the channel dimension; Indicates the first The first scale The learnable scaling parameters corresponding to each channel; Indicates the first The first scale The learnable offset parameters corresponding to each channel; This represents the smallest positive number that prevents the denominator from being zero.

[0071] The purpose of this step is to eliminate the difference in response amplitude between different channels and improve the stability of subsequent spatial enhancement and frequency enhancement calculations.

[0072] The normalized fusion features Simultaneously input spatial enhancement branch and frequency enhancement branch.

[0073] For the spatial augmentation branch, the normalized fused features are first subjected to local convolution mapping to obtain spatial context features: ; in, Indicates the first Spatial context features at various scales; In the spatial enhancement branch Convolution mapping; This represents the linear rectification activation function.

[0074] Subsequently, spatial context features Divide into two subsets along the channel dimension: ; in, This indicates an operation that divides the channel into two parts along the channel dimension; The first channel subset representing spatial context features; The second channel subset representing spatial context features.

[0075] Perform global average pooling on the first channel subset and generate the first gate weights: ; Perform global max pooling on the second channel subset and generate the second gate weights: ; in, This represents the spatial gating weight corresponding to the first channel subset; This represents the spatial gating weights corresponding to the second channel subset; This indicates a global average pooling operation, used to extract the overall statistical response; This represents the global max pooling operation, used to extract significant responses; and These represent two independent nonlinear point-to-point mapping functions; This represents the Sigmoid gate function, used to constrain the gate weights to... to Within the range.

[0076] Therefore, the output of the spatial augmentation branch is expressed as: ; in, Indicates the first Spatial augmentation results at various scales; This indicates an operation to splice along the channel dimension; This represents element-wise multiplication; This represents the spatial context residual term. This process enhances the dynamic response to texture, edge, and organizational patterns through the complementary effects of average and extreme value statistics.

[0077] For the frequency enhancement branch, first analyze the normalized fusion features. Performing a two-dimensional real-valued fast Fourier transform yields the complex spectrum representation: ; in, Indicates the first Complex spectrum corresponding to normalized fusion features at each scale; This represents a two-dimensional real-valued Fast Fourier Transform; the Fourier transform employs an orthogonal normalization method.

[0078] Subsequently, the complex spectrum Decomposed into amplitude spectrum and phase spectrum: ; ; in, Indicates the first Amplitude spectra at various scales are used to characterize frequency energy distribution; Indicates the first Phase spectra at various scales are used to characterize structural positional relationships; This represents the modulo operation for complex numbers; This indicates the phase take operation for complex numbers.

[0079] Modulate the amplitude spectrum and phase spectrum separately: ; ; in, This represents the modulated amplitude spectrum; This represents the modulated phase spectrum; Indicates the first Amplitude spectrum modulation mapping at various scales; Indicates the first Phase spectrum modulation mapping at various scales.

[0080] Then, the complex spectrum is reconstructed based on the modulated amplitude spectrum and phase spectrum: ; in, This represents the complex spectrum obtained after modulation and reconstruction. Represents the imaginary unit; Represents the cosine function; This represents the sine function.

[0081] The spatial domain is then restored using a two-dimensional real-valued inverse Fourier transform, and the residuals are superimposed with the normalized fused features. ; in, Indicates the first Frequency enhancement results at each scale; Represents the two-dimensional real-valued inverse Fourier transform; This indicates that the inverse transformation restores the original state to the state before the transformation. Original spatial dimensions at each scale; This represents the normalized fusion feature residual term.

[0082] Finally, the spatial augmentation and frequency augmentation results are added in the spatial domain and then reconstructed using residuals with the input fusion features to obtain the output of the spatial-frequency joint augmentation module: ; in, Indicates the first Enhanced skip connectivity features output by the spatial-frequency joint enhancement module at each scale; This indicates the input fusion features of the module; This indicates spatially enhanced branch output; Indicates the frequency boosting branch output; This indicates an optional random depth regularization operation. If this operation is not enabled, then... It can degenerate into an identity mapping.

[0083] This design takes into account both spatial local detail modeling and frequency domain global structure modeling, which helps to improve the structural fidelity and texture representation capabilities of fused features.

[0084] like Figure 5 As shown, in a preferred embodiment of the present invention, the multi-frequency multi-scale attention enhancement module performs the following operations: Input features are processed in parallel at multiple scales. For each scale, average pooling is performed first, and then multi-scale context is extracted by dilated depthwise convolution and pointwise convolution to obtain the intermediate features of that scale. Construct multiple fixed discrete cosine filters based on predefined discrete cosine transform frequency coordinates; The intermediate features are adaptively pooled to the target discrete cosine size, multiplied pointwise with each discrete cosine filter, and the average, maximum and minimum statistics are extracted. The statistics of multiple frequency bands are then averaged and channel attention weights are generated through shared pointwise convolution mapping. Spatial attention maps are generated for each scale, and intermediate features at that scale are differentially enhanced by combining background suppression coefficients and salient region enhancement coefficients. The enhancement results at each scale are upsampled to the original spatial scale and averaged, then superimposed with the input feature residuals to output multi-frequency, multi-scale enhanced features.

[0085] In this embodiment, the fifth-level encoded output is downsampled and then fed into a bottleneck layer. This bottleneck layer consists of multiple cascaded multi-frequency, multi-scale attention enhancement modules. The multi-frequency, multi-scale attention enhancement structure includes a multi-scale branch and a multi-frequency channel attention branch. The multi-scale branch extracts multi-level contextual information through pooling and convolutional transformations at different scales. The multi-frequency channel attention branch constructs a multi-frequency response using a preset discrete cosine transform frequency basis and generates frequency channel weights using statistical pooling to enhance the key spectral components and semantic representation capabilities in the deep features. Multiple cascaded multi-frequency, multi-scale attention enhancement structures are then used to obtain a deep enhanced feature representation.

[0086] The multi-frequency, multi-scale attention enhancement module is used to jointly model the response characteristics of different spatial scales and different frequency sub-bands, so as to enhance the expressive power of key structural information and significant spectral components in deep features.

[0087] Let the first The input features of the multi-frequency, multi-scale attention enhancement module are: ; in, Indicates the first The input features of the multi-frequency, multi-scale attention enhancement module; This indicates the sequence number of the multi-frequency, multi-scale attention enhancement module in the bottleneck layer; Indicates batch size; Indicates the number of feature channels in the bottleneck layer; and These represent the height and width of the bottleneck layer feature, respectively. Represents the real number field.

[0088] Suppose this module contains The scale branch, for the ... There are several scale branches, among which: ; in, Indicates the number of scale branches; Indicates the scale branch index.

[0089] In the In each scale branch, the first step is to determine the scale factor. Average pooling is performed on the input features to obtain the scaled input features at this scale: ; in, Indicates the first The first multi-frequency, multi-scale attention enhancement module Pooling input features for each scale branch; Indicates the pooling kernel size and step size with The corresponding average pooling operation. When the spatial resolution at the current scale is lower than the preset minimum resolution, the average pooling operation degenerates into an identity mapping to avoid over-compressing spatial structure information.

[0090] Subsequently, contextual features at this scale are extracted using convolutional mappings with different receptive fields: ; in, Indicates the first In the module, the first Intermediate context features obtained from each scale branch; This indicates the number of intermediate channels output by this scale branch; and These represent the height and width of the output feature of this scale branch, respectively; Indicates the kernel size as Expansion rate The grouped convolution operator is used to expand the effective receptive field at this scale; Indicates the first The first module In each scale branch Point-to-point mapping operators are used for channel compression and channel reassembly. This represents the batch normalization operator; This represents the linear rectification activation function.

[0091] In multi-channel attention processing, pre-setting Discrete cosine transform frequency coordinates: ; in, This indicates the number of discrete cosine frequency branches selected, preferably... ; Indicates the frequency branch index; Indicates the first Frequency index of a frequency base in the height direction; Indicates the first A frequency base with frequency indices in the width direction.

[0092] Construct the first frequency coordinate based on the above. A fixed discrete cosine frequency basis: ; in, Indicates the first The discrete cosine frequency basis is at the th... individual channels, spatial locations The value at; Indicates the channel index; and These represent the position indices in the height and width directions within the discrete cosine dimension of the target, respectively. and These represent the height and width of the target's discrete cosine transform, respectively. This represents a one-dimensional discrete cosine basis function.

[0093] The one-dimensional discrete cosine basis function is defined as follows: ; in, Represents a one-dimensional position index; Represents a one-dimensional frequency index; Indicates the total length in that direction; Represents pi; This represents the cosine function; this function is used to construct orthogonal cosine bases in different directions and at different frequencies.

[0094] For the Each scale branch will incorporate intermediate context features. Adaptive pooling to target discrete cosine size: ; in, This represents the intermediate context features after adaptive pooling; This indicates that the input features are adaptively averaged and pooled to... Size manipulation; and These represent the height and width of the target discrete cosine transform dimension, respectively.

[0095] Will With the Multiplying each discrete cosine frequency basis point by point yields the frequency response of that frequency branch: ; in, Indicates the first The first module, the first The first scale branch, the first Discrete cosine frequency response corresponding to each frequency basis; This represents element-wise multiplication; Indicates the first A fixed discrete cosine frequency basis.

[0096] The average statistic, maximum statistic, and minimum statistic are extracted from the responses of different frequency branches, and the average is aggregated across all frequency branches: ; ; ; in, Indicates the first The first module Multi-frequency average response statistics for each scale branch; This represents the multi-frequency maximum response statistic for this scale branch; This represents the multi-frequency minimum response statistic for this scale branch; This indicates a global average pooling operation; This represents the global max pooling operation; This represents the global minimum pooling operation; This indicates the number of discrete cosine frequency branches.

[0097] Channel frequencies are generated based on the above three types of statistics, paying attention to the weights: ; in, Indicates the first The first module Pay attention to the weighting of the channel frequencies in each scale branch; The shared channel mapping function is preferably represented by two layers. Point-to-point mapping constitutes; This refers to the Sigmoid function, used to constrain weights to... to Within the range.

[0098] Reweighting intermediate context features using channel frequency attention weights: ; in, This represents the scale characteristics after multi-frequency channel reweighting; Indicates the first Intermediate context features of each scale branch; The corresponding channel frequency is indicated by its weight.

[0099] To further enhance spatial adaptability, a spatially significant weight map is generated for each scale branch: ; in, Indicates the first The first module Spatial salient weight map of each scale branch; This indicates the data used to generate the spatial weighted graph. Point-to-mapping operator; This represents the Sigmoid function.

[0100] By combining the background response modifier and the significant region enhancement coefficient, differential enhancement features are constructed: ; in, Indicates the first The first module Features enhanced by scale-based branches; This represents the background response adjustment coefficient corresponding to this scale branch; This represents the salient region enhancement coefficient corresponding to this scale branch; The weight complement represents the spatially insignificant region; This represents a weighted map of spatially salient regions. The formula achieves spatial differentiation enhancement of deep features by applying different learnable adjustment coefficients to salient and insignificant regions.

[0101] Subsequently, the features enhanced by this scale branch are restored to the original number of channels in the bottleneck layer and upsampled to the input space size: ; in, Indicates the first The first module Enhanced features after restoring each scale branch to a uniform size; This indicates the number of channels used for recovery and integration of local responses. Convolution mapping; This indicates that the features are upsampled to... Size manipulation.

[0102] Finally, the enhanced features from all scale branches are averaged and aggregated, and then residually superimposed with the module input features to obtain the th... The output of the multi-frequency, multi-scale attention enhancement module: ; in, Indicates the first The output characteristics of a multi-frequency, multi-scale attention enhancement module; This indicates the input characteristics of the module; Indicates the number of scale branches; Indicates the first Enhanced features after restoring each scale branch to a uniform size; Indicates the scale branch index.

[0103] When the bottleneck layer is connected in series When multiple frequency and multi-scale attention enhancement modules are used, their overall output can be expressed as: ; in, This represents the initial deep features entering the bottleneck layer; Indicates Deep enhancement features after multi-frequency, multi-scale attention enhancement modules are connected in series; Indicates the first Each multi-frequency, multi-scale attention enhancement module corresponds to a feature enhancement mapping; This indicates the composition of functions; This indicates the number of multi-frequency, multi-scale attention enhancement modules in the bottleneck layer, with the preferred selection being... .

[0104] To verify the practical effect of the method of the present invention in medical image fusion, MRI-PET medical image fusion was selected as the experimental object, and ten existing representative methods, including CDDFuse, TarD, FATFSion, DeFusion, RFNet, GeSeNet, FusionMamba, SwinFusion, U2Fusion, and EMFusion, were used as control methods. The comparative analysis was carried out from two aspects: subjective visual effect and objective evaluation index.

[0105] For quantitative evaluation, EN, SD, SF, MI, PSNR, SSIM, VIF, and Q were selected. AB / F The fusion results are evaluated using the following metrics: EN (Enhanced Information Representation), SD (Solution of Gray-Level Distribution and Contrast Variation), SF (Spatial Detail and Texture Activity), MI (Meaning of Information Retained from Source Images), PSNR (Power Shift Reduction), SSIM (Structure Preservation), VIF (Visual Information Fidelity), and Q (Quantization). AB / F Used to evaluate the preservation of edge and gradient information.

[0106] Table 1. Quantitative comparison results of MRI-PET fusion

[0107] The experimental results show that the method of this invention achieved good results in multiple evaluation indicators. Among them, the MI index reached 2.09, the highest value among all comparison methods, indicating that the fused image can more fully absorb the anatomical structural information from MRI images and the functional metabolic information from PET images, giving the fusion result more complete multimodal complementary features. Compared with simple pixel-level superposition, the method of this invention selectively fuses effective information from different modalities, making the fused image more advantageous in terms of information integrity.

[0108] In terms of image fidelity, the PSNR of the method of this invention reaches 32.82, which is higher than other comparative methods. A higher PSNR usually indicates less distortion, weaker noise interference, and more stable overall gray-level distribution in the fused image. Therefore, the fused image generated by the method of this invention can reduce the impact of local artifacts, gray-level abrupt changes, and invalid noise while maintaining the main information of the source image.

[0109] In terms of structural preservation, the SSIM index of the method of this invention reaches 1.52, which is the best value among all methods. This value corresponds to strong structural consistency and spatial continuity, and the tissue contours, edge structures, and local texture distribution in the fused image are clearer. For MRI-PET medical image fusion, structural information is directly related to the identification of tissue boundaries and lesion areas, so improving this index has strong practical significance.

[0110] In terms of visual information preservation, the method of this invention achieves a VIF index of 0.84, which is superior to other comparative methods. A higher VIF value indicates that the fused image retains more effective visual information that meets the needs of human visual observation and medical interpretation, which helps to highlight key tissue areas and functional response areas, while reducing the interference of background redundancy information on observation and judgment.

[0111] Regarding edge detail preservation, the Q of the method of the present invention AB / F The index reached 0.71, achieving the best performance among similar methods. This index corresponds to the degree of preservation of edge gradient information in the source image in the fusion result; the higher the value, the clearer the tissue boundaries, texture details, and local structures. Edge and gradient information in medical images are usually closely related to lesion boundaries, tissue contours, and structural changes; therefore, this index reflects the advantage of the method of this invention in terms of detail preservation.

[0112] As can be seen from the various indicators in the table, the method of this invention achieves good results in MI, PSNR, SSIM, VIF, and Q. AB / F The method achieves leading performance in key indicators such as EN, SD, and SF, while also maintaining good overall performance. This indicates that the method of the present invention can further improve the fidelity, structural consistency, and visual recognizability of the fused image while ensuring the amount of image information and spatial details. The generated fused image has a more balanced overall brightness distribution, clearer anatomical structure boundaries, more complete preservation of texture details, and better presentation of significant functional areas in the PET image.

[0113] In summary, the method of this invention can effectively improve the fusion quality of MRI-PET medical images, demonstrating good performance in preserving multimodal information, structural continuity, edge detail representation, noise suppression, and visual naturalness. This method can provide clearer, more stable, and more information-rich fused images for subsequent medical image observation, lesion localization, assisted diagnosis, and clinical analysis, and has significant practical application value.

[0114] This invention also provides a multimodal image fusion system based on dynamic frequency domain fusion, the system comprising: The image stitching unit is used to acquire a first modality medical image and a second modality medical image, and stitch the two images together in the channel dimension to form a dual-channel input tensor; A step-by-step encoding unit is used to input the dual-channel input tensor into the encoder path. The encoder includes a multi-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The basic convolutional feature extraction module is used to extract local structural and texture information at the current scale; the split multi-scale feature module is used to perform multi-branch partitioning and adaptive fusion of the input features, and output multi-scale enhanced features; the dynamic frequency domain selection fusion module is used to receive two features from the basic convolutional feature extraction module and the split multi-scale feature module, adaptively generate inter-modal fusion weights based on the content of the input features, and selectively fuse the two features through dynamic frequency domain filtering to obtain the fused features at the current scale. An enhancement unit is used to input the final output of the encoder path into a bottleneck layer. The bottleneck layer consists of at least one multi-frequency multi-scale attention enhancement module, which is used to enhance the key spectral components and semantic representation capabilities in deep features to obtain deep enhanced feature representations. The decoding unit is used to input the deep enhanced feature representation into the decoder path. The decoder includes a multi-level layer-by-layer upsampling reconstruction structure. In each level of decoding, the current low-resolution feature is upsampled and then fused with the encoder output of the corresponding scale through a skip connection. After convolutional reconstruction, the spatial resolution and texture details are restored step by step to obtain a high-resolution reconstructed feature map. The fusion unit is used to map the high-resolution reconstructed feature map into a single-channel image to obtain the final fused image.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal image fusion method based on dynamic frequency domain fusion, characterized in that, The method includes: Acquire the first modality medical image and the second modality medical image, and stitch the two images together in the channel dimension to form a dual-channel input tensor; The dual-channel input tensor is input into the encoder path. The encoder includes a multi-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The final output of the encoder path is input into the bottleneck layer, which consists of at least one multi-frequency multi-scale attention enhancement module, used to enhance the key spectral components and semantic representation capabilities in the deep features, to obtain the deep enhanced feature representation. The deep enhancement feature representation is input into the decoder path. The decoder includes a multi-level layer-by-layer upsampling reconstruction structure. In each level of decoding, the current low-resolution feature is upsampled and then fused with the encoder output of the corresponding scale via a skip connection. After convolutional reconstruction, the spatial resolution and texture details are restored step by step to obtain a high-resolution reconstructed feature map. The high-resolution reconstructed feature map is mapped to a single-channel image to obtain the final fused image; The basic convolutional feature extraction module is used to extract local structural and texture information at the current scale; the split multi-scale feature module is used to perform multi-branch partitioning and adaptive fusion of the input features, and output multi-scale enhanced features; the dynamic frequency domain selection fusion module is used to receive two features from the basic convolutional feature extraction module and the split multi-scale feature module, adaptively generate inter-modal fusion weights based on the content of the input features, and selectively fuse the two features through dynamic frequency domain filtering to obtain the fused features at the current scale. The dynamic frequency domain selection fusion module performs the following operations: The first and second features are received and preprocessed using lightweight convolutions respectively. The preprocessed two-way features are summed to generate conditional features, and the conditional features are mapped to adaptive weights corresponding to the two-way features respectively. For any feature, input it into the dynamic frequency domain filtering submodule to obtain the dynamic frequency domain filtering result of that feature; The dynamic frequency domain filtering results of the two features are added together to obtain the fused features at the current scale; The step of inputting any feature into the dynamic frequency domain filtering submodule to obtain the dynamic frequency domain filtering result of the feature includes: mapping the spatial domain feature to the frequency domain via Fourier transform; Filter combination coefficients are generated based on the aforementioned conditional features, and the coefficients are used to perform weighted combination on a pre-set complex-valued basic filter bank to obtain a sample-adaptive and channel-adaptive frequency domain kernel. Perform element-wise multiplication between the frequency domain kernel and the spectral features; The result is then restored to the spatial domain by inverse Fourier transform and added to the residual term to obtain the dynamic frequency domain filtering result of this path feature. The dynamic frequency domain filtering submodule further includes: performing reflection filling before performing Fourier transform on the input features; After the inverse Fourier transform, pruning is performed to restore the original size; the complex-valued fundamental filter bank includes real and imaginary parts, which are adapted to the frequency domain size of the current scale after interpolation transformation.

2. The multimodal image fusion method based on dynamic frequency domain fusion according to claim 1, characterized in that, The split-scale feature module performs the following operations: The input features are divided into local branches, context branches, and lightweight branches along the channel dimension; The local branch uses a combination of depthwise convolution and pointwise convolution to extract edge and texture details; the context branch uses large kernel decomposition convolution and dilated convolution to extract structural information under a large receptive field; and the lightweight branch uses pointwise convolution to maintain the basic intensity response. Global average pooling is performed on the original input to generate three-way gated weights. Based on the three-way gated weights, the output features of the three branches are concatenated by channels, and then the fusion result is obtained by channel shuffling and pointwise convolution. The fusion result is added to the shortcut mapping of the input features through residual connections and then output through an activation function.

3. The multimodal image fusion method based on dynamic frequency domain fusion according to claim 1, characterized in that, Each level of coding structure also includes a spatial-frequency joint enhancement module. The spatial-frequency joint enhancement module is used to receive the fusion feature output by the dynamic frequency domain selection fusion module, and enhance the feature from the spatial domain and the frequency domain respectively. Then, the spatial enhancement result is fused with the frequency enhancement result, and residual superimposed with the input feature to output the enhanced skip connection feature.

4. The multimodal image fusion method based on dynamic frequency domain fusion according to claim 3, characterized in that, The space-frequency joint enhancement module performs the following operations: Perform layer normalization on the input features, and then input the normalized features into both the spatial enhancement branch and the frequency enhancement branch. The spatial augmentation branch includes: extracting local spatial context through convolution to obtain intermediate features; dividing the intermediate features into a first subset and a second subset along the channel dimension; applying global average pooling to the first subset to generate a first gate weight; applying global max pooling to the second subset to generate a second gate weight; multiplying the two subsets by their corresponding gate weights and concatenating them; and adding them to the local spatial context to obtain the spatial augmentation result. The frequency enhancement branch includes: performing a Fourier transform on the normalized features to obtain a complex spectrum, decomposing the complex spectrum into an amplitude spectrum and a phase spectrum, modulating the amplitude spectrum and phase spectrum respectively, reconstructing the complex spectrum and restoring it to the spatial domain through an inverse Fourier transform, and then adding it to the normalized feature residual to obtain the frequency enhancement result. The spatial augmentation result is added to the frequency augmentation result, and then superimposed with the input feature residual to output the joint augmentation result.

5. The multimodal image fusion method based on dynamic frequency domain fusion according to claim 1, characterized in that, The multi-frequency, multi-scale attention enhancement module performs the following operations: Input features are processed in parallel at multiple scales. For each scale, average pooling is performed first, and then multi-scale context is extracted by dilated depthwise convolution and pointwise convolution to obtain the intermediate features of that scale. Construct multiple fixed discrete cosine filters based on predefined discrete cosine transform frequency coordinates; The intermediate features are adaptively pooled to the target discrete cosine size, multiplied pointwise with each discrete cosine filter, and the average, maximum and minimum statistics are extracted. The statistics of multiple frequency bands are then averaged and channel attention weights are generated through shared pointwise convolution mapping. Spatial attention maps are generated for each scale, and intermediate features at that scale are differentially enhanced by combining background suppression coefficients and salient region enhancement coefficients. The enhancement results at each scale are upsampled to the original spatial scale and averaged, then superimposed with the input feature residuals to output multi-frequency, multi-scale enhanced features.

6. A multimodal image fusion system based on dynamic frequency domain fusion, used to implement the multimodal image fusion method based on dynamic frequency domain fusion as described in any one of claims 1-5, characterized in that, The system includes: The image stitching unit is used to acquire a first modality medical image and a second modality medical image, and stitch the two images together in the channel dimension to form a dual-channel input tensor; A step-by-step encoding unit is used to input the dual-channel input tensor into the encoder path. The encoder includes a multi-level downsampling feature extraction structure. Each level of the encoding structure includes a basic convolutional feature extraction module, a split multi-scale feature module, and a dynamic frequency domain selection fusion module. The basic convolutional feature extraction module is used to extract local structural and texture information at the current scale; the split multi-scale feature module is used to perform multi-branch partitioning and adaptive fusion of the input features, and output multi-scale enhanced features; the dynamic frequency domain selection fusion module is used to receive two features from the basic convolutional feature extraction module and the split multi-scale feature module, adaptively generate inter-modal fusion weights based on the content of the input features, and selectively fuse the two features through dynamic frequency domain filtering to obtain the fused features at the current scale. An enhancement unit is used to input the final output of the encoder path into a bottleneck layer. The bottleneck layer consists of at least one multi-frequency multi-scale attention enhancement module, which is used to enhance the key spectral components and semantic representation capabilities in deep features to obtain deep enhanced feature representations. The decoding unit is used to input the deep enhanced feature representation into the decoder path. The decoder includes a multi-level layer-by-layer upsampling reconstruction structure. In each level of decoding, the current low-resolution feature is upsampled and then fused with the encoder output of the corresponding scale through a skip connection. After convolutional reconstruction, the spatial resolution and texture details are restored step by step to obtain a high-resolution reconstructed feature map. The fusion unit is used to map the high-resolution reconstructed feature map into a single-channel image to obtain the final fused image.

Images