PET (Positron Emission Tomography) and MRI (Magnetic Resonance Imaging) multi-modal medical image fusion method and system based on Kolmogorov-Arnold network

Abstract

The invention relates to a PET (Positron Emission Tomography) and MRI (Magnetic Resonance Imaging) multi-modal medical image fusion method and system based on a Kolmogorov-Arnold network. The method comprises the steps of collecting PET and MRI images for normalization processing, and extracting an initial feature map, a first scale feature map, a second scale feature map, a third scale feature map and a fourth scale feature map through a hierarchical feature extraction network of a multi-scale dynamic convolution kernel architecture; inputting the first-scale feature map, the second-scale feature map and the third-scale feature map into a KAN for pyramid coding, decomposition and reconstruction, multi-scale decomposition and other operations to obtain a decoded feature map; and inputting the fourth scale feature map into a frequency domain-spatial domain collaborative fusion framework based on dynamic sparse attention guidance for feature integration, performing frequency domain expansion based on reversible frequency domain up-sampling, and performing channel weighted splicing, spatial-channel decoupling processing and interactive fusion processing to obtain a final fused medical image. The image details can be obtained, noise can be effectively suppressed, and the accuracy and reliability of the fused image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a PET and MRI multimodal medical image fusion method and system based on a Kolmogorov-Arnold network. Background Art

[0002] In the field of medical imaging, magnetic resonance imaging (MRI) is an anatomical image with high spatial texture information, while positron emission tomography (PET) is a functional imaging technique that provides information on human physiological metabolism, but its spatial resolution is relatively low. In order to better leverage the advantages of different medical images, a medical imaging technique has emerged that fuses two different medical images obtained by two different imaging technologies. Currently, multimodal medical image fusion technology has important application value in clinical scenarios such as precise diagnosis, tumor localization, and surgical navigation. Among them, image fusion, as a key technology for the collaborative analysis of functional metabolic information and high-resolution anatomical structures, faces core challenges such as significant differences in feature distribution between modalities, insufficient retention of high-frequency details, and difficulty in ensuring semantic consistency across scales.

[0003] Traditional medical image fusion often directly fuses the entire images corresponding to two different medical images. This can be achieved through deep convolutional networks, multi-scale fusion architectures, and frequency domain fusion methods. However, existing mainstream methods have three main limitations: First, the feature extraction module based on deep convolutional networks is limited by the expressive power of fixed kernel functions, making it difficult to adaptively balance the coupling strength of metabolic features and texture details, resulting in loss of high-frequency information or noise amplification; second, traditional multi-scale fusion architectures generally adopt a cascaded downsampling strategy, which causes gradient diffusion during the cross-modal feature alignment process and cannot achieve accurate matching of fine-grained anatomical boundaries; third, existing frequency domain fusion methods mostly rely on static filters and lack a mechanism for dynamically enhancing metabolically active areas, resulting in spatial misalignment of functional information and anatomical structures.

[0004] Therefore, traditional medical image fusion methods often have problems such as poor image fusion effect and poor accuracy and reliability of the fused images due to limited image feature extraction and mismatched image boundaries. Summary of the Invention

[0005] Based on this, in order to solve the above technical problems, a PET and MRI multimodal medical image fusion method and system based on the Kolmogorov-Arnold network is provided, which can preserve image details, suppress cross-modal noise interference, and improve the accuracy and reliability of the fused image.

[0006] A PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network, the method comprising:

[0007] PET and MRI medical images are collected and normalized, and then input into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate first-scale feature maps, second-scale feature maps, third-scale feature maps, and fourth-scale feature maps from the initial feature maps;

[0008] Inputting the first-scale feature map, the second-scale feature map, and the third-scale feature map into the Kolmogorov-Arnold network for pyramid encoding, and implementing cross-modal feature fusion of each feature map through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map;

[0009] A bidirectional phase interference mechanism is introduced to decompose and reconstruct the first fused feature map, and a reversible decomposition network with a mirror-symmetrical structure is designed to perform multi-scale decomposition to obtain a decoded feature map;

[0010] Inputting the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework guided by dynamic sparse attention, achieving feature fusion based on a dual-path cross-attention mechanism, and obtaining a second fused feature map; performing feature integration on the second fused feature map using reversible instance normalization to obtain an integrated feature map;

[0011] The integrated feature map is subjected to frequency domain expansion based on reversible frequency domain upsampling, and is subjected to channel weighted splicing, space-channel decoupling processing and interactive fusion processing to obtain a final fused medical image.

[0012] In one embodiment, PET and MRI medical images are collected and normalized, and then input into a multi-scale dynamic convolution kernel architecture, including:

[0013] Acquire PET and MRI medical images and perform modality-specific preprocessing before inputting them into the normalization layer of their respective modalities;

[0014] The preprocessed PET medical image is subjected to logarithmic normalization by the normalization layer, and the preprocessed MRI medical image is subjected to Z-score normalization;

[0015] Input the normalized PET and MRI medical images into the multi-scale dynamic convolution kernel architecture;

[0016] Among them, the KAN convolution kernel in the multi-scale dynamic convolution kernel architecture follows the Kolmogorov-Arnold theorem, decomposing the multivariable function into a combination of univariate basis functions, which is used to realize the adaptive extraction of multi-scale features through the inter-modal parameter sharing and exclusive mechanism.

[0017] In one embodiment, the multi-scale dynamic convolution kernel architecture extracts initial feature maps respectively through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate a first-scale feature map, a second-scale feature map, a third-scale feature map, and a fourth-scale feature map according to the initial feature map, including:

[0018] The hierarchical feature extraction network uses sine basis functions and B-spline basis functions to implement joint modeling in the space-channel domain; the spatial basis function and the channel basis function are determined, and the single-layer KAN convolution kernel in the hierarchical feature extraction network calculates and generates an initial feature map according to the spatial basis function and the channel basis function;

[0019] Based on the initial feature map, a first-scale feature map is generated using dynamic KAN convolution and learnable downsampling;

[0020] Based on the first-scale feature map, a second-scale feature map is generated using depth-separable KAN convolution and cross-modal interaction;

[0021] Based on the second-scale feature map, a third-scale feature map is generated using a residual KAN block and frequency domain constraints;

[0022] Based on the third-scale feature map, a fourth-scale feature map is generated by adopting cross-modal entanglement and gradient coupling compression.

[0023] In one embodiment, inputting the first scale feature map, the second scale feature map, and the third scale feature map into a Kolmogorov-Arnold network for performing a pyramid encoding operation includes:

[0024] Inputting the first scale feature map, the second scale feature map, and the third scale feature map into a Kolmogorov-Arnold network, and determining a scale level corresponding to each scale feature map;

[0025] Obtain the univariate basis functions corresponding to each scale feature map and determine the superposition operator of the Kolmogorov-Arnold network;

[0026] A KAN pyramid encoding operation is performed on each scale feature map according to the scale level, the single variable basis function, and the superposition operator.

[0027] In one embodiment, cross-modal feature fusion of each feature map is achieved through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map, including:

[0028] The dynamic weight sharing mechanism is used to enforce basis function parameter sharing and achieve cross-modal pyramid alignment.

[0029] Through the asymmetric attention mechanism, the features in the feature maps of each scale are guided and complemented with each other, and the parameters are dynamically balanced in the bidirectional KAN attention fusion to achieve bidirectional information flow adaptive adjustment, complete cross-modal feature fusion, and obtain the first fused feature map.

[0030] In one embodiment, a bidirectional phase interference mechanism is introduced to decompose and reconstruct the first fused feature map, and a reversible decomposition network with a mirror-symmetrical structure is designed to perform multi-scale decomposition to obtain a decoded feature map, including:

[0031] Introducing a bidirectional phase interference mechanism, mapping the real features in the first fusion feature map to the complex domain for interference operation, and reconstructing the real features by decomposing the real and imaginary parts;

[0032] Designing a reversible decomposition network with a mirror-symmetric structure, performing multi-scale edge extraction on the first fused feature map to obtain a scale-sensitive mask;

[0033] Based on the scale-sensitive mask, the first fused feature map is decomposed into multiple scales through an inverse KAN to obtain a decoded feature map.

[0034] In one embodiment, the fourth-scale feature map is input into a frequency-domain-spatial-domain collaborative fusion framework based on dynamic sparse attention guidance, and feature fusion is achieved based on a dual-path cross-attention mechanism to obtain a second fused feature map; reversible instance normalization is used to perform feature integration on the second fused feature map to obtain an integrated feature map, including:

[0035] Inputting the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework based on dynamic sparse attention guidance, performing frequency band decomposition through improved discrete cosine transform to generate frequency band features;

[0036] Constructing a dual-path cross-attention mechanism, calculating a fusion feature according to the dual-path cross-attention mechanism, and performing adaptive feature sharpening on the frequency band feature through a Laplace-Gaussian operator based on the fusion feature;

[0037] Improved reversible instance normalization is used to integrate the sharpened features and fused features to obtain the second fused feature map.

[0038] In one embodiment, the integrated feature map is subjected to frequency domain expansion based on reversible frequency domain upsampling, and channel weighted splicing, space-channel decoupling processing, and interactive fusion processing to obtain a final fused medical image, including:

[0039] Performing frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling to obtain an upsampling result;

[0040] Performing channel-weighted splicing on the upsampling result and the third feature map output by the pyramid to obtain a spliced ​​feature, and performing spatial-channel decoupling processing on the spliced ​​feature using a depthwise separable KAN convolution to obtain a decoupled feature map;

[0041] A cross-modal residual compensation mechanism is introduced to interactively fuse the decoupled feature map with the second feature map output by the pyramid to obtain a fused preliminary feature map;

[0042] Performing an upsampling operation on the fused preliminary feature map, and constructing a multi-granularity feature expression using dynamic pyramid pooling to obtain an expression feature map;

[0043] A phase synthesis method inspired by quantum entanglement is introduced to synthesize the expression feature map and the first feature map output by the pyramid to obtain a final feature map;

[0044] The multimodal features of the final feature map are decoupled and projected, and the mapping of features to pixel space is achieved through a dynamic parameterized kernel. The anatomical structure in the feature image is activated by the complex domain to align the frequency domain of metabolic information to obtain the final fused medical image.

[0045] In one embodiment, the method further comprises:

[0046] Set content loss, frequency domain energy loss, structural similarity loss, and gradient alignment loss respectively;

[0047] Determine a dynamic weighting coefficient, and calculate a composite loss function based on each set loss and the dynamic weighting coefficient;

[0048] The composite loss function is used to enhance feature fusion.

[0049] A PET and MRI multimodal medical image fusion system based on a Kolmogorov-Arnold network, the system comprising:

[0050] A multi-scale dynamic feature extraction module is used to collect and normalize PET and MRI medical images, and input them into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate first-scale feature maps, second-scale feature maps, third-scale feature maps, and fourth-scale feature maps from the initial feature maps.

[0051] A cross-modal pyramid interaction module is used to input the first-scale feature map, the second-scale feature map, and the third-scale feature map into the Kolmogorov-Arnold network for pyramid encoding, and to achieve cross-modal feature fusion of each feature map through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map;

[0052] A reversible pyramid decoding and reconstruction module is used to introduce a bidirectional phase interference mechanism to decompose and reconstruct the first fused feature map, and design a reversible decomposition network with a mirror-symmetrical structure to perform multi-scale decomposition to obtain a decoded feature map;

[0053] a frequency-domain-spatial-domain collaborative fusion module, configured to input the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework guided by dynamic sparse attention, implement feature fusion based on a dual-path cross-attention mechanism, and obtain a second fused feature map; and perform feature integration on the second fused feature map using reversible instance normalization to obtain an integrated feature map;

[0054] The feature map fusion module is used to perform frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling, and perform channel weighted splicing, space-channel decoupling processing and interactive fusion processing to obtain the final fused medical image.

[0055] The above-mentioned PET and MRI multimodal medical image fusion method and system based on the Kolmogorov-Arnold network uses a hierarchical feature extraction network in a multi-scale dynamic convolution kernel structure to extract feature maps of different scales respectively, and then performs pyramid encoding and frequency domain-spatial domain collaborative fusion framework for processing, which can retain PET metabolic functional information and MRI anatomical structure details; through the combination of dynamic weight sharing mechanism, asymmetric attention mechanism, bidirectional phase interference mechanism, reversible decomposition network of mirror-symmetric structure, and dual-path cross-attention mechanism, it can effectively suppress cross-modal noise interference and improve the accuracy and reliability of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 FIG1 is a diagram showing an application environment of a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network in one embodiment;

[0057] Figure 2 1 is a flow chart of a method for fusion of PET and MRI multimodal medical images based on a Kolmogorov-Arnold network in one embodiment;

[0058] Figure 3is a structural block diagram of a PET and MRI multimodal medical image fusion system based on a Kolmogorov-Arnold network in one embodiment;

[0059] Figure 4 is a structural block diagram of a PET and MRI multimodal medical image fusion system based on a Kolmogorov-Arnold network in another embodiment;

[0060] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] It is understood that the terms "first," "second," and the like used herein may be used to describe scale characteristic maps, but these scale characteristic maps are not limited by these terms. These terms are merely used to distinguish a first scale characteristic map from another scale characteristic map. For example, without departing from the scope of this application, a first scale characteristic map may be referred to as a second scale characteristic map, and similarly, a second scale characteristic map may be referred to as a first scale characteristic map. Both the first scale characteristic map and the second scale characteristic map are scale characteristic maps, but they are not the same scale characteristic map.

[0063] The PET and MRI multimodal medical image fusion method based on the Kolmogorov-Arnold network provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1As shown, the application environment includes a computer device 110. The computer device 110 can collect PET and MRI medical images and perform normalization processing, and input them into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps respectively through a hierarchical feature extraction network, and uses a multi-scale pyramid to generate a first-scale feature map, a second-scale feature map, a third-scale feature map, and a fourth-scale feature map in sequence according to the initial feature map; the computer device 110 can input the first-scale feature map, the second-scale feature map, and the third-scale feature map into a Kolmogorov-Arnold network for pyramid encoding operation, and realize cross-modal feature fusion of each feature map through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map; the computer device 110 can introduce a bidirectional phase interference mechanism to decompose and reconstruct the first fused feature map, and design a reversible decomposition network with a mirror-symmetric structure to perform multi-scale decomposition to obtain a decoded feature map; the computer device 110 can input the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework guided by dynamic sparse attention, and achieve feature fusion based on a dual-path cross-attention mechanism to obtain a second fused feature map; reversible instance normalization is used to integrate the features of the second fused feature map to obtain an integrated feature map; the computer device 110 can perform frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling, and perform channel weighted splicing, spatial-channel decoupling processing, and interactive fusion processing to obtain a final fused medical image. The computer device 110 can be, but is not limited to, various personal computers, laptops, smart phones, robots, tablet computers, and other devices.

[0064] In one embodiment, Figure 2 As shown, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network is provided, comprising the following steps:

[0065] In step 202, PET and MRI medical images are collected and normalized, and then input into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to generate a first-scale feature map, a second-scale feature map, a third-scale feature map, and a fourth-scale feature map in sequence according to the initial feature map.

[0066] In the process of medical image fusion, it is often necessary to capture PET metabolic specificity and MRI texture details while maintaining anatomical structural commonality. In this embodiment, scale feature extraction can be performed through a hierarchical feature extraction network in a multi-scale dynamic convolution kernel architecture.

[0067] Among them, the hierarchical feature extraction network has a dual-branch structure, and the left branch is used to process PET images. OPET∈ R H ×W×3 ，右分支用于处理MRI图像 I OMRI ∈ R H×W×3 ，且左右分支结构相 same.

[0068] Specifically, the computer device can construct a hierarchical dynamically parameterized KAN convolution kernel with a unified architecture at four scales, and achieve adaptive extraction of multi-scale features through inter-modal parameter sharing and exclusive mechanisms. The convolution kernel parameters follow the Kolmogorov-Arnold theorem, decomposing the multivariate function into a combination of univariate basis functions. For the lth layer (l∈{1,2,3}), the formula for defining the convolution kernel for the modality mod∈{PET,MRI} is expressed as: in, represents the spatial domain basis function, Represents channel domain basis function; spatial domain basis function The formula is expressed as: Among them, the spatial domain basis function is a function based on SIREN (Sinusoidal Representation Network). is the amplitude coefficient, used to control the feature intensity, initialized to and It is a spatial frequency parameter used to adjust the granularity of capturing local details. exponential decay, Represents phase offset, used to enhance position sensitivity, learned through back propagation; channel domain basis function The formula is expressed as: in, Represents the learnable coefficient matrix (dimension), initialized using the He normal distribution, represents the third-order B-spline basis function, and the number of nodes M = 8, which is used to ensure smooth transition of cross-channel features; the dynamic gating function formula is expressed as: σ(z) = Sigmoid(KAN gate (z)); where KAN gate Represents a lightweight network, consisting of 2 layers of KAN, 4 basis functions in each layer, and the input is the mean and variance of the local feature block; the number of basis functions is K l =32×2 -(l-1) , it decays exponentially with the increase of scale, K1=32, K2=16, K3=8, K4=4, realizing the transition from fine-grained to coarse-grained features.

[0069] In the parameter sharing mechanism, for high frequency sharing, the first K l / 2 basis functions are shared across modalities, i.e. Used to capture common features of anatomical structures; for low-frequency independence, post-K l / 2 basis function modes are exclusive, that is, Used to preserve metabolic and functional heterotrophic properties.

[0070] In one embodiment, a Kolmogorov-Arnold network-based PET and MRI multimodal medical image fusion method is provided that may further include an image normalization process, specifically comprising: acquiring PET and MRI medical images and performing modality-specific preprocessing, and inputting them into the normalization layers of their respective modalities; performing logarithmic normalization on the preprocessed PET medical images and performing Z-score normalization on the preprocessed MRI medical images through the normalization layers; and inputting the normalized PET and MRI medical images into a multi-scale dynamic convolution kernel architecture; wherein the KAN convolution kernel in the multi-scale dynamic convolution kernel architecture follows the Kolmogorov-Arnold theorem, decomposing a multivariate function into a combination of univariate basis functions, and realizing adaptive extraction of multi-scale features through an inter-modality parameter sharing and exclusive mechanism.

[0071] Computer equipment can perform modality-specific preprocessing on PET and MRI images, mapping pixel values ​​to the range [-1, 1] through the normalization layer of each modality. PET uses logarithmic normalization, which is expressed as: MRI uses Z-score normalization, and its formula is expressed as: Among them, μ and σ are training set statistics, and ∈=1e-5 prevents numerical overflow.

[0072] In one embodiment, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network is provided, which may also include a feature extraction process, the specific process including: a hierarchical feature extraction network uses a sine basis function and a B-spline basis function to realize space-channel domain joint modeling; determining the spatial basis function and the channel basis function, and a single-layer KAN convolution kernel in the hierarchical feature extraction network calculates and generates an initial feature map according to the spatial basis function and the channel basis function; based on the initial feature map, a first-scale feature map is generated by using dynamic KAN convolution and learnable downsampling; based on the first-scale feature map, a second-scale feature map is generated by using depthwise separable KAN convolution and cross-modal interaction; based on the second-scale feature map, a third-scale feature map is generated by using a residual KAN block and frequency domain constraints; based on the third-scale feature map, a fourth-scale feature map is generated by using cross-modal entanglement and gradient coupling compression.

[0073] In this embodiment, the proposed multi-scale dynamic convolution kernel architecture constructs a hierarchical and interpretable feature extraction network through the high-frequency parameter sharing and low-frequency mode exclusive mechanism, and uses sine basis functions and spline basis functions to realize space-channel domain joint modeling.

[0074] Specifically, the input image, i.e., the normalized PET and MRI medical images, is convolutionally processed using a single-layer KAN convolution kernel to generate the initial feature map. Its formula is expressed as: where mod∈{PET,MRI}, As the spatial basis function, high frequency priority initialization is adopted, and the setting and To capture local details, is the channel basis function, using third-order B-spline interpolation, the number of nodes M = 4, σ(z) represents dynamic gating, and the input is the root mean square value of a 3×3 neighborhood. In this embodiment, the high-frequency basis function parameters are forced to be shared in the initial layer, that is, Where k≤K0 / 2.

[0075] Next, the computer device can use the multi-scale pyramid to generate the first scale feature map, the second scale feature map, the third scale feature map, and the fourth scale feature map, that is, feature maps of four different scales: scale 1, scale 2, scale 3, and scale 4. The scale 1 feature map is generated using dynamic KAN convolution and learnable downsampling. The dynamic KAN convolution formula is expressed as: stride=2,K1=32; among them, the high-frequency shared basis functions of PET and MRI feature maps account for 50%, through Parameter freezing achieves cross-modal alignment; the learnable downsampling formula is expressed as: kernel=4×4, stride=2; the dynamic pooling kernel weight is generated by local features through KAN regression (input 4×4 neighborhood, output 1 weight value); scale 2 is generated using depth-wise separable KAN convolution and cross-modal interaction. The depth-wise separable KAN convolution formula is expressed as: K2=16; where separation is spatial convolution (3×3 kernel) and channel convolution (1×1 core); the cross-modal interaction formula is: Among them, CrossGate represents the generation of attention mask based on bimodal feature similarity, and its expression is: Among them, KAN simIt is a KAN network used to calculate the similarity of bimodal features. Through the attention mask, the network can adaptively assign weights according to the similarity of PET and MRI features to enhance the fusion of cross-modal features. Scale 3 is generated using the residual KAN block and frequency domain constraints. The residual KAN block formula is: ResKAN represents a three-layer cascaded KAN convolution, where the number of channels in each layer is compressed to 1 / 2. Through residual connections, the model can alleviate the gradient vanishing problem. The skip connection projection layer Proj is a 1×1 KAN convolution with the number of basis functions K=8 to ensure the dimensionality matching of the residual connection. The frequency domain constraint expression is: Among them, the formula shows that first the feature map Perform discrete cosine transform (DCT) to convert it from the spatial domain to the frequency domain, and then KAN filter represents the frequency domain filter, the learning coefficient β l =0.3, which is used to suppress high-frequency noise. Finally, the processed frequency domain features are converted back to the spatial domain through inverse discrete cosine transform (IDCT). Scale 4 is generated using cross-modal entanglement and gradient coupling compression. The cross-modal entanglement formula is expressed as: Among them, Attn mod represents the bidirectional KAN attention module, which is used to calculate the inter-modality correlation matrix S, and The gradient coupling compression formula is expressed as: in, Represents an adaptive pooling operation, which is used to pool the output of scale 3. Following the gradient update rule, the knowledge transfer between modalities is forced, and its expression is: Among them, SIREN is implemented as a custom layer and B-spline is accelerated by sparse matrix.

[0076] In step 204, the first-scale feature map, the second-scale feature map, and the third-scale feature map are input into a Kolmogorov-Arnold network for pyramid encoding, and cross-modal feature fusion of each feature map is achieved through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map.

[0077] Since feature maps of different scales contain semantic information at different levels, large-scale feature maps retain more detailed information, while small-scale feature maps contain more abstract semantic information. The feature maps output by the first scale block, second scale block and third scale block of PET and MRI, namely the first scale feature map, second scale feature map and third scale feature map, are Kolmogorov-Arnold Networks (KAN) used to perform KAN pyramid encoding on the feature maps of PET and MRI images at different scales to mine the nonlinear information in the feature maps and enhance the feature expression ability.

[0078] A bidirectional parameter-constrained pyramid can be designed in computer equipment to screen shared basis functions through cross-scale cosine similarity and enhance the spatial consistency of multimodal features by combining the complex domain quantum entanglement algorithm.

[0079] In one embodiment, a provided PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network may further include a pyramid coding operation process, the specific process including: inputting a first scale feature map, a second scale feature map, and a third scale feature map into the Kolmogorov-Arnold network, and determining the scale level corresponding to each scale feature map; obtaining a univariate basis function corresponding to each scale feature map, and determining a superposition operator of the Kolmogorov-Arnold network; and performing a KAN pyramid coding operation on each scale feature map according to the scale level, the univariate basis function, and the superposition operator.

[0080] Among them, the feature maps output by the first scale block, the second scale block, and the third scale block of PET and MRI are used to perform KAN pyramid encoding on the feature maps of different scales of PET and MRI images using KAN. The calculation formula can be expressed as:

[0081] Among them, l∈{1,2,3} represents the scale level, l=1 corresponds to the scale level of the feature map output by the first scale block, and the size is 256×256 at this time, l=2 corresponds to the scale level of the feature map output by the second scale block, and the size is 128×128 at this time, l=3 corresponds to the scale level of the feature map output by the third scale block, and the size is 64×64 at this time, and They represent the PET feature map and MRI feature map of the lth scale respectively, Φ is the superposition operator of KAN, and are the univariate basis functions of the PET feature map and the MRI feature map of the i-th layer, respectively, i∈{1,2,…n}. Each KAN module contains 4 hidden layers, and 32 basis functions are set in each layer, that is, n=32. The activation function uses SIREN (Sinusodal Representation Networks) to capture the complex patterns in the feature map.

[0082] In one embodiment, a Kolmogorov-Arnold network-based PET and MRI multimodal medical image fusion method is provided, which may also include a feature fusion process, specifically including: forcing basis function parameter sharing through a dynamic weight sharing mechanism to achieve cross-modal pyramid alignment; guiding and complementing the features in feature maps of each scale through an asymmetric attention mechanism, and dynamically balancing parameters in bidirectional KAN attention fusion to achieve bidirectional information flow adaptive adjustment, complete cross-modal feature fusion, and obtain a first fused feature map.

[0083] Computer equipment can use a dynamic weight sharing mechanism to force the corresponding scale KAN modules of PET and MRI to share 50% of the basis function parameters, thereby achieving cross-modal pyramid alignment and enhancing the correlation between cross-modal features. The mathematical constraint formula is expressed as: Where m = 16, Top m denotes the index corresponding to the largest m values, and cos(·) represents the calculation of cosine similarity. By calculating the cosine similarity between basis function parameters, the 16 most similar basis functions are selected for sharing. The shared parameter set is updated every five training epochs, enabling the KAN modules of the two modalities to learn similar feature representations.

[0084] In this embodiment, the computer device can also design a dual-channel Kolmogorov kernel function to achieve cross-modal interaction. Through the asymmetric attention mechanism, that is, independent calculations in the PET→MRI and MRI→PET directions, the PET and MRI features can guide and complement each other. In the bidirectional KAN attention fusion, the gating mechanism realizes adaptive adjustment of the bidirectional information flow through the dynamic balance parameter γ, ultimately achieving deep cross-modal feature fusion. The PET→MRI direction formula is expressed as: The MRI→PET direction formula is expressed as: F fuse =γ·(A PET→MRI ·F MRI )+(1-γ)·(A MRI→PET ·F PET ); where the gate network formula is expressed as: γ=σ(Wg Concat(A PET→MRI ,A MRI→PET )+b g ).in, The tensor product operation implemented by KAN, d = 256 is the feature dimension, F PET and F MRI The KAN module is used to align the PET and MRI feature maps with the original input features after cross-modal pyramid alignment (i.e. parameter sharing). and The processed result, γ is the dynamic balance parameter, W g and b g is a trainable parameter, σ represents the Sigmoid activation function, and Concat(2) represents concatenating the bidirectional attention matrix along the channel dimension.

[0085] Step 206 , introduce a bidirectional phase interference mechanism to decompose and reconstruct the first fused feature map, and design a reversible decomposition network with a mirror-symmetric structure to perform multi-scale decomposition to obtain a decoded feature map.

[0086] A bidirectional phase interference mechanism is introduced to map real features into the complex domain for interference operations, enhancing the coupling strength between features and improving their expressiveness. Specifically, the computer device can construct a reversible pyramid decoding network, achieve cross-scale feature refinement through phase modulation fusion and residual gradient compensation, and adopt a momentum-constrained parameter update strategy to enhance the stability of feature reconstruction.

[0087] In one embodiment, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network is provided, which may also include a decomposition and reconstruction process, the specific process including: introducing a bidirectional phase interference mechanism, mapping the real features in the first fused feature map to the complex domain for interference operation, and reconstructing the real features by decomposing the real and imaginary parts; designing a reversible decomposition network with a mirror-symmetric structure, performing multi-scale edge extraction on the first fused feature map to obtain a scale-sensitive mask; based on the scale-sensitive mask, performing multi-scale decomposition of the first fused feature map through an inverse KAN to obtain a decoded feature map.

[0088] After completing the orthogonal projection in the complex domain, the computer device can reconstruct the real number features by decomposing the real and imaginary parts. The formula is expressed as: Among them, θ k For independently learnable phase parameters, k = 1, 2 correspond to the fusion feature components in the PET→MRI and MRI→PET directions, respectively.

[0089] In this embodiment, the computer device can design a reversible decomposition network based on the mirror symmetric structure of KAN. The network consists of three groups of inverse KAN modules. Each module corresponds to the feature reconstruction of a specific scale level to ensure the lossless recovery of the bidirectional fusion information. First, the fusion feature F fuse Perform multi-scale edge extraction to obtain a scale-sensitive mask, which is expressed as: M l =Sobel(F fuse )·G σ +Upsample(B l ); where G σ is a Gaussian kernel, and σ=1.2, B l is the binary mask corresponding to each scale, M l is the scale mask matrix generated by the Sobel operator and satisfies To achieve non-overlapping coverage of features; the entangled enhanced fusion feature F fuse The multi-scale decomposition is achieved through the inverse KAN module to obtain the decoded features The lossless recovery of bidirectional fusion information is ensured, and the decoding process is expressed as: in, is the kth univariate basis function of the reverse path, parameterized by quintic B-spline, is the j-th univariate function of the forward path, which is mirror-symmetric with the forward KAN module. is the inverse KAN module of the lth scale, the inverse KAN module The formula is expressed as: Among them, RevIN represents reversible instance normalization, the parameters share the mean and variance of the forward module, DWT represents discrete wavelet transform, which is used to separate low-frequency and high-frequency components. The cubic B-spline basis function represents the adaptive node distribution, and the node positions are learned by the learnable parameters At the same time, the adaptive sample point distribution is based on the gradient amplitude of the current basis function, and the node interval is adjusted using the node adaptive strategy, that is, the three spline node intervals Δ0 = 0.1 are first initialized, and the basis function gradient amplitude is calculated after each update of the basis function parameters. Then according to the exponential decay rule Adjust the node interval of the next basis function; in order to ensure the symmetry and strict reversibility of the reverse module and the forward module, the reversible decomposition network adopts parameter symmetry constraints and Jacobian determinant orthogonalization respectively. The formula to ensure symmetry is expressed as: in, represents the inverse KAN module of the lth scale The parameter set, represents the forward KAN module Ψ of the lth scale l The parameter set is ρ = 0.7, which is the symmetry coefficient. Flip represents the spatial flip operation of the parameter tensor. The formula to ensure reversibility is expressed as: Where I is the identity matrix, λ is a hyperparameter used to control the strength of the constraint, diag(∈) is a diagonal matrix, and ∈ is usually a small positive vector to avoid the problem that the strict equality is difficult to satisfy due to numerical calculations and other reasons; then the fused features are transformed and reconstructed in the complex domain, and the following steps are used to enhance the feature reversibility and representation ability. The complex domain transformation formula for the fused features is expressed as: Among them, the phase angle θ is a learnable parameter, and i is an imaginary unit. It should be noted that in order to ensure that the decoded features maintain topological consistency with the encoded features in high-frequency information such as edge response and texture gradient, multi-scale consistency loss is added in the back propagation, and its formula is expressed as follows: Among them, D l is a dynamic downsampling operator, and its kernel function parameters are adaptively adjusted through bilinear interpolation. Finally, the regularization term is introduced to constrain the reversible error and the smoothness of the basis function, and its formula is expressed as: Among them, λ1=0.1, λ2=0.01, constraining the reversible error can ensure that the decoder can restore the input features as accurately as possible, improving the reversibility of the entire reversible network; constraining the smoothness of the basis function helps to avoid overfitting, and is also conducive to ensuring the stability and continuity of feature transformation.

[0090] In step 208, the fourth-scale feature map is input into the frequency-domain-spatial-domain collaborative fusion framework based on dynamic sparse attention guidance, and feature fusion is achieved based on the dual-path cross-attention mechanism to obtain a second fused feature map; reversible instance normalization is used to perform feature integration on the second fused feature map to obtain an integrated feature map.

[0091] In order to achieve frequency-band adaptive fusion of anatomical structure and functional metabolic information in medical images, a frequency-domain-spatial-domain collaborative fusion framework based on dynamic sparse attention guidance was proposed. The dynamic sparse attention mechanism was used to achieve adaptive weighting of multi-band features, and reversible instance normalization technology was introduced to ensure frequency domain energy conservation.

[0092] In one embodiment, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network is provided, which may also include a feature fusion process, the specific process including: inputting the fourth-scale feature map into a frequency domain-spatial domain collaborative fusion framework based on dynamic sparse attention guidance, performing frequency band decomposition through an improved discrete cosine transform to generate frequency band features; constructing a dual-path cross-attention mechanism, and calculating fusion features based on the dual-path cross-attention mechanism, and adaptively sharpening the frequency band features through the Laplace-Gaussian operator based on the fusion features; and integrating the sharpened features and fusion features using improved reversible instance normalization to obtain a second fusion feature map.

[0093] In this embodiment, a dynamic sparse attention-guided frequency-spatial collaborative fusion framework (DSAG-FSCF) is proposed to process the output feature maps of the fourth scale block of the two branches of PET and MRI images. and First, the frequency band is decomposed by the improved discrete cosine transform (DCT), and the formula is expressed as follows: Among them, k∈{PET, MRI}, z∈{low, mid, high}, Represents a learnable frequency domain mask matrix, initialized with exponential decay, and its formula is expressed as: Among them, α l ={0.1, 0.5, 1.0}, controls the energy attenuation rate of different frequency bands (low, mid, high), d is the Euclidean distance from the two-dimensional frequency plane coordinate to the center point, used to construct the radial attenuation weight distribution and generate three sets of frequency band features and Next, the computer device can construct a dual-path cross-attention mechanism, which is expressed as: Among them, the query matrix Q, key matrix, and K value matrix V are generated by separable convolution, and their formula is expressed as: The convolution kernel The size is 3×3; the dynamic weight coefficient β z = Among them, σ represents the Sigmoid activation function; the final fusion feature The formula is expressed as: Adaptive feature sharpening is achieved by enhancing high-frequency features through the Laplace-Gaussian operator. The formula is expressed as: Among them, σ i Represents the scale parameter of the Laplace-Gaussian (LoG) filter, which is distributed in a geometric sequence, i.e. σ i+1=0.6σ i , initial σ1=1.2, used for multi-scale edge detection, adaptive sharpening factor λ i Through adaptive adjustment of the gradient amplitude, the formula is expressed as: Where N is the number of pixels in the feature map, γ = 0.8 is the empirical coefficient, and the amplitude is normalized to (0, 1] by the tanh function to avoid over-enhancement. The improved reversible instance normalization (ReVIN++) is used for feature integration, and its formula is expressed as: Among them, the inverse transformation Ψ -1 The following equations are satisfied: Among them, ∈ is a small perturbation term, and |∈|<0.01, to ensure numerical stability; and the parameter update adopts momentum constraint, which is expressed as: θ t+1 =0.9θ t +0.1θ rev ; where θ rev To reverse the path parameters, the feature distribution consistency is maintained through momentum constraints; the computer equipment also designed a dual-path verification loss function based on the multi-level gradient alignment verification method, which is expressed as: Among them, φ s is the three-scale spatial pyramid pooling, D s is a learnable downsampling kernel, It represents the features of the third scale block encoding output, which implies the texture features of PET and MRI dual modalities that have been fused in the first three convolution layers. It is used as a verification benchmark to ensure that the fusion of high-level features does not destroy the complementary relationship of the underlying modalities. The total variation (TV) constraint term ||X|| TV The sum of the absolute values ​​of the first-order differences in the horizontal and vertical directions strengthens edge continuity. The formula is expressed as: ||X|| TV =∑ i,j |X i+1,j -X i,j |+|X i,j+1 -X i,j |.

[0094] In step 210 , the integrated feature map is subjected to frequency domain expansion based on reversible frequency domain upsampling, and channel weighted splicing, space-channel decoupling processing, and interactive fusion processing to obtain a final fused medical image.

[0095] In this embodiment, a cross-scale progressive fusion mechanism based on dynamic reversible KAN pyramid (DTR-KAN) is proposed to The splicing and decoding operations are used to fuse the multi-scale feature maps of PET and MRI, and finally the fused feature map is obtained.

[0096] In one embodiment, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network may further include an image feature fusion process, specifically comprising: performing frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling to obtain an upsampling result; performing channel-weighted splicing on the upsampling result and the third feature map output by the pyramid to obtain a spliced ​​feature, and performing spatial-channel decoupling processing on the spliced ​​feature using a depthwise separable KAN convolution to obtain a decoupled feature map; introducing a cross-modal residual compensation mechanism to compensate for the decoupled feature map. The feature map is interactively fused with the second feature map output by the pyramid to obtain a preliminary fused feature map; the preliminary fused feature map is upsampled, and a multi-granularity feature expression is constructed using dynamic pyramid pooling to obtain an expression feature map; a quantum entanglement-inspired phase synthesis method is introduced to synthesize the expression feature map and the first feature map output by the pyramid to obtain a final feature map; the multimodal features of the final feature map are decoupled and projected, and the mapping of features to pixel space is achieved through a dynamic parameterized kernel, and the anatomical structure in the complex domain activation feature image is aligned with the frequency domain of metabolic information to obtain the final fused medical image.

[0097] Among them, the feature map generated by the DSAG-FSCF framework Frequency domain expansion is performed through a reversible KAN module based on reversible frequency domain upsampling, and its formula is expressed as: Among them, the RevKAN module structure formula is expressed as: in, For an invertible frequency domain filter, the parameter constraint is satisfied as follows: Among them, δ is the Dirac function, ∈ is the Gaussian noise term (σ=0.1); then, the upsampling result can be Feature map output from the pyramid Perform channel weighted splicing, and the formula is expressed as: Among them, α is the dynamic weight, and its formula is expressed as: Among them, KAN gate It is a dual-input single-output network (2 layers, 8 basis functions per layer). Then, the computer device can use the depth-separable KAN convolution (DS-KANConv) to concatenate features. Perform space-channel decoupling processing, and the formula is expressed as: Among them, in the depth-separable KAN convolutional network, the spatial convolution kernel parameters satisfy the following formula: The amplitude A k satisfy λ k=0.2 l , l = 2 corresponds to this layer; channel convolution uses B-spline basis function, and its formula is expressed as: in, It is a 4th-order B-spline with a knot spacing of Δ=0.25.

[0098] Then, the computer device can introduce a cross-modal residual compensation mechanism to achieve the splicing feature Feature map after space-channel decoupling processing With feature map For interactive fusion, the formula is expressed as: Among them, Attn cross represents the cross attention mechanism, and its formula is expressed as: Among them, τ = 0.7 represents the temperature coefficient, KAN sim The network contains 16 radial basis functions;

[0099] Computer equipment can also use dynamic pyramid pooling to construct multi-granularity feature expressions to achieve feature maps The upsampling operation is expressed as follows: Among them, the upsampling kernel parameter formula is expressed as: Among them, w up Through KAN regression learning; introduce quantum entanglement inspired phase synthesis method to feature map and feature maps The synthesis formula is expressed as: Where θ represents the phase parameter, which is obtained by the following formula: Among them, the multi-layer perceptron MLP contains 3 layers of KAN, each layer has 4 basis functions; then, the computer device can Multimodal feature decoupling projection realizes the mapping of features to pixel space through dynamic parameterized kernel, which is expressed as: in, The projection weight kernel representing the PET modality is dynamically generated by the KAN network. Represents the projection weight kernel of the MRI modality, which satisfies the orthogonal constraint with the PET kernel. ⊙ represents channel-by-channel multiplication to achieve modality-specific feature enhancement. Complex domain activation is used to ensure the frequency domain alignment of anatomical structure and metabolic information. Its formula is expressed as: I act =ReLU(Real(I fusion ))+i·Sigmoid(Imag(I fusion )); where Real(·) represents the extraction of the real part corresponding to the anatomical structure details, Imag(·) retains the metabolic function information, and finally the real part Real(·) is taken as the output of the final fusion image, i.e., I output=Real(I fusion ); edge consistency can also be enhanced by dual-path constraint loss, which is expressed as: in, is the gradient field calculated by the Sobel operator, It is the affine transformation matrix learned by KAN, which realizes the adaptive mapping of feature gradient to image gradient.

[0100] In one embodiment, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network is provided, which may also include a process of setting a loss function. The specific process includes: setting content loss, frequency domain energy loss, structural similarity loss, and gradient alignment loss respectively; determining a dynamic weighting coefficient, and calculating a composite loss function based on the set losses and dynamic weighting coefficients; wherein the composite loss function is used to enhance feature fusion.

[0101] In order to achieve loss optimization of the overall network architecture, a multi-task composite loss function system can be designed, integrating pixel-level content constraints, frequency domain energy distribution alignment, multi-scale structural similarity measurement and directional gradient field matching, and realizing adaptive optimization of loss weights through dynamic gating network. Specifically, a composite loss function can be designed. Composite loss function combined with content loss Frequency domain energy loss Structural similarity loss and gradient alignment loss To further enhance the feature fusion effect and optimize the model, the formula of the composite loss function is expressed as: Among them, λ1=0.5, λ2=0.3, λ3=0.4, and λ4=1.2 are dynamic weighting coefficients, which are adaptively adjusted by the KAN gating network. Gate It is composed of two layers of KAN modules stacked together. The formula of the first layer of the KAN module for feature mapping is expressed as: The formula for the second layer of the KAN module for weight generation is expressed as: Among them, φ jk (·) and ψ lj (·) is a learnable spline basis function, parameterized as a B-spline curve, the number of basis functions B = 5, the order k = 3, and the activation function φ(u) adopts a piecewise polynomial form, which is expressed as follows: Among them, b c (u) is the normalized B-spline basis function, w c is a trainable coefficient; by the gated network KAN Gate Adaptively adjusted λ i The formula is as follows: Among them, the input feature is the multimodal feature statistics of the current training stage, epoch_ratio = t / T represents the training progress, t is the current epoch, T is the total epoch, and the remaining items are the proportions of each loss item to the total loss; the output of the gating network is the normalized weight coefficient, which is expressed as: [λ′1,λ′2,λ′3,λ′4] = Softmax(KAN Gate (x)); the final dynamic weight is adjusted by scaling, and its formula is expressed as: i =σ i ·λ′ i +β i , (i=1,2,3,4); where, σ i and β i is a learnable parameter used to control the initial range and offset of the weight; content loss It is used to ensure that the fused image is aligned with the ground truth (GT) at the pixel level, preserving PET metabolic activity and MRI anatomical details. Its expression is: Among them, ROI mod represents the modality-specific region of interest mask, which is generated by threshold segmentation (SUV value>2.5 region in PET, edge gradient>0.1 region in MRI), and ||·||1 represents the L1 norm; it should be noted that the content loss A phased weighting is used, which is higher (0.8) in the early stage of training and drops to 0.5 in the later stage to balance detail preservation and noise suppression; frequency domain energy loss It is used to constrain the fusion image to be consistent with the joint frequency band energy distribution of the input dual-modal image in the frequency domain to avoid high-frequency artifacts and low-frequency distortion. Its formula is expressed as: in, represents the discrete cosine transform (DCT) frequency band decomposition operator, with cutoff frequencies set to 0.2 (low frequency), 0.5 (medium frequency), and 0.8 (high frequency). Represents the frequency domain superposition of input PET and MRI; it should be noted that in calculating the frequency domain energy loss When , the loss weight should be adjusted according to the frequency band energy ratio (60% for low frequency, 30% for medium frequency, and 10% for high frequency) to suppress overfitting of high frequency noise; structural similarity loss It is used to maintain the structural similarity between the fused image and the GT in the multi-scale spatial domain. Its formula is expressed as: Among them, I (s) It is obtained by downsampling the Gaussian pyramid with a scaling factor of 0.5 s-1 , covering three scales: full resolution, 1 / 2 resolution, and 1 / 4 resolution. SSIM is the structural similarity index, which is used to calculate the product of brightness, contrast, and structural similarity; gradient alignment loss It is used to force the fusion image to be consistent with the GT in terms of edge and texture gradient distribution, and to improve the clarity of the lesion boundary. The formula is expressed as: in, Represents a directional Sobel operator, the weight matrix is ​​determined by the angle θ, and the TV (Total Variation) norm is used to penalize the gradient amplitude difference while suppressing discontinuous mutations.

[0102] In this application, a PET and MRI multimodal medical image fusion method based on the Kolmogorov-Arnold network is provided. By constructing a four-level dynamic interpretable feature extraction framework, a mechanism for sharing high-frequency common features and decoupling low-frequency specific features between modalities is proposed to achieve hierarchical adaptive expression of PET metabolic functional information and MRI anatomical structural features; a frequency domain-spatial domain dual-driven fusion paradigm is designed, and a phase modulation technology inspired by complex domain quantum entanglement is adopted to enhance spatial detail coupling while ensuring cross-modal frequency domain energy conservation; an innovative reversible pyramid decoding architecture is used to achieve high-fidelity reconstruction of multi-granularity features through a dynamic parameterized upsampling kernel and a cross-scale residual gradient compensation path; a multi-physical constraint optimization system is established, which integrates the direction-sensitive gradient alignment mechanism and the adaptive frequency band energy distribution constraint to form a closed-loop optimization in three dimensions: pixel-level fidelity, multi-scale structural consistency, and cross-modal noise suppression. It breaks through the traditional method's reliance on linear feature superposition, deeply integrates the function approximation characteristics of the Kolmogorov-Arnold network with the physical priors of medical imaging, and achieves theoretical innovations in three aspects: dynamic parameter sharing mechanism, frequency domain constraint modeling, and reversible feature propagation path. It provides a solution for multi-parameter medical image fusion that combines anatomical structure clarity and metabolic function sensitivity.

[0103] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0104] In one embodiment, Figure 3As shown, a PET and MRI multimodal medical image fusion system based on a Kolmogorov-Arnold network is provided, comprising: a multi-scale dynamic feature extraction module 310, a cross-modal pyramid interaction module 320, a reversible pyramid decoding and reconstruction module 330, a frequency domain-spatial domain collaborative fusion module 340, and a feature map fusion module 350, wherein:

[0105] A multi-scale dynamic feature extraction module 310 is used to collect and normalize PET and MRI medical images, and input them into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate first-scale feature maps, second-scale feature maps, third-scale feature maps, and fourth-scale feature maps based on the initial feature maps.

[0106] A cross-modal pyramid interaction module 320 is configured to input the first-scale feature map, the second-scale feature map, and the third-scale feature map into a Kolmogorov-Arnold network for pyramid encoding, and to achieve cross-modal feature fusion of each feature map through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map.

[0107] The reversible pyramid decoding and reconstruction module 330 is used to introduce a bidirectional phase interference mechanism to decompose and reconstruct the first fused feature map, and design a reversible decomposition network with a mirror-symmetric structure to perform multi-scale decomposition to obtain a decoded feature map;

[0108] The frequency-spatial collaborative fusion module 340 is configured to input the fourth-scale feature map into the frequency-spatial collaborative fusion framework based on dynamic sparse attention guidance, perform feature fusion based on a dual-path cross-attention mechanism, and obtain a second fused feature map; and perform feature integration on the second fused feature map using reversible instance normalization to obtain an integrated feature map.

[0109] The feature map fusion module 350 is used to perform frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling, and perform channel weighted splicing, space-channel decoupling processing and interactive fusion processing to obtain the final fused medical image.

[0110] In one embodiment, the multi-scale dynamic feature extraction module 310 is also used to collect PET and MRI medical images and perform modality-specific preprocessing, and input them into the normalization layer of each modality respectively; the preprocessed PET medical images are logarithmically normalized through the normalization layer, and the preprocessed MRI medical images are Z-score normalized; the normalized PET and MRI medical images are input into the multi-scale dynamic convolution kernel architecture; wherein, the KAN convolution kernel in the multi-scale dynamic convolution kernel architecture follows the Kolmogorov-Arnold theorem, decomposing the multivariate function into a combination of univariate basis functions, which is used to realize the adaptive extraction of multi-scale features through the inter-modality parameter sharing and exclusive mechanism.

[0111] In one embodiment, the multi-scale dynamic feature extraction module 310 is also used to implement spatial-channel domain joint modeling in a hierarchical feature extraction network using sine basis functions and B-spline basis functions; determine the spatial basis functions and channel basis functions, and calculate the single-layer KAN convolution kernel in the hierarchical feature extraction network to generate an initial feature map based on the spatial basis functions and channel basis functions; based on the initial feature map, use dynamic KAN convolution and learnable downsampling to generate a first-scale feature map; based on the first-scale feature map, use depth-separable KAN convolution and cross-modal interaction to generate a second-scale feature map; based on the second-scale feature map, use residual KAN blocks and frequency domain constraints to generate a third-scale feature map; based on the third-scale feature map, use cross-modal entanglement and gradient coupling compression to generate a fourth-scale feature map.

[0112] In one embodiment, the cross-modal pyramid interaction module 320 is further used to input the first scale feature map, the second scale feature map, and the third scale feature map into the Kolmogorov-Arnold network, and determine the scale level corresponding to each scale feature map; obtain the univariate basis function corresponding to each scale feature map, and determine the superposition operator of the Kolmogorov-Arnold network; and perform a KAN pyramid encoding operation on each scale feature map according to the scale level, the univariate basis function, and the superposition operator.

[0113] In one embodiment, the cross-modal pyramid interaction module 320 is further used to enforce basis function parameter sharing through a dynamic weight sharing mechanism to achieve cross-modal pyramid alignment; to guide and complement the features in each scale feature map through an asymmetric attention mechanism, and to dynamically balance parameters in the bidirectional KAN attention fusion to achieve bidirectional information flow adaptive adjustment, complete cross-modal feature fusion, and obtain a first fused feature map.

[0114] In one embodiment, the reversible pyramid decoding and reconstruction module 330 is also used to introduce a bidirectional phase interference mechanism, map the real features in the first fused feature map to the complex domain for interference operation, and reconstruct the real features through real and imaginary part decomposition; design a reversible decomposition network with a mirror-symmetric structure, perform multi-scale edge extraction on the first fused feature map to obtain a scale-sensitive mask; based on the scale-sensitive mask, the first fused feature map is decomposed into multiple scales through an inverse KAN to obtain a decoded feature map.

[0115] In one embodiment, the frequency domain-spatial domain collaborative fusion module 340 is also used to input the fourth-scale feature map into the frequency domain-spatial domain collaborative fusion framework based on dynamic sparse attention guidance, perform frequency band decomposition through improved discrete cosine transform, and generate frequency band features; construct a dual-path cross-attention mechanism, and calculate the fusion features according to the dual-path cross-attention mechanism, and based on the fusion features, adaptive feature sharpening of the frequency band features is achieved through the Laplace-Gaussian operator; and use improved reversible instance normalization to integrate the sharpened features and fusion features to obtain a second fusion feature map.

[0116] In one embodiment, the feature map fusion module 350 is also used to perform frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling to obtain an upsampling result; perform channel-weighted splicing on the upsampling result and the third feature map output by the pyramid to obtain a spliced ​​feature, and use depth-separable KAN convolution to perform spatial-channel decoupling processing on the spliced ​​feature to obtain a decoupled feature map; introduce a cross-modal residual compensation mechanism to interactively fuse the decoupled feature map with the second feature map output by the pyramid to obtain a fused preliminary feature map; perform an upsampling operation on the fused preliminary feature map, and use dynamic pyramid pooling to construct a multi-granularity feature expression to obtain an expression feature map; introduce a quantum entanglement-inspired phase synthesis method to synthesize the expression feature map and the first feature map output by the pyramid to obtain a final feature map; decouple the multimodal features of the final feature map, realize feature to pixel space mapping through a dynamic parameterized kernel, and use the complex domain activation feature image to align the anatomical structure and metabolic information in the frequency domain to obtain a final fused medical image.

[0117] In one embodiment, Figure 4 As shown, a PET and MRI multimodal medical image fusion system based on a Kolmogorov-Arnold network can also include a multi-task optimization module 360 ​​for setting content loss, frequency domain energy loss, structural similarity loss, and gradient alignment loss respectively; determining a dynamic weighting coefficient, and calculating a composite loss function based on the set losses and dynamic weighting coefficients; wherein the composite loss function is used to enhance feature fusion.

[0118] In one embodiment, the multi-scale dynamic feature extraction module constructs a four-scale dynamic convolution kernel architecture, adopts a high-frequency parameter sharing and low-frequency modality exclusive mechanism, and introduces sine basis functions and B-spline basis functions in the spatial domain and channel domain, respectively, to achieve hierarchical decoupling of anatomical structural commonality and metabolic feature specificity. The cross-modal pyramid interaction module innovatively proposes a bidirectional parameter constraint mechanism, screens cross-modal shared basis functions based on cosine similarity, and combines a parameter entanglement layer inspired by complex domain quantum entanglement to enhance the spatial consistency of multi-scale features. The frequency-spatial collaborative fusion module designs a dynamic sparse attention guidance mechanism, which achieves the adaptive fusion of high-frequency detail enhancement and low-frequency energy conservation through multi-band feature decomposition and cross-modal dynamic weight allocation. The reversible pyramid decoding and reconstruction module adopts phase modulation fusion technology to construct a reversible instance normalization layer and residual gradient compensation path, and realizes high-fidelity reconstruction of cross-scale features through dynamic upsampling kernels. The multi-task optimization module innovatively integrates directional gradient field matching loss and frequency domain energy constraint, and achieves a dynamic balance between pixel-level content preservation, structural similarity optimization, and cross-modal noise suppression through a learnable gating network.

[0119] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0120] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0121] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network.

[0122] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the steps of a PET and MRI multimodal medical image fusion method based on a Kolmogorov-Arnold network.

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network, characterized in that: The method comprises: PET and MRI medical images are collected and normalized, and then input into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate first-scale feature maps, second-scale feature maps, third-scale feature maps, and fourth-scale feature maps from the initial feature maps; Inputting the first-scale feature map, the second-scale feature map, and the third-scale feature map into the Kolmogorov-Arnold network for pyramid encoding, and implementing cross-modal feature fusion of each feature map through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map; A bidirectional phase interference mechanism is introduced to decompose and reconstruct the first fused feature map, and a reversible decomposition network with a mirror-symmetrical structure is designed to perform multi-scale decomposition to obtain a decoded feature map; Inputting the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework guided by dynamic sparse attention, achieving feature fusion based on a dual-path cross-attention mechanism, and obtaining a second fused feature map; performing feature integration on the second fused feature map using reversible instance normalization to obtain an integrated feature map; The integrated feature map is subjected to frequency domain expansion based on reversible frequency domain upsampling, and is subjected to channel weighted splicing, space-channel decoupling processing and interactive fusion processing to obtain a final fused medical image.

2. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: PET and MRI medical images are acquired and normalized, and then fed into a multi-scale dynamic convolution kernel architecture, including: Acquire PET and MRI medical images and perform modality-specific preprocessing before inputting them into the normalization layer of their respective modalities; The preprocessed PET medical image is subjected to logarithmic normalization by the normalization layer, and the preprocessed MRI medical image is subjected to Z-score normalization; Input the normalized PET and MRI medical images into the multi-scale dynamic convolution kernel architecture; Among them, the KAN convolution kernel in the multi-scale dynamic convolution kernel architecture follows the Kolmogorov-Arnold theorem, decomposing the multivariable function into a combination of univariate basis functions, which is used to realize the adaptive extraction of multi-scale features through the inter-modal parameter sharing and exclusive mechanism.

3. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate a first-scale feature map, a second-scale feature map, a third-scale feature map, and a fourth-scale feature map according to the initial feature map, including: The hierarchical feature extraction network uses sine basis functions and B-spline basis functions to implement joint modeling in the space-channel domain; the spatial basis function and the channel basis function are determined, and the single-layer KAN convolution kernel in the hierarchical feature extraction network calculates and generates an initial feature map according to the spatial basis function and the channel basis function; Based on the initial feature map, a first-scale feature map is generated using dynamic KAN convolution and learnable downsampling; Based on the first-scale feature map, a second-scale feature map is generated using depth-separable KAN convolution and cross-modal interaction; Based on the second-scale feature map, a third-scale feature map is generated using a residual KAN block and frequency domain constraints; Based on the third-scale feature map, a fourth-scale feature map is generated by adopting cross-modal entanglement and gradient coupling compression.

4. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: Inputting the first scale feature map, the second scale feature map, and the third scale feature map into a Kolmogorov-Arnold network for performing a pyramid encoding operation, including: Inputting the first scale feature map, the second scale feature map, and the third scale feature map into a Kolmogorov-Arnold network, and determining a scale level corresponding to each scale feature map; Obtain the univariate basis functions corresponding to each scale feature map and determine the superposition operator of the Kolmogorov-Arnold network; A KAN pyramid encoding operation is performed on each scale feature map according to the scale level, the single variable basis function, and the superposition operator.

5. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: The cross-modal feature fusion of each feature map is achieved through the dynamic weight sharing mechanism and the asymmetric attention mechanism to obtain the first fused feature map, including: The dynamic weight sharing mechanism is used to enforce basis function parameter sharing and achieve cross-modal pyramid alignment. Through the asymmetric attention mechanism, the features in the feature maps of each scale are guided and complemented with each other, and the parameters are dynamically balanced in the bidirectional KAN attention fusion to achieve bidirectional information flow adaptive adjustment, complete cross-modal feature fusion, and obtain the first fused feature map.

6. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: A bidirectional phase interference mechanism is introduced to decompose and reconstruct the first fused feature map, and a reversible decomposition network with a mirror-symmetrical structure is designed to perform multi-scale decomposition to obtain a decoded feature map, including: Introducing a bidirectional phase interference mechanism, mapping the real features in the first fusion feature map to the complex domain for interference operation, and reconstructing the real features by decomposing the real and imaginary parts; Designing a reversible decomposition network with a mirror-symmetric structure, performing multi-scale edge extraction on the first fused feature map to obtain a scale-sensitive mask; Based on the scale-sensitive mask, the first fused feature map is decomposed into multiple scales through an inverse KAN to obtain a decoded feature map.

7. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: Inputting the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework guided by dynamic sparse attention, achieving feature fusion based on a dual-path cross-attention mechanism, and obtaining a second fused feature map; Reversible instance normalization is used to perform feature integration on the second fused feature map to obtain an integrated feature map, including: Inputting the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework based on dynamic sparse attention guidance, performing frequency band decomposition through improved discrete cosine transform to generate frequency band features; Constructing a dual-path cross-attention mechanism, calculating a fusion feature according to the dual-path cross-attention mechanism, and performing adaptive feature sharpening on the frequency band feature through a Laplace-Gaussian operator based on the fusion feature; Improved reversible instance normalization is used to integrate the sharpened features and fused features to obtain the second fused feature map.

8. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: The integrated feature map is subjected to frequency domain expansion based on reversible frequency domain upsampling, and channel weighted splicing, space-channel decoupling processing and interactive fusion processing to obtain a final fused medical image, including: Performing frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling to obtain an upsampling result; Performing channel-weighted splicing on the upsampling result and the third feature map output by the pyramid to obtain a spliced ​​feature, and performing spatial-channel decoupling processing on the spliced ​​feature using a depthwise separable KAN convolution to obtain a decoupled feature map; A cross-modal residual compensation mechanism is introduced to interactively fuse the decoupled feature map with the second feature map output by the pyramid to obtain a fused preliminary feature map; Performing an upsampling operation on the fused preliminary feature map, and constructing a multi-granularity feature expression using dynamic pyramid pooling to obtain an expression feature map; A phase synthesis method inspired by quantum entanglement is introduced to synthesize the expression feature map and the first feature map output by the pyramid to obtain a final feature map; The multimodal features of the final feature map are decoupled and projected, and the mapping of features to pixel space is achieved through a dynamic parameterized kernel. The anatomical structure in the feature image is activated by the complex domain to align the frequency domain of metabolic information to obtain the final fused medical image.

9. The PET and MRI multimodal medical image fusion method based on Kolmogorov-Arnold network according to claim 1, characterized in that: The method further comprises: Set content loss, frequency domain energy loss, structural similarity loss, and gradient alignment loss respectively; Determine a dynamic weighting coefficient, and calculate a composite loss function based on each set loss and the dynamic weighting coefficient; The composite loss function is used to enhance feature fusion.

10. A PET and MRI multimodal medical image fusion system based on Kolmogorov-Arnold network, characterized in that: The system comprises: A multi-scale dynamic feature extraction module is used to collect and normalize PET and MRI medical images, and input them into a multi-scale dynamic convolution kernel architecture. The multi-scale dynamic convolution kernel architecture extracts initial feature maps through a hierarchical feature extraction network, and uses a multi-scale pyramid to sequentially generate first-scale feature maps, second-scale feature maps, third-scale feature maps, and fourth-scale feature maps from the initial feature maps. A cross-modal pyramid interaction module is used to input the first-scale feature map, the second-scale feature map, and the third-scale feature map into the Kolmogorov-Arnold network for pyramid encoding, and to achieve cross-modal feature fusion of each feature map through a dynamic weight sharing mechanism and an asymmetric attention mechanism to obtain a first fused feature map; A reversible pyramid decoding and reconstruction module is used to introduce a bidirectional phase interference mechanism to decompose and reconstruct the first fused feature map, and design a reversible decomposition network with a mirror-symmetrical structure to perform multi-scale decomposition to obtain a decoded feature map; a frequency-domain-spatial-domain collaborative fusion module, configured to input the fourth-scale feature map into a frequency-domain-spatial-domain collaborative fusion framework guided by dynamic sparse attention, implement feature fusion based on a dual-path cross-attention mechanism, and obtain a second fused feature map; and perform feature integration on the second fused feature map using reversible instance normalization to obtain an integrated feature map; The feature map fusion module is used to perform frequency domain expansion on the integrated feature map based on reversible frequency domain upsampling, and perform channel weighted splicing, space-channel decoupling processing and interactive fusion processing to obtain the final fused medical image.

Images