Cross-modal magnetic resonance image synthesis method and system

Abstract

The invention provides a cross-modal magnetic resonance image synthesis method and system, relates to the technical field of magnetic resonance imaging, and aims to solve the problems of data missing, low calculation efficiency and poor image quality in the existing image synthesis technology. The method comprises the following steps: encoding a multi-modal image through a multi-layer encoder to obtain multi-level encoding features; performing feature extraction on the multi-level coding features through a plurality of residual error mixing Mama modules in sequence to obtain corresponding optimization features respectively; inputting each optimized feature into a multi-feature extraction module for feature aggregation to obtain a multi-scale aggregation feature; performing multi-level feature fusion on the multi-scale aggregation features based on a plurality of modulation fusion modules; and through an implicit neural representation module, mapping the fusion features into pixel values by using a multi-layer perceptron, and combining the pixel values to generate a target modal image. According to the invention, the problems in the prior art are solved, and the image synthesis quality is improved.

Description

Technical Field

[0001] This invention belongs to the field of magnetic resonance imaging technology, and particularly relates to a method and system for synthesizing cross-modal magnetic resonance images. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Magnetic resonance imaging (MRI) is a non-invasive, radiation-free advanced medical imaging technique that uses signals generated by atomic nuclei under the influence of strong magnetic fields and radio frequency pulses to produce high-resolution images of human tissues. MRI has become an indispensable technique in clinical practice and research due to its high contrast, multiplanar imaging, and superior representation of soft tissues.

[0004] However, traditional MRI technology has certain limitations. The equipment is expensive and scanning efficiency is low. Routine MRI examinations typically require patients to remain still, which can easily cause anxiety and discomfort for children, elderly patients, and those with trauma or illness. Furthermore, carrying metal implants or devices poses safety risks, leading to some patients being unable to undergo MRI examinations due to equipment limitations, thus preventing the acquisition of complete multimodal data and affecting diagnostic results. Simultaneously, traditional MRI image synthesis and processing methods are susceptible to noise and motion artifacts, making it difficult to guarantee image quality under complex imaging conditions, and consequently failing to meet the needs for rapid and accurate diagnosis.

[0005] Furthermore, existing synthesis techniques, typically based on Transformer methods, suffer from high computational complexity, making it difficult to efficiently model long-range dependencies. While state-space models are emerging as a promising alternative due to their lower computational cost and superior long-range modeling capabilities, they still face insufficient spatial modeling capabilities in medical image processing, affecting the quality of image synthesis. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a method and system for synthesizing cross-modal magnetic resonance images. By using a residual hybrid Mamba module, long-range dependencies are effectively captured without increasing computational burden. Furthermore, by combining multi-stage, multi-scale feature extraction and fusion mechanisms, multi-level features are integrated to enhance the richness and robustness of feature representation. Simultaneously, implicit neural representation modeling is used to reflect the relationship between multimodal data and image space, enhancing the model's representational power. This overcomes problems such as equipment limitations and patient discomfort, thereby improving image quality.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention discloses a method for synthesizing cross-modal magnetic resonance images, comprising: Multimodal images are encoded using a multi-layer encoder to obtain multi-level encoded features; The multi-level encoded features are sequentially extracted through multiple residual hybrid Mamba modules to obtain the corresponding optimized features. Each optimized feature is input into a multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features; Based on multiple modulation and fusion modules, multi-scale aggregated features are fused at multiple levels. Through the implicit neural representation module, the fused features are mapped to pixel values ​​using a multilayer perceptron and combined to generate the target modality image; Each residual hybrid Mamba module comprises a selective spatial modeling layer, a channel compression layer, and a residual convolutional layer connected in sequence. Multi-level encoded features are input into the first residual hybrid Mamba module for feature extraction. The specific process is as follows: Multi-level encoded features are input into a selective spatial modeling layer for feature association, resulting in spatially enhanced attention features. The spatially enhanced attention features are input into the channel compression layer for channel compression to obtain the channel-compressed features. The channel-compressed features are input into the residual convolutional layer for feature aggregation to obtain the first optimized features.

[0008] As a further technical solution, the multi-level encoded features are sequentially extracted through multiple residual hybrid Mamba modules to obtain the corresponding optimized features. The specific process is as follows: The multi-level encoded features are input into the first residual hybrid Mamba module for feature extraction to obtain the first optimized features; The first optimized feature is input into the residual convolutional group, and the feature is extracted and fused through three consecutive convolutional residual blocks to obtain the convolutional group fused feature; The convolutional group fused features are input into the second residual hybrid Mamba module for feature extraction, resulting in the second optimized features; The second optimized feature is input into the residual convolutional block for feature extraction and fusion to obtain the convolutional block fused feature. The convolutional block fusion features are input into the third residual hybrid Mamba module for feature extraction, resulting in the third optimized features.

[0009] As a further technical solution, multi-level encoded features are input into a selective spatial modeling layer for feature association to obtain spatially enhanced attention features. The specific process is as follows: The multi-level encoded features are divided into multiple non-overlapping blocks of a set size; The non-overlapping blocks are sequentially processed with layer normalization and activation functions to obtain the first transformation feature; Non-overlapping blocks are sequentially input into a linear layer, a deep convolutional layer, and an activation function layer for linear transformation and feature extraction to obtain transformed depth features. The transformation depth features are input into the selected state space modeling unit for feature association to obtain spatial association features; The first transformation feature and the spatial correlation feature are multiplied element-wise to obtain the spatially enhanced attention feature.

[0010] As a further technical solution, the spatially enhanced attention features are input into a channel compression layer for channel compression to obtain channel-compressed features. The channel compression layer consists of two cascaded convolutional layers and a channel compression unit connected in parallel. The specific process is as follows: After performing layer normalization on the spatially enhanced attention features, they are added element-wise to the non-overlapping block features to obtain the fused features; The fused features are input into the channel compression unit for channel compression to obtain preliminary compressed features; The fused features are input into two consecutive convolutional layers for feature extraction, and the extracted features are added element-wise to the preliminary compressed features to obtain the channel-compressed features.

[0011] As a further technical solution, the channel-compressed features are input into a residual convolutional layer for feature aggregation to obtain the first optimized features. The residual convolutional layer includes multiple sets of convolutional streams and residual connections. The specific process is as follows: The channel-compressed features are input into the first set of convolutional streams, and feature extraction is performed through the convolutional layers. The extracted features are then batch normalized and processed by activation functions to obtain the first refined features. The first refined feature and the channel-compressed feature are fused through residual connection to obtain the first residual fused feature; The first residual fusion feature is input into the second set of convolutional streams, and features are extracted through convolutional layers. The extracted features are then batch normalized and processed by activation functions to obtain the second refined feature. The second refined feature and the channel-compressed feature are fused through residual connection to obtain the first optimized feature.

[0012] As a further technical solution, each optimized feature is input into a multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features. The multi-feature extraction module includes a concatenation layer, a convolutional layer, a global average pooling layer, a global max pooling layer, four parallel convolutional downsampling layers, and a multi-scale selection fusion unit. The specific process is as follows: Each optimized feature is fused through a concatenation layer and then feature extraction is performed through a convolutional layer to obtain the integrated feature. The integrated features are input into four parallel convolutional downsampling layers, and downsampling is performed using different convolutional sampling rates to obtain features at different scales for the corresponding layers. Features at different scales are input into a multi-scale selection fusion unit for fusion to obtain cross-scale fused features; Multi-scale aggregated features are obtained by adding and aggregating cross-scale fused features and integrated features element-wise through residual connections.

[0013] As a further technical solution, multi-scale features are input into a multi-scale selection fusion unit for fusion to obtain cross-scale fused features. The multi-scale selection fusion unit sequentially includes a global pooling layer, a concatenation layer, a multilayer perceptron, an activation function, a normalization layer, and residual connections. The specific process is as follows: Multi-scale features are input into the global average pooling layer and the global max pooling layer respectively for pooling processing to obtain global average pooling features and global max pooling features. The global average pooling feature and the global max pooling feature are concatenated through a concatenation layer to obtain the concatenated pooling feature; The concatenated pooled features are input into a multilayer perceptron to obtain attention weights; The attention weights are then processed sequentially through activation functions and normalization to obtain the attention weight coefficients. Based on the attention weight coefficients, the features of different scales of the corresponding layers are weighted and fused to obtain the corresponding fused multi-scale features. The fused multi-scale features are added element-wise through residual connections to obtain cross-scale fused features.

[0014] As a further technical solution, multi-level feature fusion is performed on multi-scale aggregated features based on multiple modulation fusion modules. The specific process is as follows: The multi-scale aggregated features are input into the corresponding three convolutional layers to obtain the first adaptation feature, the second adaptation feature, and the third adaptation feature; The first decoded feature is obtained by performing a convolution operation on the third optimized feature; The first adaptation feature and the first decoder feature are input into the first modulation and fusion module for feature modulation and fusion to obtain the first set of fused features; Perform a convolution operation on the first set of fused features to obtain the second decoded features; The second adaptation feature and the second decoder feature are input into the second modulation and fusion module for feature modulation and fusion to obtain the second set of fused features; The second set of fused features is convolved to obtain the third decoded feature; The third adaptation feature and the third decoder feature are input into the third modulation and fusion module for feature modulation and fusion to obtain the third set of fused features, that is, the multi-level fused features.

[0015] As a further technical solution, an implicit neural representation module is used to map the fused features into pixel values ​​using a multilayer perceptron, and then combine them to generate the target modality image. The specific process is as follows: Based on the fused features, a set of relative coordinates is generated and mapped to a high-dimensional space. In high-dimensional space, the corresponding positional code is obtained through a spatial coding function; The fused features and their corresponding location codes are combined and input into the multilayer perceptron decoder to obtain the final target point values. All target points are then combined to obtain the synthesized target modal image.

[0016] The second aspect discloses a system for synthesizing cross-modal magnetic resonance images, comprising: The encoding module is used to encode multimodal images through a multi-layer encoder to obtain multi-level encoded features; The residual hybrid Mamba module is used to sequentially pass multi-level encoded features through multiple residual hybrid Mamba modules for feature extraction, obtaining corresponding optimized features. Each residual hybrid Mamba module includes a selective spatial modeling layer, a channel compression layer, and a residual convolutional layer connected in sequence. The multi-level encoded features are input into the first residual hybrid Mamba module for feature extraction. Specifically, the multi-level encoded features are input into the selective spatial modeling layer for feature association, obtaining spatially enhanced attention features; the spatially enhanced attention features are input into the channel compression layer for channel compression, obtaining channel-compressed features; and the channel-compressed features are input into the residual convolutional layer for feature aggregation, obtaining the first optimized feature. The multi-feature extraction module is used to input each optimized feature into the multi-feature extraction module for feature aggregation, thereby obtaining multi-scale aggregated features; The modulation fusion module is used to perform multi-level feature fusion on multi-scale aggregated features based on multiple modulation fusion modules; The implicit neural representation module is used to map fused features to pixel values ​​using a multilayer perceptron and combine them to generate a target modality image.

[0017] The above one or more technical solutions have the following beneficial effects: In this embodiment, multiple residual hybrid Mamba modules are used to effectively capture long-range dependencies in images without increasing computational burden. This overcomes the problem that traditional Transformer-based methods, due to their high computational complexity, struggle to efficiently model long-range dependencies. Furthermore, a multi-feature extraction module, combined with multi-stage and multi-scale feature extraction and fusion mechanisms, promotes cross-layer information exchange, integrates multi-level features from different stages and scales, and enhances the richness and robustness of feature representation. Simultaneously, implicit neural representations are used for modeling, mapping multimodal features to the image space, thereby significantly enhancing the model's representational ability and its capacity to synthesize fine-grained anatomical structures, thus improving image synthesis quality.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a flowchart illustrating the framework of a method for synthesizing cross-modal magnetic resonance images according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the residual hybrid Mamba module according to Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the multi-feature extraction module in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the multi-scale selection fusion unit in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the implicit neural representation module according to Embodiment 1 of the present invention; Figure 6 This is a visual result diagram of synthesizing the T2 mode from the T1 and FLAIR modes in Embodiment 1 of the present invention; Figure 7 This is a visual result diagram of synthesizing the T1 mode from the T2 and FLAIR modes according to Embodiment 1 of the present invention; Figure 8 This is a visual result diagram of the synthesis of FLAIR modes from T1 and T2 modes in Embodiment 1 of the present invention. Detailed Implementation

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0023] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0024] Example 1 This embodiment discloses a method for synthesizing cross-modal magnetic resonance images.

[0025] To more clearly illustrate this embodiment, the synthesis process based on a cross-modal magnetic resonance image can be specifically described as follows: This embodiment provides a method for synthesizing cross-modal magnetic resonance images, including: S1. Multimodal images are encoded using a multi-layer encoder to obtain multi-level encoded features; S2. The multi-level encoded features are sequentially extracted through multiple residual mixing Mamba modules to obtain the corresponding optimized features; S3. Input each optimized feature into the multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features; S4. Based on multiple modulation fusion modules, multi-scale aggregated features are fused at multiple levels. S5. Through the implicit neural representation module, the fused features are mapped to pixel values ​​using a multilayer perceptron and combined to generate the target modality image.

[0026] In this embodiment, a cross-modal magnetic resonance image synthesis model mainly includes a multi-layer encoder, multiple residual hybrid Mamba modules, residual convolution groups and residual convolution blocks (occasional bottlenecks), a multi-feature extraction module (MFEB), and an implicit neural representation decoder.

[0027] like Figure 1 As shown, in step S1, the multimodal image is encoded by a multi-layer encoder to obtain multi-level encoded features.

[0028] By setting up multiple encoders, multi-layer feature extraction is performed on multimodal images. The extracted features include deep features of multimodal images. After spatial transformation, the deep features are input into the next stage for feature extraction and fusion.

[0029] Specifically, a three-layer encoder is set up, and the multimodal image is input into the three-layer encoder to obtain deep features. ,in The superscript 3 and subscript e in the code represent the result of the third encoder layer.

[0030] like Figure 1 As shown, in step S2, the multi-level encoded features are sequentially extracted through multiple residual mixing Mamba modules to obtain the corresponding optimized features.

[0031] S2-1. Input the multi-level encoded features into the first residual hybrid Mamba module for feature extraction to obtain the first optimized features.

[0032] In this embodiment, each residual hybrid Mamba module includes a selective spatial modeling layer, a channel compression layer, and a residual convolutional layer connected in sequence. Multi-level encoded features are input into the first residual hybrid Mamba module for feature extraction. The specific process is as follows: (1) Input the multi-level encoded features into the selective spatial modeling layer for feature association to obtain spatially enhanced attention features.

[0033] Residual hybrid Mamba module, with features of the third-layer encoder Using this as input, a selective state-space mechanism is employed to capture long-range correlations between features, thereby generating spatially enhanced attention features. The specific process is as follows: 1) Divide the multi-level coded features into multiple non-overlapping blocks of a set size.

[0034] 2) Perform layer normalization and activation function processing on the non-overlapping blocks in sequence to obtain the first transformation feature.

[0035] 3) Input the non-overlapping blocks sequentially into the linear layer, the deep convolutional layer and the activation function layer for linear transformation and feature extraction to obtain the transformed depth features.

[0036] 4) Input the transformation depth features into the selected state space modeling unit for feature association to obtain spatial association features.

[0037] 5) Multiply the first transformation feature and the spatial association feature element by element to obtain the spatially enhanced attention feature.

[0038] In the first residual hybrid Mamba module, the specific formula is: ; ); ; in, express It is divided into P non-overlapping blocks of size (p,p), and Lin represents layer normalization. This represents the activation function, and SSM represents the choice of state-space modeling. Represents the features after spatial enhancement. Represents a gated variable. It represents spatial correlation characteristics.

[0039] Choosing state-space modeling, the formula is: ; ; Where h represents the hidden state and z represents the input sequence. This represents the output sequence, where n is the sequence index. N is the number of dimensions in the state space representation, where N is the number of learnable parameters selected for state space modeling.

[0040] (2) Input the spatially enhanced attention features into the channel compression layer for channel compression to obtain the channel-compressed features.

[0041] (3) Input the channel-compressed features into the residual convolutional layer for feature aggregation to obtain the first optimized features.

[0042] Through steps (2) and (3), the channel compression layer (cmConv) and the residual convolutional layer (RCNN) further enhance the compactness and representational ability of feature representation.

[0043] ; ; in, This indicates the characteristics of the channel after compression; This represents the first optimized feature.

[0044] The model sequentially completes the capture of remote context information, the modeling of channel dependencies, and the extraction of local features, taking into account both global perception and local detail representation, effectively improving the overall feature representation capability. For example... Figure 2 As shown.

[0045] After the above steps, the long-range dependencies between features are captured by the selective state space modeling method, thereby generating a feature map with spatial enhancement capabilities. Based on the obtained feature map, the compactness and representation ability of the feature representation are further improved by channel compression and residual convolutional layers.

[0046] S2-2. Input the first optimized feature into the residual convolutional group, and perform feature extraction and fusion through three consecutive convolutional residual blocks to obtain the convolutional group fused feature; input the convolutional group fused feature into the second residual hybrid Mamba module for feature extraction to obtain the second optimized feature.

[0047] In this embodiment, the residual convolution group includes three residual convolution blocks, each of which includes two convolutional layers, a batch normalization layer, and an activation function layer connected in sequence.

[0048] The second optimized feature is obtained through the specific process of step S2-1.

[0049] S2-3. Input the second optimized feature into the residual convolutional block for feature extraction and fusion to obtain the convolutional block fused feature; input the convolutional block fused feature into the third residual hybrid Mamba module for feature extraction to obtain the third optimized feature.

[0050] In this embodiment, the third optimized feature is obtained through the specific process of step S2-1.

[0051] like Figure 1 As shown, in step S3, each optimized feature is input into the multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features.

[0052] like Figure 3 As shown, in this embodiment, the Multi Feature Extraction Module (MFEB) includes a concatenation layer, a convolutional layer, four parallel convolutional downsampling layers, and a multi-scale selection fusion unit (MSF). The specific process is as follows: (1) Each optimized feature is fused through a splicing layer and then extracted through a convolutional layer to obtain integrated features.

[0053] The outputs of the three RHMB modules are used as inputs, that is, the first optimized feature, the second optimized feature and the third optimized feature are concatenated in the channel dimension.

[0054] (2) Input the integrated features into four parallel convolutional downsampling layers respectively, and downsample them by different convolutional sampling rates to obtain different scale features of the corresponding layers.

[0055] Multi-scale features are obtained by using dilated convolutions with four different dilation rates.

[0056] The specific formulas for steps (1) and (2) are as follows:

[0057]

[0058] in, These represent the first optimization feature, the second optimization feature, and the third optimization feature, respectively. Represents dilated convolutions with different expansion rates; This represents multi-scale features, i.e., features at different scales corresponding to the same layer; This indicates an integrated feature.

[0059] (3) Input the features of different scales into the multi-scale selection fusion unit for fusion to obtain cross-scale fusion features.

[0060] The obtained channel context descriptors are convolved, summed, and combined. The multi-scale channel weights are then normalized using the sigmoid function and softmax operation to obtain the channel attention map.

[0061] like Figure 4 As shown, the multi-scale selection fusion unit sequentially includes a global average pooling layer, a global max pooling layer, a splicing layer, a multilayer perceptron, an activation function, a normalization layer, and residual connections. The specific process is as follows: 1) Input the multi-scale features into the global average pooling layer and the global max pooling layer respectively for pooling processing to obtain the global average pooling features and the global max pooling features.

[0062] The formulas for global average pooling feature and global max pooling feature are: ； ; in, and These represent global average pooling and global max pooling, respectively. and Channel context descriptor.

[0063] 2) The global average pooling feature and the global max pooling feature are concatenated through a concatenation layer to obtain the concatenated pooling feature.

[0064] ; ; in, Represents the pooling feature at the i-th scale. This represents the pooling features after concatenation.

[0065] 3) Input the concatenated pooled features into a multilayer perceptron to obtain attention weights.

[0066] 4) Pass the attention weights through activation functions and normalization processes in sequence to obtain the attention weight coefficients.

[0067] Attention weight coefficient, the formula is: ; in, This represents the j-th element of the channel attention map. express The j-th element, , , , These are the exponential mapping results of the j-th element at the corresponding four scales.

[0068] 5) Based on the attention weight coefficients, the features of different scales in the corresponding layers are weighted and fused to obtain the corresponding fused multi-scale features.

[0069] 6) The fused multi-scale features are added element-wise through residual connections to obtain cross-scale fused features.

[0070] (4) The cross-scale fusion features and the integrated features are aggregated element by element through residual connection to obtain multi-scale aggregated features.

[0071] Multi-scale aggregation features, the formula is: .

[0072] Through the above steps, the model captures multi-scale features through multi-scale receptive fields and adaptively adjusts the weights of features at each scale according to the feature content, learning and understanding the complementary information between multiple features.

[0073] like Figure 1 As shown, in step S4, multi-level feature fusion is performed on the multi-scale aggregated features based on multiple modulation fusion modules.

[0074] Based on multiple modulation and fusion modules, multi-scale aggregated features are fused at multiple levels. The specific process is as follows: (1) Input the multi-scale aggregated features into the corresponding three convolutional layers to obtain the first adaptation feature, the second adaptation feature and the third adaptation feature.

[0075] (2) Perform convolution operation on the third optimized feature to obtain the first decoded feature.

[0076] (3) Input the first adaptation feature and the first decoder feature into the first modulation and fusion module for feature modulation and fusion to obtain the first set of fused features.

[0077] In this embodiment, each modulation fusion module (MFM) includes a global average pooling layer, a multilayer perceptron, a normalization layer, an attention matrix layer, a splicing layer, and a convolutional layer.

[0078] In the first adjustment and fusion module, the third adaptation feature and the first decoder feature are input to the first modulation and fusion module for feature modulation and fusion. The specific process is as follows: 1) Input the first adaptation feature and the first decoding feature into the global average pooling layer for average pooling processing, and obtain the fused global average pooling feature by adding them element by element.

[0079] 2) Input the fused global average pooling features into a multilayer perceptron for nonlinear transformation to obtain the transformed global pooling features.

[0080] 3) Normalize the transformed global pooling features to obtain the feature weight matrix.

[0081] The specific formulas for steps 1) to 3) are as follows: ; ; ; ; in ， This indicates the result of convolution of the features extracted by the previous module MFEB, i.e., the first fitting feature. The superscript 1 indicates the input of the first layer of MFEB. This represents the output of the first layer of the corresponding decoder, i.e., the first decoding feature; and This represents the result after global average pooling of the input features; It is an intermediate variable, namely the transformed global pooling feature. This represents the feature weight matrix.

[0082] 4) Multiply the feature weight matrix element-wise with the first adaptation feature and the first decoding feature to obtain the enhanced first adaptation feature and the enhanced first decoding feature.

[0083] 5) Add the enhanced first adaptation feature and the enhanced first decoding feature element by element to obtain the fused enhanced feature.

[0084] 6) The fused enhanced features and the first decoded features are sequentially input into the splicing layer and the convolution layer for fusion and feature extraction to obtain the first set of fused features.

[0085] The first set of fusion features, the formula is: ; ; in, This represents the output of the first-layer MFM module, i.e., the first set of fused features. This indicates the enhanced features after fusion.

[0086] (4) Perform convolution operation on the first set of fused features to obtain the second decoded features.

[0087] (5) Input the second adaptation feature and the second decoder feature into the second modulation and fusion module for feature modulation and fusion to obtain the second set of fused features.

[0088] The second set of fusion features is obtained through steps 1) to 6) in step (3).

[0089] (6) Perform convolution operation on the second set of fused features to obtain the third decoding features.

[0090] (7) Input the third adaptation feature and the third decoder feature into the third modulation and fusion module for feature modulation and fusion to obtain the third set of fused features, that is, to obtain the multi-level fused features.

[0091] The third set of fusion features is obtained through steps 1) to 6) in step (3), that is, the multi-level fusion features are obtained.

[0092] Through the above steps, the model effectively promotes multi-scale feature integration, alleviates the feature sparsity problem encountered in the decoding stage, and thus improves the overall network performance.

[0093] like Figure 1 As shown, in step S5, the implicit neural representation module uses a multilayer perceptron to map the fused features into pixel values ​​and combine them to generate the target modality image.

[0094] like Figure 5 As shown, in this embodiment, the specific process is as follows: (1) Based on the fused features, a set of relative coordinates is generated and mapped to a high-dimensional space.

[0095] (2) In high-dimensional space, relative position encoding is obtained through spatial encoding function.

[0096] ; ; in, Let X represent a high-dimensional coordinate space, where X represents the set of relative position coordinates, and x represents the coordinate value in X. This represents a spatial encoding function that encodes two-dimensional coordinates into a higher dimension, where L equals 4.

[0097] (3) Combine the fused features and the corresponding position codes and input them into the multilayer perceptron decoder to obtain the final target point value.

[0098] The specific formula is as follows: ; ; in, This represents the feature vectors of four locally correlated points. Indicates relative position encoding. This represents a local point representation with location information. Represents the local integration weights. This represents a multilayer perceptron, i.e., a continuous image generation function. This represents the target point value. Combining all target points yields a synthetic target modal image.

[0099] Through the above steps, the model effectively models the continuous mapping between multimodal features and medical images, thereby improving the model's expressive power and detail restoration effect in medical image generation tasks.

[0100] By implementing the above steps, multimodal magnetic resonance images can be processed better, and multi-feature extraction, multi-level feature fusion, and a continuous mapping between features and images can be established, which significantly improves the accuracy and generation quality of cross-modal magnetic resonance image synthesis.

[0101] The cross-modal magnetic resonance image synthesis model is trained according to steps S1 to S5.

[0102] Different data augmentation and preprocessing techniques can be used during model training to improve model performance.

[0103] During training, a combination of multiple loss functions can be used. Optionally, a main loss function and an auxiliary loss function can be used to improve the stability and performance of the model.

[0104] By implementing the above steps, multimodal magnetic resonance images can be processed better, and feature extraction and view fusion of multimodal magnetic resonance images can be achieved, which helps to improve the cross-modal magnetic resonance image synthesis method.

[0105] S6. Verify and evaluate the cross-modal magnetic resonance image synthesis method.

[0106] To verify the effectiveness of this implementation method, a powerful computing tool, the NVIDIA RTX A6000 GPU (48G), and the popular deep learning library PyTorch were used for verification.

[0107] In this validation process, the parameters set included a learning rate of 1e-4, a total of 60 training iterations, and the use of the Adam optimization method.

[0108] To evaluate the performance of the method in this embodiment, the evaluation metrics used include: The metrics include Peak Signal-to-Noise Ratio (PSNR), which measures the similarity between the synthesized image and the real image. A higher PSNR value indicates that the image quality is closer to the original image, and the reconstruction effect is better.

[0109] The Structural Similarity Index (SSIM) is used to comprehensively measure the structural similarity between a reconstructed image and a real image, considering factors such as brightness, contrast, and structure. The closer the index is to 1, the more consistent the image structure, and the better the reconstruction result. Normalized root mean square error (NMSE) measures the overall error between the synthesized image and the real image. The smaller the value, the closer the synthesized result is to the real image.

[0110] These metrics allow for a comprehensive evaluation of the performance of the cross-modal magnetic resonance image synthesis method described in this embodiment. For example... Figure 6 The visual result of synthesizing the T2 mode from the T1 and FLAIR modes is shown in the figure; for example Figure 7 The visual result of synthesizing the T1 mode from the T2 and FLAIR modes is shown in the figure; for example Figure 8 Visual results of synthesizing FLAIR modes from T1 and T2 modes are presented; Tables 1-3 show the performance scores of different technical solutions under the same conditions. It is evident that the method in this embodiment outperforms existing methods in all aspects.

[0111] Existing algorithms for comparison include: PT-Net employs a multi-scale pyramid Transformer, emphasizing cross-scale context modeling capabilities to improve image synthesis performance.

[0112] CACR-Net proposes a confidence-guided fusion and cross-modal refinement network to further refine the target modality image.

[0113] HiNet designs specific sub-networks for different modalities to learn representations, and uses a fusion network to learn common latent representations of multimodal data, thereby synthesizing target images.

[0114] ResVIT employs a hybrid CNN-Transformer framework, combining residual Transformer blocks and channel compression modules to achieve multimodal medical image synthesis.

[0115] IxI-mamba is based on selective state space modeling, which effectively captures remote context dependencies while maintaining local structural accuracy, thus balancing global consistency with detailed expressiveness.

[0116] This embodiment calculates the performance metrics for three multimodal synthesis tasks. The method in this embodiment consistently outperforms existing methods in all metrics, demonstrating the best performance score.

[0117] Table 11+FLAIR generates T2 task

[0118] Table 2 shows the generation of task T1 using T2+FLAIR.

[0119] Table 3. FALIR task generation in T1+T2

[0120] The method described in this embodiment has broad application prospects and high application value in clinical practice and scientific research.

[0121] Diagnostic and research support: Multimodal images are encoded using a multi-layer encoder to obtain multi-level encoded features; The multi-level encoded features are sequentially extracted through multiple residual hybrid Mamba modules to obtain the corresponding optimized features. Each optimized feature is input into a multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features; Based on multiple modulation and fusion modules, multi-scale aggregated features are fused at multiple levels. By employing an implicit neural representation module, a multilayer perceptron is used to map fused features to pixel values ​​and combine them to generate a target modality image. Missing modalities can lead to incomplete medical image data, which can affect clinical diagnosis and scientific research. By synthesizing missing modalities, doctors and researchers can obtain more comprehensive image information, helping to make more accurate diagnoses and conduct more in-depth research.

[0122] Cost savings: Obtaining complete multimodal medical image data typically requires additional cost and time, including extra scanning time and equipment. By synthesizing missing modalities, the necessary information can be obtained without increasing cost and time, thereby improving the efficiency of healthcare.

[0123] Case data augmentation: Having more case data is invaluable for medical research and education. Synthesizing missing modalities can be used to augment existing medical image datasets, enabling researchers to conduct broader studies and analyses.

[0124] Treatment planning: In some cases, physicians may require multimodal information to develop a patient's treatment plan. Synthesizing missing modalities can provide physicians with complete information, helping to better plan treatment strategies.

[0125] Performance Improvement: Performance analysis using metrics such as PSNR, SSIM, and NMSE demonstrates a significant performance improvement in this embodiment. For example, in synthesizing the T2 mode from T1 and FLAIR modes, the PSNR is 0.47 higher, the SSIM is 0.014 higher, and the NMSE is 0.015 lower than the previous state-of-the-art I2I-Mamba technology, showing superior performance across all aspects.

[0126] Example 2 This embodiment discloses a system for synthesizing cross-modal magnetic resonance images, including: The encoding module is used to encode multimodal images through a multi-layer encoder to obtain multi-level encoded features; The residual hybrid Mamba module is used to sequentially pass multi-level encoded features through multiple residual hybrid Mamba modules for feature extraction, obtaining corresponding optimized features. Each residual hybrid Mamba module includes a selective spatial modeling layer, a channel compression layer, and a residual convolutional layer connected in sequence. The multi-level encoded features are input into the first residual hybrid Mamba module for feature extraction. Specifically, the multi-level encoded features are input into the selective spatial modeling layer for feature association, obtaining spatially enhanced attention features; the spatially enhanced attention features are input into the channel compression layer for channel compression, obtaining channel-compressed features; and the channel-compressed features are input into the residual convolutional layer for feature aggregation, obtaining the first optimized feature. The multi-feature extraction module is used to input each optimized feature into the multi-feature extraction module for feature aggregation, thereby obtaining multi-scale aggregated features; The modulation fusion module is used to perform multi-level feature fusion on multi-scale aggregated features based on multiple modulation fusion modules; The implicit neural representation module is used to map fused features to pixel values ​​using a multilayer perceptron and combine them to generate a target modality image.

[0127] Based on a synthesis system for cross-modal magnetic resonance images, the method steps in Embodiment 1 are implemented. Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0128] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method of synthesis across modalities of magnetic resonance images, characterized in that, The method comprises the following steps: encoding the multi-modal image through a multi-layer encoder to obtain multi-level encoding features; extracting features of the multi-level encoding features through a plurality of residual mixed Mamba modules in sequence to obtain corresponding optimization features; inputting each optimization feature into a multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features; performing multi-level feature fusion on the multi-scale aggregated features based on a plurality of modulation fusion modules; mapping the fused features to pixel values by using a multi-layer perception through an implicit neural representation module, and combining to generate a target modal image; wherein each residual mixed Mamba module comprises a selective spatial modeling layer, a channel compression layer and a residual convolution layer connected in sequence, the multi-level encoding features are input into the first residual mixed Mamba module for feature extraction, and the specific process is as follows: inputting the multi-level encoding features into the selective spatial modeling layer for feature association to obtain spatially enhanced attention features; inputting the spatially enhanced attention features into the channel compression layer for channel compression to obtain channel-compressed features; inputting the channel-compressed features into the residual convolution layer for feature aggregation to obtain the first optimization feature.

2. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, extracting features of the multi-level encoding features through a plurality of residual mixed Mamba modules in sequence to obtain corresponding optimization features, and the specific process is as follows: inputting the multi-level encoding features into the first residual mixed Mamba module for feature extraction to obtain the first optimization feature; inputting the first optimization feature into a residual convolution group to extract and fuse features through three consecutive convolution residual blocks to obtain convolution group fused features; inputting the convolution group fused features into the second residual mixed Mamba module for feature extraction to obtain the second optimization feature; inputting the second optimization feature into a residual convolution block for feature extraction and fusion to obtain convolution block fused features; inputting the convolution block fused features into the third residual mixed Mamba module for feature extraction to obtain the third optimization feature.

3. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, inputting the multi-level encoding features into the selective spatial modeling layer for feature association to obtain spatially enhanced attention features, and the specific process is as follows: dividing the multi-level encoding features into a plurality of non-overlapping blocks of a set size; performing layer normalization and activation function processing on the non-overlapping blocks in sequence to obtain first transformed features; inputting the non-overlapping blocks into a linear layer, a deep convolution layer and an activation function layer in sequence for linear transformation and feature extraction to obtain transformed deep features; inputting the transformed deep features into a selective state space modeling unit for feature association to obtain spatially associated features; element-wise multiplying the first transformed features and the spatially associated features to obtain spatially enhanced attention features.

4. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, inputting the spatially enhanced attention features into the channel compression layer for channel compression to obtain channel-compressed features, wherein the channel compression layer is composed of two series of convolution layers and a channel compression unit in parallel, and the specific process is as follows: performing layer normalization on the spatially enhanced attention features, and then element-wise adding the fusion features to the non-overlapping block features to obtain fused features; inputting the fused features into the channel compression unit for channel compression to obtain preliminary compressed features; The fused features are input into two continuous convolution layers for feature extraction, and the extracted features are element-wise added to the preliminary compressed features to obtain channel-compressed features.

5. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, The channel-compressed features are input into a residual convolution layer for feature aggregation to obtain first optimized features, wherein the residual convolution layer includes multiple groups of convolution streams and residual connections, and the specific process is as follows: The channel-compressed features are input into the first group of convolution streams, and the extracted features are subjected to batch normalization and activation function processing to obtain first refined features. The first refined features and the channel-compressed features are fused through the residual connection to obtain first residual fusion features. The first residual fusion features are input into the second group of convolution streams, and the extracted features are subjected to batch normalization and activation function processing to obtain second refined features. The second refined features and the channel-compressed features are fused through the residual connection to obtain the first optimized features.

6. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, Each optimized feature is input into a multi-feature extraction module for feature aggregation to obtain multi-scale aggregated features, wherein the multi-feature extraction module includes a concatenation layer, a convolution layer, a global average pooling layer, a global maximum pooling layer, four parallel convolution down-sampling layers, and a multi-scale selection fusion unit, and the specific process is as follows: The integrated features are input into the four parallel convolution down-sampling layers, and different convolution sampling rates are used for down-sampling to obtain different scale features of the corresponding layers. The different scale features are input into the multi-scale selection fusion unit for fusion to obtain cross-scale fusion features. The cross-scale fusion features and the integrated features are element-wise added through the residual connection to obtain multi-scale aggregated features. The multi-scale features are input into the multi-scale selection fusion unit for fusion to obtain cross-scale fusion features, wherein the multi-scale selection fusion unit sequentially includes a global pooling layer, a concatenation layer, a multi-layer perceptron, an activation function, a normalization layer, and a residual connection, and the specific process is as follows:

7. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 6, wherein, The multi-scale features are input into the global average pooling layer and the global maximum pooling layer for pooling processing to obtain global average pooling features and global maximum pooling features. The global average pooling features and the global maximum pooling features are concatenated through the concatenation layer to obtain p concatenated and pooled features. The concatenated and pooled features are input into the multi-layer perceptron to obtain attention weights. The attention weights are sequentially subjected to activation function and normalization processing to obtain attention weight coefficients. Based on the attention weight coefficients, the different scale features of the corresponding layers are weighted and fused to obtain corresponding fused multi-scale features. The fused multi-scale features are element-wise added through the residual connection to obtain cross-scale fusion features. Based on the multiple modulation fusion modules, multi-level feature fusion is performed on the multi-scale aggregated features, and the specific process is as follows:

8. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, The multi-scale aggregated features are input into the corresponding three convolution layers to obtain first adaptive features, second adaptive features, and third adaptive features. The third optimized feature is subjected to convolution operation to obtain first decoding features. ​ The first adaptation feature and the first decoder feature are input into the first modulation and fusion module for feature modulation and fusion to obtain the first set of fused features; Perform a convolution operation on the first set of fused features to obtain the second decoded features; The second adaptation feature and the second decoder feature are input into the second modulation and fusion module for feature modulation and fusion to obtain the second set of fused features; The second set of fused features is convolved to obtain the third decoded feature; The third adaptation feature and the third decoder feature are input into the third modulation and fusion module for feature modulation and fusion to obtain the third set of fused features, that is, multi-level feature fusion.

9. The method of synthesizing cross-modality magnetic resonance images as claimed in claim 1, wherein, Through an implicit neural representation module, a multilayer perceptron is used to map fused features into pixel values ​​and combine them to generate the target modality image. The specific process is as follows: Based on the fused features, a set of relative coordinates is generated and mapped to a high-dimensional space. In high-dimensional space, the corresponding positional code is obtained through a spatial coding function; The fused features and their corresponding location codes are combined and input into the multilayer perceptron decoder to obtain the final target point value. Based on the final target point values, a synthesized target modal image is obtained.

10. A system for synthesizing across modalities of magnetic resonance images, characterized in that, include: The encoding module is used to encode multimodal images through a multi-layer encoder to obtain multi-level encoded features; The residual hybrid Mamba module is used to sequentially pass multi-level encoded features through multiple residual hybrid Mamba modules for feature extraction, obtaining corresponding optimized features. Each residual hybrid Mamba module includes a selective spatial modeling layer, a channel compression layer, and a residual convolutional layer connected in sequence. The multi-level encoded features are input into the first residual hybrid Mamba module for feature extraction. Specifically, the multi-level encoded features are input into the selective spatial modeling layer for feature association, obtaining spatially enhanced attention features; the spatially enhanced attention features are input into the channel compression layer for channel compression, obtaining channel-compressed features; and the channel-compressed features are input into the residual convolutional layer for feature aggregation, obtaining the first optimized feature. The multi-feature extraction module is used to input each optimized feature into the multi-feature extraction module for feature aggregation, thereby obtaining multi-scale aggregated features; The modulation fusion module is used to perform multi-level feature fusion on multi-scale aggregated features based on multiple modulation fusion modules; The implicit neural representation module is used to map fused features to pixel values ​​using a multilayer perceptron and combine them to generate a target modality image.

Images