Optimized image synthesis method and system for multi-mode magnetic resonance imaging

Abstract

The invention provides an optimized image synthesis method and system for multi-mode magnetic resonance imaging, and belongs to the technical field of image synthesis, and the method comprises the steps: constructing a multi-mode image synthesis model, and extracting a first feature from each mode of multi-mode magnetic resonance imaging data; extracting context information from the first feature to obtain a second feature; extracting information of different scales in the image from the second feature to obtain a first half-stage feature and a second half-stage feature; connecting the front-half-stage features to the rear-half-stage features in a jumping manner, and integrating to obtain multi-scale features; cascading and fusing the multi-scale features of each mode to obtain a fused feature; and synthesizing a target modal image according to the fusion features. According to the method, more complete image data are obtained through the optimized image synthesis method, and accurate clinical diagnosis and scientific research are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image synthesis, and in particular relates to an optimized image synthesis method and system for multi-modal magnetic resonance imaging. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Magnetic resonance imaging (MRI) is a non-invasive, radiation-free imaging technology widely used in medical diagnosis. It can provide high-resolution soft tissue images, especially in the imaging of soft tissues such as the brain, spine, muscles and joints. However, despite the important position of MRI in the medical field, there are still many challenges. First, MRI equipment is expensive and has high maintenance costs, which limits its popularity in some medical institutions, especially in resource-poor areas. Second, MRI scans often take a long time, which may cause discomfort to some patients, especially those who cannot remain still, and thus affect the image quality. In addition, patients with metal implants or medical devices may encounter safety risks when undergoing MRI scans. Furthermore, due to noise interference, motion artifacts and limitations in imaging resolution, MRI images may be incomplete or of poor quality in some clinical applications, affecting the accuracy of diagnosis.

[0004] The prior art discloses a Hi-Net network, which includes a modality-specific network, a multimodal fusion network, and a multimodal synthesis network. Among them, the modality-specific network is used to learn the feature representation of each modality, and the multimodal fusion network adaptively fuses the features of different modalities through a hybrid fusion block, and finally generates the target modality image through a generative adversarial network (GAN). However, there are several problems with the Hi-Net network in practical applications. First, when fusing information from different modalities, the detailed features of a specific modality may be lost, which affects the accuracy of the synthesized image. Secondly, the training process of GAN is prone to instability, resulting in the generated images being not diverse enough or unrealistic, thus affecting the quality of the image.

[0005] The prior art discloses the CACR-Net model, which mainly includes a confidence-guided aggregation module (CGA) and a cross-modal refinement module (CMR). CGA adaptively aggregates the target modality image generated by multiple source modalities through confidence maps, while CMR further mines the correlation information between the multimodal input and the target modality image to optimize the target modality image. Although the model performs well in multimodal MR image synthesis, there are still some potential problems. First, the accuracy of the confidence map is crucial to the quality of the target image. If the confidence map cannot accurately reflect the similarity between the generated image and the real image, it may lead to the loss of image details and structural integrity. Secondly, when the cross-modal refinement module fuses multimodal features, if the fusion is insufficient, it may cause the synthesized image to lack details or have incomplete structures in some areas, affecting the diagnostic value of the image.

[0006] The prior art discloses the ResViT model, which combines an encoder, an information bottleneck, and a decoder, and uses convolutional layers to capture local features. The information bottleneck contains an aggregated residual Transformer module to combine the advantages of Transformer and CNN to achieve high-quality medical image synthesis. Although the model performs well in local feature extraction and global context modeling, it also has several obvious problems. First, CNN performs poorly in dealing with long-range spatial dependencies, which may cause defects in the global structure and context consistency of the synthesized image, and the computational burden is heavy. Secondly, although the Transformer can capture long-range contextual information, its positioning ability is weak, which may result in inaccurate details. Summary of the invention

[0007] In order to overcome the deficiencies of the above-mentioned prior art, the present invention proposes an optimized image synthesis method and system for multimodal magnetic resonance imaging, designs a modality-specific encoder to process the data of each modality separately, retains the features of different modalities, and avoids detail loss; adopts the ViT backbone and Mamba module to solve the problem of CNN processing long-distance dependency differences and reduce the computational burden. In addition, the dual-scale feature fuser can better capture information of different scales, enhance the expression ability of the network, and effectively reduce the problem of detail loss. Finally, by optimizing the feature aggregation module and the dual-stream fusion module, the present invention indirectly enhances the stability of the training process, improves the quality of the generated image, and avoids the problem of unstable GAN training.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, a method for optimizing image synthesis of multimodal magnetic resonance imaging is disclosed, comprising: Acquiring multimodal magnetic resonance imaging data; Construct a multimodal image synthesis model and synthesize multimodal magnetic resonance imaging data to obtain the target modality image; Among them, the multimodal image synthesis model extracts the first feature of each modality of the multimodal magnetic resonance imaging data; extracts context information from the first feature to obtain the second feature; extracts information of different scales in the image from the second feature to obtain the first half stage feature and the second half stage feature; jumps the first half stage feature to the second half stage feature to obtain multi-scale features; cascades and fuses the multi-scale features of each modality to obtain fused features; and synthesizes the target modality image based on the fused features.

[0009] In a second aspect, an optimized image synthesis system for multimodal magnetic resonance imaging is disclosed, comprising: A data acquisition module, configured to: acquire multimodal magnetic resonance imaging data; A multimodal image synthesis module is configured to: construct a multimodal image synthesis model, and synthesize multimodal magnetic resonance imaging data to obtain a target modality image; Among them, the multimodal image synthesis model extracts the first feature of each modality of the multimodal magnetic resonance imaging data; extracts context information from the first feature to obtain the second feature; extracts information of different scales in the image from the second feature to obtain the first half stage feature and the second half stage feature; jumps the first half stage feature to the second half stage feature to obtain multi-scale features; cascades and fuses the multi-scale features of each modality to obtain fused features; and synthesizes the target modality image based on the fused features.

[0010] In a third aspect, an electronic device is disclosed, comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the above-mentioned method for optimizing image synthesis of multimodal magnetic resonance imaging are completed.

[0011] In a fourth aspect, a computer-readable storage medium is disclosed for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned method for optimizing image synthesis of multimodal magnetic resonance imaging.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The proposed method uses a specific encoder for each modality, which can preserve the characteristics of each modality and avoid losing details during the fusion process. Compared with the traditional convolutional neural network (CNN), the Transformer model better captures the high-level information of each modality through global context perception, providing accurate and complete input representation for subsequent multimodal image synthesis.

[0013] The SSM layer proposed in this paper enhances the global understanding of the image by effectively capturing long-distance contextual information, making up for the shortcomings of traditional CNN in long-distance dependency modeling. This enables the network to better maintain the overall structure and consistency of the image during the synthesis process, reducing the loss of details in the image, which is especially important for complex anatomical structures.

[0014] Through the fusion of multi-scale features, the present invention's Dual-Fuser can better process detailed information of different scales in the image and improve the ability to express details and structures. The high-resolution branch can more accurately capture the key information of a specific modality through more downsampling stages, while the low-resolution branch can better capture global features, and ultimately retain more details of a specific modality through jump connections, effectively improving the quality and diagnostic value of the image.

[0015] The present invention introduces a dual-scale feature fuser, which is similar to the U-Net structure, contains multiple downsampling stages, and uses the Swin Transformer layer for feature fusion. The output feature map is bilinearly interpolated to form a high-resolution image, and then multi-scale feature fusion is performed.

[0016] The dual-stream structure proposed in the present invention can effectively process the spatial information and channel information of the feature map respectively, suppress less useful features, and retain more effective information. The spatial stream can enhance the expression of spatial features by calculating the spatial attention map; while the channel stream captures the relationship between channels through global average pooling, so that the network can more accurately select features and suppress irrelevant features. This structure improves the efficiency of feature fusion, making the final synthesized image more realistic and rich in details.

[0017] The FA module proposed in the present invention effectively fuses the features of multimodal images through global and local attention mechanisms, enhancing the complementary information between different modalities. Global attention enables the network to focus on the global features of the image, while local attention helps extract the details. This fusion method improves the quality of the image, making the synthesized image have high accuracy and consistency in both overall structure and detail presentation.

[0018] The present invention optimizes resource utilization: by synthesizing missing modalities, it effectively reduces the reliance on additional MRI scans, saves scanning time and equipment costs, and thus enables more efficient utilization of medical resources.

[0019] The present invention enhances image usability: In traditional methods, missing modalities may affect the comprehensiveness of clinical judgment and research. By synthesizing multimodal images, doctors can obtain more complete image data, which helps to make more accurate clinical diagnosis and scientific research.

[0020] The present invention improves diagnostic efficiency: This method reduces the risk of misdiagnosis caused by image noise and artifacts by improving image quality. Higher quality synthetic images help doctors make decisions quickly in complex conditions and improve diagnostic efficiency.

[0021] The present invention reduces the burden on patients: for some patients who cannot remain still for a long time, or who cannot undergo a comprehensive scan due to medical conditions, the image synthesis technology of the present invention can provide high-quality image support without increasing the burden on patients, thereby ensuring the accuracy of the diagnosis results.

[0022] The present invention promotes the expansion of medical data: the synthesized multimodal images can provide more sample data for medical research and education, promote the expansion of data sets, help carry out more extensive clinical experiments and scientific research work, and promote the further development of the field of medical imaging.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is an overall framework diagram of the optimized image synthesis method for multimodal magnetic resonance imaging described in Example 1 of the present invention.

[0026] Figure 2 This is a schematic diagram of the dual-stream fusion module described in Embodiment 1 of the present invention.

[0027] Figure 3 This is the visualization result of the T1 and T2 synthesis FLAIR task performed by the multimodal image synthesis model described in Example 1 of the present invention.

[0028] Figure 4 This is the visualization result of the T1 and FLAIR synthesis T2 task performed by the multimodal image synthesis model described in Example 1 of the present invention.

[0029] Figure 5 This is the visualization result of the T2 and FLAIR synthesis T1 task performed by the multimodal image synthesis model described in Example 1 of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0032] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0033] Embodiment 1 In one or more embodiments, a method for optimizing image synthesis of multimodal magnetic resonance imaging is disclosed, comprising the following steps: Step 1: Obtain magnetic resonance imaging (MRI) of different modalities, including MRI T1, MRI T2, and MRI FLAIR. When the FLAIR modality is missing, obtain T1 and T2 modalities as input; when the T1 modality is missing, obtain T2 and FLAIR modalities as input; when the T2 modality is missing, obtain 1 and FLAIR modalities as input, so as to perform the synthesis task of the missing modality.

[0034] Step 2: Build a multimodal image synthesis model, such as Figure 1 As shown in the figure, the model includes an encoder, a Mamba module, a dual-scale feature fuser, a dual-stream fusion module and a feature aggregation module connected in sequence. Specifically: Step 2-1: construct an encoder, and use the encoder to extract the first feature of each mode of the multi-modal magnetic resonance image.

[0035] In order to maintain the integrity of different modality data, a modality-specific encoder is designed for each input modality. The specific encoder first assigns each input modality image to a separate branch. In this embodiment, the modality-specific encoder includes two branches, each of which uses the ViT (Vision Transformer) backbone architecture to process the image.

[0036] Following the ViT method, the input is first split into a series of non-overlapping patches, each of which is treated as a sequence and converted to a vector through an embedding layer to be consistent with the input specification of the Transformer. The vector sequence is obtained by an ordered sorting set of multiple vectors. This serialization process enables the present invention to use the Transformer model to process image data, overcoming the limitations of traditional convolutional neural networks.

[0037] These vector sequences are then fed into the Transformer encoder for additional feature extraction and processing. This modality-specific encoder structure effectively captures the features of different modality data and maintains their integrity, providing more accurate input representations for subsequent tasks.

[0038] Among them, the main difference between the specific encoder and the general encoder lies in its design purpose and processing method. The general encoder usually processes all input data through a single structure, while the specific encoder designs independent branches for each input modality (such as T1, T2, FLAIR). Each branch specializes in processing images of one modality to ensure that the features of each modality can be effectively extracted and retained. The encoder is designed to have a two-branch structure, each of which is composed of a ViT backbone, which can effectively process data of different modalities. The input data of each modality will be serialized and feature extracted through the Transformer encoder.

[0039] Step 2-2, constructing a Mamba module, and extracting context information from the first feature through the Mamba module to obtain a second feature; The Mamba block contains an SSM layer to capture long-range contextual information. The Mamba block includes two branches. First, the input feature map is split into The size is different blocks to generate the sequence Enter the two branches separately; (1) in, is the number of blocks; and are the height and width of the input feature map respectively; It is the size of each small block after the feature map is divided into small blocks.

[0040] In the first branch, the sequence is linearly embedded to calculate the gating variable as shown below: (2) in, is the gating variable; represents the activation function; is a linear transformation. In the second branch, the sequence undergoes linear embedding, processed using depthwise convolution, and then projected through the SSM layer, as shown below: (3) in, It is the feature representation after processing by linear transformation, depthwise separable convolution and state-space model; It is a depth-wise separable convolution; It is the SSM layer.

[0041] In the above formula, the SSM layer is implemented based on the selective state space sequence model. Initially, the patch sequence is expanded by performing selective scanning in various directions on the 2D feature map. The expanded sequence is then processed by state space modeling and merged back together. In this process, a discrete state space model is established.

[0042] (4) (5) in, Indicates the hidden state, represents the input sequence, represents the output sequence, represents the sequence index, , , , are learnable SSM parameters, and , , as well as ,in represents the dimension of the state space. Finally, it is linearly projected and combined with the residual of the input sequence by multiplication: (6) in, Output features for the Mamba module, is a linear transformation, It is a pixel-by-pixel multiplication operation.

[0043] The Mamba module contains a selective state-space model (SSM) layer that expands the sequence by selectively scanning the 2D feature map and processes the expanded sequence using state-space modeling.

[0044] Step 2-3: Construct a dual-scale feature fuser to extract information of different scales in the image and obtain the first half stage features and the second half stage features.

[0045] In order to better capture the information of different scales in the image and enhance the understanding and representation ability of the model, this paper introduces an innovative dual-scale feature fuser (Dual-Fuser), which aims to make full use of the output of the Mamba module to build the powerful functions of a specific modality. Figure 1 (b) shown.

[0046] First, the Mamba module outputs features as low-resolution feature maps ,at the same time Apply bilinear interpolation to generate higher resolution feature maps Next, the low-resolution feature map and high-resolution feature maps It is fed into separate branches of the dual-scale fuser, each of which has several downsampling stages. In each stage, a SwinTransformer layer and a patch merging unit are applied to reduce the resolution of the feature map while increasing its dimensionality to form a hierarchical feature representation.

[0047] It is worth noting that the high-resolution branch has more downsampling stages than the low-resolution branch, which enhances the network's ability to capture features. In order to retain the key details of a specific modality in the final output, the present invention uses patch expansion operations and Swin Transformer layers in the later stages to enhance feature resolution. In the dual-scale feature fuser, the downsampling stage is the first half of the dual-scale fuser, while the second half is upsampling. By introducing an innovative twin-stream fusion module (TSF), jump connections are used to retain more multi-scale information of a specific modality. This design allows the Dual-Fuser to capture the relationship between different scales more flexibly and effectively during feature fusion.

[0048] Step 2-4: Construct a dual-stream fusion module (TSF), through which the first half of the dual-scale feature fuser’s features are jump-connected to the second half of the features to integrate multi-scale features; In order to better fuse the features extracted in the first half of the Dual-Fuser, a dual-stream fusion (TSF) module is proposed. The feature map extracted by the first half of the dual-scale feature fuser after multiple Swin transformers and block merging is used as the first half stage features, and the feature information extracted by the second half of the dual-scale feature fuser after multiple Swin transformers and block expansion is used as the second half stage features. The first half stage features are fused into the second half stage features through the dual-stream fusion module to effectively integrate multi-scale features.

[0049] As an implementation method, this embodiment uses Swin Transformer combined with block merging and block expansion. The low-resolution feature map and the high-resolution feature map are respectively input into the Swin Transformer and block merging branches, and sequentially pass through multiple Swin Transformer and block merging combination modules, which are referred to as the first half of the dual-scale feature fuser. The second half of the dual-scale feature fuser is composed of multiple Swin Transformers and block expansions. The output of the Swin Transformer and block merging passes through the TSF module, is spliced ​​with the output of the second half of the dual-scale feature fuser, the Swin Transformer and block expansion, and is input into the next Swin Transformer and block expansion module, realizing the jump connection between the first half and the second half.

[0050] The TSF module consists of two special data streams, namely the spatial stream and the channel stream. Figure 2 As shown, the multi-scale features extracted in the first half are expressed as and The concatenated features are input into the point-by-point convolution operation to generate a multi-scale feature map Then, it enters the spatial stream and channel stream stages respectively. The TSF module is designed to suppress less useful features and only allow more informative features to pass through.

[0051] (1) Channel Flow (CF): This branch uses compression and excitation operations to exploit the relationship between channels in the convolutional feature map. Initially, global average pooling is performed on the spatial dimension to capture the global context, producing a feature descriptor Then it goes through two convolutional layers and an activation function Sigmoid during the excitation process. The CF branch output is obtained by using the activation value Rescale the original feature map And obtained.

[0052] (2) Spatial Stream (SF): This branch is designed to exploit the spatial relationships within the convolutional features. The purpose of SF is to create a recalibrated input feature map To construct the spatial attention map, the SF branch first follows the feature map Global average pooling and maximum pooling are performed on the channel dimension of , and then the results are combined to form a new feature map Mapping Then through the convolutional layer, and then activation, resulting in a spatial attention map This map is then used to adjust the feature map , generating the output of the SF branch.

[0053] The outputs of the channel stream branch and the spatial stream branch are respectively compared with the multi-scale feature map After performing the multiplication operation, the concatenation is performed and the output of the two-stream fusion module (TSF) is obtained through pixel-by-pixel convolution.

[0054] Step 2-5: construct a feature aggregation module (FA), through which the multi-scale features of each modality are cascaded and fused to obtain fused features; Although the two-stream fusion module (TSF) has successfully integrated the multi-resolution information of a specific modality, a mechanism is still needed to manage the multi-scale information between different modalities. The FA module is specifically as follows: first, the multi-scale features from different modalities are fused by the dual fuser module and integrated through cascade. Then, the FA module is divided into two branches: one branch fuses features from a global attention perspective, and the other fuses features from a local attention perspective. Both branches undergo pixel-by-pixel convolution processing. In the last step, the features from the two branches are combined again and multiplied pixel-by-pixel with the initial features to produce the final fused features.

[0055] The corresponding formula is as follows: (7) (8) in, represents the final fused features, represents the intermediate features before entering the activation function, GA represents the global branch of the FA module, and LA represents the local branch of the FA module. and Represents multi-scale features of different modalities, is the activation function, is a pixel-by-pixel addition operation. In this way, the FA module effectively integrates features from different modalities while retaining the importance of multi-scale information. This design enables the model of the present invention to better understand and process the complex relationship between different modal data, thereby improving the performance of the model in various tasks.

[0056] The FA module effectively fuses the features of multimodal images through global and local attention mechanisms, enhancing the complementary information between different modalities. Global attention enables the network to focus on the global features of the image, while local attention helps extract the details. This fusion method improves the quality of the image, making the synthesized image have high accuracy and consistency in both overall structure and detail presentation.

[0057] Step 2-6: The output of the feature aggregation module passes through the Mamba module to generate the target modality image.

[0058] Step 3: Input the acquired multimodal image into the constructed multimodal image synthesis model to obtain the target modality.

[0059] The optimized image synthesis method of multimodal MRI provided by the present invention effectively compensates for the problem of image loss or quality degradation caused by factors such as scanning limitations, insufficient time or patient discomfort by fusing MRI image information of different modalities. Compared with traditional methods, this image synthesis method not only avoids the time and economic cost of additional scanning, but also can significantly improve the integrity and quality of the image, thereby effectively improving the application effect and accuracy of MRI in clinical diagnosis.

[0060] Embodiment 2 In one or more embodiments, a system for optimizing image synthesis of multimodal magnetic resonance imaging is disclosed, specifically comprising: A data acquisition module, configured to: acquire multimodal magnetic resonance imaging data; A multimodal image synthesis module is configured to: construct a multimodal image synthesis model, and synthesize multimodal magnetic resonance imaging data to obtain a target modality image; Among them, the multimodal image synthesis model extracts the first feature of each modality of the multimodal magnetic resonance imaging data; extracts context information from the first feature to obtain the second feature; extracts information of different scales in the image from the second feature to obtain the first half stage feature and the second half stage feature; jumps the first half stage feature to the second half stage feature to obtain multi-scale features; cascades and fuses the multi-scale features of each modality to obtain fused features; and synthesizes the target modality image based on the fused features.

[0061] Embodiment 3 This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned method for optimizing image synthesis of multi-modal magnetic resonance imaging are completed.

[0062] Embodiment 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for optimizing image synthesis of multi-modality magnetic resonance imaging are completed.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing image synthesis of multimodal magnetic resonance imaging, characterized in that: include: Acquiring multimodal magnetic resonance imaging data; Construct a multimodal image synthesis model and synthesize multimodal magnetic resonance imaging data to obtain the target modality image; Among them, the multimodal image synthesis model extracts the first feature of each mode of the multimodal magnetic resonance imaging data; extracts context information from the first feature to obtain the second feature; extracts information of different scales in the image from the second feature to obtain the first half stage feature and the second half stage feature; jump connects the first half stage feature to the second half stage feature integration to obtain the multi-scale feature; cascades and fuses the multi-scale features of each mode to obtain the fusion feature; The target modality image is synthesized based on the fused features.

2. The method for optimizing image synthesis of multimodal magnetic resonance imaging according to claim 1, characterized in that: The multimodal image synthesis model includes: an encoder, a Mamba module and a dual-scale feature fuser; The encoder includes two branches, each of which inputs magnetic resonance imaging data of a modality and outputs the first features of each modality; the first features of each modality are respectively subjected to the Mamba module to extract context information to obtain the second features of each modality; the second features of each modality and the feature maps obtained by interpolating the second features are respectively input into separate branches of the dual-scale feature fuser to extract the first half stage features and the second half stage features.

3. The method for optimizing image synthesis of multimodal magnetic resonance imaging according to claim 2, characterized in that: The Mamba module includes two branches. In the first branch, the sequence generated by the segmentation of the input feature map is linearly embedded to calculate the gated variables; in the second branch, the sequence undergoes linear embedding, is processed using deep convolution, and then projected through the SSM layer.

4. The method for optimizing image synthesis of multimodal magnetic resonance imaging according to claim 1, characterized in that: The multimodal image synthesis model also includes: a dual-stream fusion module; The dual-stream fusion module consists of two data streams, namely a channel stream and a spatial stream. The input of the dual-stream fusion module is the first half stage features and the second half stage features. The first half stage features and the second half stage features are connected and the multi-scale feature maps obtained by point-by-point convolution operation enter the spatial stream and the channel stream respectively. The outputs of the spatial stream and the channel stream are connected and convolved to obtain multi-scale features.

5. The method for optimizing image synthesis of multimodal magnetic resonance imaging according to claim 4, characterized in that: The spatial stream performs global average pooling and maximum pooling on the multi-scale feature map along the channel dimension, combines the pooling results to form a new feature map, and then passes through a convolution layer and an activation function to obtain a spatial attention map.

6. The method for optimizing image synthesis of multimodal magnetic resonance imaging according to claim 4, characterized in that: The channel flow performs global average pooling on the multi-scale feature map in the spatial dimension to generate a feature descriptor, which then undergoes two convolutional layers and an activation function.

7. The method for optimizing image synthesis of multimodal magnetic resonance imaging according to claim 1, characterized in that: The multimodal image synthesis model further includes: a feature aggregation module; The feature aggregation module cascades and integrates the multi-scale features of each modality, and fuses the features from the global attention perspective and the local attention perspective respectively to obtain the global attention feature and the local attention feature, and then combines the global attention feature and the local attention feature again and multiplies them pixel by pixel with the multi-scale features of different modalities to generate the final fused feature.

8. An optimized image synthesis system for multimodal magnetic resonance imaging, characterized in that: include: A data acquisition module, configured to: acquire multimodal magnetic resonance imaging data; A multimodal image synthesis module is configured to: construct a multimodal image synthesis model, and synthesize multimodal magnetic resonance imaging data to obtain a target modality image; Among them, the multimodal image synthesis model extracts the first feature of each mode of the multimodal magnetic resonance imaging data; extracts context information from the first feature to obtain the second feature; extracts information of different scales in the image from the second feature to obtain the first half stage feature and the second half stage feature; jump connects the first half stage feature to the second half stage feature integration to obtain the multi-scale feature; cascades and fuses the multi-scale features of each mode to obtain the fusion feature; The target modality image is synthesized based on the fused features.

9. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the optimized image synthesis method for multi-modal magnetic resonance imaging according to any one of claims 1 to 7 is completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the optimized image synthesis method for multimodal magnetic resonance imaging as described in any one of claims 1-7.

Images