MRI (Magnetic Resonance Imaging) missing mode generation method based on learnable frequency domain module

Abstract

The invention provides an MRI (Magnetic Resonance Imaging) missing mode generation method based on a learnable frequency domain module, which is used for reconstructing and synthesizing any missing mode of a brain glioma medical multi-mode image. The method comprises the steps that a training set and a test set of brain tumor multi-mode MRI images are acquired, the training set and the test set comprise a plurality of samples, each sample comprises four original mode images, and each original mode image is an available mode or a missing mode; a missing modal brain tumor generation network model is constructed, and the missing modal brain tumor generation network model comprises a learnable frequency domain module and a condition diffusion module based on modal attention; inputting the training set into the model for training, setting a joint loss function, and optimizing parameters of the model through the joint loss function of each sample to obtain an optimized model; and inputting the multi-mode MRI image in the test set into the optimized model to obtain a generated image corresponding to the missing mode.

Description

Technical Field

[0001] The present application relates to the field of digital image processing, and in particular to a method, system, device and storage medium for generating MRI missing modalities based on a learnable frequency domain module. Background Art

[0002] In medical imaging, multimodal images often provide more comprehensive information about diseased tissue, thereby improving the accuracy of disease diagnosis and the effectiveness of treatment plans. In particular, in the diagnosis of brain tumors, multiple MRI modalities (T1, T1ce, T2, and FLAIR) are crucial for determining tumor location and properties.

[0003] However, the lack of modalities across different clinical centers is unavoidable. For example, modalities are often missing due to limitations such as scan time, scan corruption, artifacts, motion, and contrast intolerance. These factors limit the ubiquity of multimodality and affect various downstream tasks such as segmentation, detection, and quantification. The problem of missing modalities significantly reduces the diagnostic performance of multimodal imaging and affects subsequent tumor segmentation and treatment planning. Therefore, synthesizing missing multimodal MRI has become a hot research issue as a means to overcome the limitations of insufficient modalities in clinical practice and research.

[0004] Traditional image segmentation and detection methods typically rely on complete multimodal data. Their accuracy degrades significantly when modal data is missing. With the development of deep learning, numerous deep learning-based models have been proposed, but most methods only support one-to-one or many-to-one model training, making them inadequate for real-world applications where modal data is randomly missing. While methods based on generative adversarial networks can achieve many-to-many model training, they suffer from modal collapse, non-convergence, instability, and high sensitivity to hyperparameters.

[0005] Therefore, studying a unified missing modality generation method for MRI images is of great significance for improving the performance of downstream tumor diagnosis tasks such as segmentation, detection and quantification, as well as the quality of subsequent treatment. Summary of the Invention

[0006] The embodiments of the present application provide a method, system, device and storage medium for generating missing modalities of MRI based on a learnable frequency domain module, which can realize the reconstruction and synthesis of any missing modalities of multimodal medical images of glioma.

[0007] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application provides a method for generating MRI missing modalities based on a learnable frequency domain module, the method comprising: Acquire a training set and a test set of multimodal MRI images of brain tumors, wherein the training set and the test set include multiple samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality; A missing modality brain tumor generation network model is constructed, wherein the missing modality brain tumor generation network model includes a learnable frequency domain module and a conditional diffusion module based on modal attention; the learnable frequency domain module is used to extract the frequency domain information of the available modality in the i-th sample, and perform adaptive separation of high-frequency features and low-frequency features to obtain high-frequency information and low-frequency information of the i-th sample; the conditional diffusion module based on modal attention includes a forward denoising diffusion module and a reverse denoising generation module based on modal attention, the forward denoising diffusion module is used to gradually add Gaussian noise to the missing modality in the i-th sample until T noise images are generated; the reverse denoising generation module based on modal attention is used to gradually denoise the T-th noise image generated by the forward denoising diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T denoised images are generated, wherein T is a positive integer greater than 1, and i is a positive integer greater than or equal to 1; Input the training set into the model for training, set the joint loss function, optimize the model parameters through the joint loss function of each sample, and obtain the optimized model; The multimodal MRI images in the test set are input into the optimized model to obtain the generated images corresponding to the missing modalities.

[0008] In a second aspect, the present application provides an MRI missing modality generation system based on a learnable frequency domain module, the system comprising: a data acquisition module, configured to acquire a training set and a test set of multimodal MRI images of brain tumors, wherein the training set and the test set include a plurality of samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality; A model construction module is used to construct a missing modality brain tumor generation network model, wherein the missing modality brain tumor generation network model includes a learnable frequency domain module and a conditional diffusion module based on modal attention; the learnable frequency domain module is used to extract the frequency domain information of the available modality in the i-th sample, and perform adaptive separation of high-frequency features and low-frequency features to obtain high-frequency information and low-frequency information of the i-th sample; the conditional diffusion module based on modal attention includes a forward denoising diffusion module and a reverse denoising generation module based on modal attention, the forward denoising diffusion module is used to gradually add Gaussian noise to the missing modality in the i-th sample until T noise images are generated; the reverse denoising generation module based on modal attention is used to gradually denoise the T-th noise image generated by the forward denoising diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T denoised images are generated, wherein T is a positive integer greater than 1, and i is a positive integer greater than or equal to 1; The training module is used to input the training set into the model for training and set the joint loss function. The model parameters are optimized through the joint loss function of each sample to obtain the optimized model. The generation module is used to input the multimodal MRI images in the test set into the optimized model to obtain the generated images corresponding to the missing modalities.

[0009] In a third aspect, an apparatus for generating missing modalities of MRI based on a learnable frequency domain module is provided. The apparatus for generating missing modalities of MRI based on a learnable frequency domain module includes a module for executing the method of the first aspect.

[0010] In one possible design, the apparatus for generating MRI missing modalities based on a learnable frequency domain module in the third aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used to enable the apparatus for generating MRI missing modalities based on a learnable frequency domain module in the third aspect to communicate with other devices.

[0011] In one possible design, the MRI missing modality generation apparatus based on a learnable frequency domain module of the third aspect may further include a memory. The memory may be integrated with the processor or provided separately. The memory may be used to store instructions involved in the method of the first aspect.

[0012] In a fourth aspect, a device for generating missing modalities in MRI based on a learnable frequency domain module is provided. The device for generating missing modalities in MRI based on a learnable frequency domain module comprises: a processor coupled to a memory, the processor configured to execute instructions stored in the memory, so that the device for generating missing modalities in MRI based on a learnable frequency domain module performs the method of the first aspect.

[0013] In one possible design, the apparatus for generating MRI missing modalities based on a learnable frequency domain module in the fourth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used to enable the apparatus for generating MRI missing modalities based on a learnable frequency domain module in the fourth aspect to communicate with other devices.

[0014] In a fifth aspect, an MRI missing modality generation device based on a learnable frequency domain module is provided, comprising: a processor and a memory; the memory is used to store instructions, and when the processor executes the instructions, the MRI missing modality generation device based on the learnable frequency domain module performs the method of the first aspect.

[0015] In one possible design, the apparatus for generating MRI missing modalities based on a learnable frequency domain module in the fifth aspect may further include a transceiver. The transceiver may be a transceiver circuit or an interface circuit. The transceiver may be used for the apparatus for generating MRI missing modalities based on a learnable frequency domain module in the fifth aspect to communicate with other devices.

[0016] In a sixth aspect, a computer-readable storage medium is provided, which includes a computer program or instruction stored therein. When the computer program or instruction is executed, the MRI missing modality generation method based on a learnable frequency domain module of the first aspect is executed.

[0017] Based on the above technical solution, this application can achieve the following technical effects: The missing modality brain tumor modality generation network model in this application is guided by the high-frequency information and low-frequency information obtained by the learnable frequency domain module, and the modal attention module introduced in the modal attention-based reverse denoising generation module, to learn to capture the relevant features between different modalities and promote the complementarity of feature information; then, the noisy image obtained by the forward denoising diffusion module and the denoised image obtained by the modal attention-based reverse denoising generation module using high-frequency information and low-frequency information and the relevant features between different modalities are used to optimize the model using the set loss function, thereby improving the accuracy and robustness of the missing modality brain tumor modality generation network model for missing modality generation, and only a single model is required to handle different missing modality combinations.

[0018] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flow chart of a method for generating MRI missing modalities based on a learnable frequency domain module provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of the missing modality brain tumor generation network model provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a learnable frequency domain module in the missing modality brain tumor generation network model provided in an embodiment of the present application; Figure 4 Schematic diagram of the structure of the MRI missing mode generation device based on the learnable frequency domain module provided in the embodiment of the present application Figure 1 ; Figure 5 Schematic diagram of the structure of the MRI missing mode generation device based on the learnable frequency domain module provided in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0022] Figure 1 A flow chart of a method for generating MRI missing modalities based on a learnable frequency domain module provided in an embodiment of the present application.

[0023] The process of the MRI missing modality generation method based on the learnable frequency domain module is as follows: Step S101: Acquire a training set and a test set of multimodal MRI images of brain tumors.

[0024] The training set and the test set include multiple samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality.

[0025] In addition, the detailed steps of obtaining the training set and the test set of the brain tumor multimodal MRI images in step S101 can be understood by referring to steps S201-S204, which are as follows: Step S201: Acquire a brain tumor multimodal MRI image dataset.

[0026] The multimodal MRI image dataset includes multiple samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality.

[0027] In addition, each of the above samples includes four original modal images, namely fluid-attenuated inversion recovery imaging (FLAIR), T1-weighted imaging (T1), T1-weighted contrast-enhanced imaging (T1ce), and T2-weighted imaging (T2). The four original modal images may be available modalities (also referred to as non-missing modalities) or missing modalities, without specific restrictions. In this application, FLAIR and T2 are used as missing modalities, and T1ce and T1 are used modalities as an example. This will not be emphasized later, but in actual application, it will depend on the specific situation. It should also be noted that in this application, FLAIR and T2 are missing modalities in the training set, which are simulated missing modalities after processing by available modalities.

[0028] Step S202 : preprocessing the multimodal MRI image dataset to obtain a two-dimensional multimodal MRI slice image dataset, and dividing the two-dimensional multimodal MRI slice image dataset into a first dataset and a second dataset according to a preset ratio.

[0029] The multimodal MRI image dataset in step S201 is a three-dimensional image, and each sample can be represented as , where C represents the number of modes of the multimodal MRI image. In this application, C is 4. W, H, and D represent the width, height, and number of slices of the multimodal MRI image, respectively. The number of slices can be understood as the number of two-dimensional images obtained after slicing the three-dimensional image.

[0030] In this application, the middle two-dimensional slice is selected from multiple two-dimensional slice images, which is specifically expressed as: ,in, For each sample, the length and width of the two-dimensional slice in the middle of the four modal three-dimensional images are consistent with the three-dimensional image. The specific reason for selecting the middle two-dimensional slice is that tumors usually present a three-dimensional structure. In order to obtain complete tumor slice information, slice sampling is usually selected from the middle of the tumor, because this position can more comprehensively reflect the overall morphology and tissue characteristics of the tumor, avoiding insufficient sample representativeness caused by sampling only the surface or edge parts.

[0031] According to the above processing, a two-dimensional multimodal MRI slice image dataset is obtained, in which each sample in the two-dimensional multimodal MRI slice image dataset includes two-dimensional slice images of four modalities. Then, the two-dimensional multimodal MRI slice image dataset is divided into a first dataset and a second dataset according to a preset ratio.

[0032] Among them, the preset ratio is generally 8:2, but in actual applications, 6:4, 7:3, etc. can also be selected. The specific ratio is determined according to actual conditions and is not specifically limited here.

[0033] Step S203 : performing data enhancement on the two-dimensional multimodal MRI slice images in the first data set to obtain a third data set.

[0034] Data augmentation is a technique that generates new data samples by transforming, amplifying, or perturbing the original data. It can effectively increase the diversity of training data, improve the generalization ability of the model, and prevent overfitting. Common image data augmentation methods include random rotation, random flipping, random cropping, and random noise injection. Random rotation involves randomly rotating the image by a specific angle, typically within a certain range (e.g., -30° to +30°); random flipping involves randomly flipping the image horizontally or vertically; and random cropping involves randomly cropping a region of a specified size from the image. The size and position of the cropped region can be randomly selected, and the cropped image retains its original label. Random noise injection involves randomly adding noise (such as Gaussian noise or salt and pepper noise) to the image, blurring or introducing random interference.

[0035] In step S204, a state code is generated for the data-enhanced two-dimensional multimodal MRI slice image in the third data set to obtain a corresponding state code, and the state code is spliced ​​with the corresponding original modality image in the multimodal MRI image data set to obtain a fourth data set.

[0036] A status code is generated for the data-enhanced two-dimensional multimodal MRI slice image in the third data set to obtain a corresponding status code. Specifically, the data-enhanced two-dimensional multimodal MRI slice image in the third data set is processed by a missing state code random generator to generate a status code containing only "0" and "1", where "0" corresponds to the two-dimensional multimodal MRI slice image being a missing mode, and "1" corresponds to the two-dimensional multimodal MRI slice image being an available mode.

[0037] That is, the missing state code random generator samples binary state variables independently for each modality, satisfying the following conditions: ,i=1,...,N and , Among them, the symbol It means "subject to"; In this application, show There are only two possible values, 0 or 1, and the probability of taking the value 0 or 1 is equal; Indicates the number of modes; , a value of 0 indicates a missing modality, and a value of 1 indicates an available modality.

[0038] For example, in a data-enhanced two-dimensional multimodal MRI slice image in the third dataset, FLAIR and T2 are missing modalities, and T1ce and T1 are available modalities, and the resulting status code is 0110.

[0039] Subsequently, the state code is concatenated with the corresponding original modality image in the multimodal MRI image dataset to obtain the fourth dataset, which can be specifically expressed as: , in, represents the fourth dataset (also known as the random missing modality image set); Represents a multimodal MRI image dataset; is the status code obtained above; {;} indicates the splicing operation.

[0040] In addition, the fourth data set obtained above is a training set of multimodal MRI images of brain tumors, and the second data set obtained above is a test set of multimodal MRI images of brain tumors.

[0041] Step S102: constructing a missing modality brain tumor generation network model, wherein the missing modality brain tumor generation network model includes a learnable frequency domain module and a conditional diffusion module based on modality attention.

[0042] The learnable frequency domain module is used to extract the frequency domain information of the available modes in the i-th sample and perform adaptive separation of high-frequency features and low-frequency features to obtain the high-frequency information and low-frequency information of the i-th sample.

[0043] The conditional diffusion module based on modal attention includes a forward noise addition and diffusion module and a reverse noise reduction generation module based on modal attention. The forward noise addition and diffusion module is used to gradually add Gaussian noise to the missing mode in the i-th sample until T noisy images are generated; the reverse noise reduction generation module based on modal attention is used to gradually reduce the noise of the T-th noise image generated by the forward noise addition and diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T noise-reduced images are generated, where T is a positive integer greater than 1 and i is a positive integer greater than or equal to 1.

[0044] It should be noted that the above-mentioned i-th sample represents any sample, or can be called each sample, and no further details will be given below.

[0045] Missing modality brain tumor generation network model Figure 2 As shown, the frequency domain module can be learned as Figure 3 For details, please refer to Figure 2 、 Figure 3 And the following expressions are understood.

[0046] In an embodiment, a frequency domain module can be learned to extract the frequency domain information of the available mode in the i-th sample, and to perform adaptive separation of high-frequency features and low-frequency features to obtain the high-frequency information and low-frequency information of the i-th sample. For specific implementation, please refer to steps S201 to S206.

[0047] Step S201, perform Fourier transform on the available modes in the i-th sample, transform the spatial domain information into the frequency domain, and obtain the frequency domain information of the available modes in the i-th sample; wherein, if there is more than one available mode in the i-th sample, the available mode with the highest priority is selected for processing according to the priority, and the frequency domain information of the i-th sample is obtained, and the priority is preset.

[0048] In the above example, FLAIR and T2 are missing modalities, and T1ce and T1 are available modalities. As can be seen, when more than one available modality is available for the i-th sample, the available modality with the highest priority is selected for processing to obtain the frequency domain information of the i-th sample. The priority is preset.

[0049] Among the preset priorities, T1 is the first priority, T2 is the second priority, FLAIR is the third priority, and T1ce is the fourth priority. It can be seen that the available mode of T1 is Fourier transformed, and the spatial domain information is transformed into the frequency domain to obtain the frequency domain information of the available mode in the i-th sample.

[0050] Specifically, the T1 can be represented by an image of the modality: (where H represents the height of the image and W represents the width of the image), perform Fourier transform (DFT), the formula is: Where, is the frequency domain transformation function, that is, the frequency domain information mentioned above; u and v are frequency variables, representing the spatial frequency of the image in the horizontal (x-axis) and vertical (y-axis) directions respectively; For spatial domain images at position Pixel value of is the Fourier basis function.

[0051] Step S202 : obtaining the amplitude spectrum and phase spectrum of the i-th sample based on the frequency domain information of the i-th sample.

[0052] That is, based on the frequency domain information of T1 in the i-th sample, the fast Fourier transform (FFT) is used to decompose it and realize the accelerated operation of Fourier transform. The output of FFT is is a complex matrix, The amplitude and phase calculation formula is used to obtain the amplitude spectrum and phase spectrum ,as follows: Above, the amplitude spectrum As the main basis for subsequent frequency band division, the phase spectrum Used for inverse transformation to reconstruct the image. Please refer to the following description for details.

[0053] Step S203 , downsampling the amplitude spectrum of the i-th sample and normalizing it to a preset range to obtain a processed amplitude spectrum of the i-th sample.

[0054] The amplitude spectrum Downsampling is performed (e.g., from H×W to H / 4×W / 4) to reduce the amount of computation, alleviating the problem of large diffusion model parameters and long training time due to high resolution of medical images. The amplitude spectrum values ​​are normalized to the range of [0,1] or other preset ranges to obtain the amplitude spectrum of the i-th sample after processing.

[0055] Step S204: input the processed amplitude spectrum of the i-th sample into a dynamic mask generation module to obtain a frequency band mask of the i-th sample.

[0056] The processed spectral features are fed into the dynamic mask generation module, which consists of a convolutional layer, a global pooling layer, and a fully connected layer, and the output is a frequency band mask. .

[0057] It should also be noted that traditional frequency domain information separation methods (such as Gaussian filters) use fixed thresholds to divide low-frequency and high-frequency components. This method cannot adapt to the frequency domain characteristics of different modes and it is difficult to achieve reliable adaptive division of high- and low-frequency information under different available modes. Therefore, this application designs a module that can dynamically capture the effective frequency domain information of different modes and adaptively adjust the weights of high- and low-frequency features to improve the rationality and reliability of frequency domain information. The goal of dynamic mask generation is to learn the frequency band division of the input mode through a neural network and generate an adaptive mask. (The area close to 0 represents low-frequency components (such as smooth areas), and the area close to 1 represents high-frequency components (such as edges and textures)), so as to achieve adaptive extraction of frequency domain information of different modes and separation of high and low frequency features.

[0058] Step S205 : weighting is performed using the frequency band mask of the i-th sample to separate and obtain the low-frequency amplitude spectrum and the high-frequency amplitude spectrum of the i-th sample.

[0059] Utilizing masks Perform mask weighting to separate low-frequency and high-frequency components as follows: Low frequency amplitude spectrum: , high-frequency amplitude spectrum: .

[0060] Step S206 : Combining the low-frequency amplitude spectrum and the high-frequency amplitude spectrum of the i-th sample with the phase spectrum of the i-th sample respectively, and using inverse fast Fourier transform to obtain high-frequency information and low-frequency information of the i-th sample.

[0061] The network is trained by supervised learning, and the generated mask It can effectively separate low-frequency and high-frequency components. The separated amplitude spectrum is compared with the original phase spectrum. Combined, the spatial domain image is reconstructed by inverse FFT (IFFT) to obtain the high-frequency image of the i-th sample (also called the high-frequency information of the i-th sample) and the low-frequency image of the i-th sample (also called the low-frequency information of the i-th sample), as follows: Low frequency image: , high-frequency image: .

[0062] In an embodiment, the conditional diffusion module based on modal attention includes a forward noise addition and diffusion module and a reverse noise reduction generation module based on modal attention. The forward noise addition and diffusion module is used to gradually add Gaussian noise to the missing modality in the i-th sample until T noise images are generated. The reverse noise reduction generation module based on modal attention is used to gradually reduce the noise of the T-th noise image generated by the forward noise addition and diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T noise-reduced images are generated. Wherein, T is a positive integer greater than 1, and i is a positive integer greater than or equal to 1.

[0063] The above forward noise diffusion module can be specifically referred to Figure 2 and the following contents are understood as follows: It should be noted that this application uses FLAIR and T2 as missing modalities, and T1ce and T1 as available modalities. Therefore, both the missing FLAIR and T2 modalities require noise addition and noise reduction. The following explanation focuses on T2 processing; the FLAIR processing is similar and can be used for reference.

[0064] It should also be noted that the forward diffusion module of the diffusion model can be regarded as a Markov process. During this forward propagation process, noise is gradually added to the given input image (the image simulating the missing mode, such as T2), turning it from a clear image into a completely random noisy image.

[0065] Assume the number of noise adding steps is t, are the t noise image patches generated at each step, where is the original image without noise, is the complete Gaussian noise image obtained after t steps of noise addition.

[0066] At time t, the latent variable After adding Gaussian noise, we get The process can be expressed as: , in, express exist The distribution under the conditions, indicating that Add noise to get The process follows a Gaussian distribution , represents Gaussian distribution; is a predefined variance used to control the noise weight in each diffusion time step.

[0067] According to the above conditions, the distribution , it can be deduced that The expression is: , in, represents the Gaussian distribution noise added at time t; is the variance corresponding to the Gaussian distribution noise.

[0068] In addition, there is another way to get The expression is as follows: If the original image is given , then at time t, the initial variable Conditional distribution after adding Gaussian noise for: , in, ; (Accumulated from time t=1 to time t=i).

[0069] Then the initial variable at time 0 is After random t-step noise addition, the output is obtained The expression is: .

[0070] The I that appears multiple times above is the unit matrix and is explained here uniformly.

[0071] In this embodiment, the number of noise addition steps is T, and T noisy images of the missing modes FLAIR and T2 are generated through the forward noise diffusion module.

[0072] The above-mentioned reverse denoising generation module based on modal attention can be specifically referred to Figure 2 and the following contents are understood as follows: The modality attention-based reverse denoising generation module is composed of T sub-generation modules, each of which includes a modality attention module and a U-Net encoding-decoding module; The modality attention module is configured to obtain, at time t, a t-th embedded feature map of the four modal images in the ith sample based on the four modal images in the ith sample, determine a t-th attention weight between any two modalities in the ith sample based on the t-th embedded feature map of the four modal images in the ith sample and the state code of the ith sample, and obtain a t-th new feature map of the four modal images in the ith sample based on the t-th attention weight between any two modalities in the ith sample; The U-Net encoding-decoding module is configured to, at time t, reconstruct the available modality and denoise the missing modality based on the t-th new feature map of the four modal images in the ith sample and under the guidance of the high-frequency information or low-frequency information of the ith sample, thereby obtaining a reconstructed image of the available modality and a denoised image of the missing modality at time t; Wherein, when t is T, the four modal images are the original modal image of the available modality and the Tth noise image of the missing modality; when t is not T, the four modal images are the reconstructed image of the available modality at the previous moment and the denoised image of the missing modality, and t is a positive integer greater than or equal to 1 and less than or equal to T.

[0073] It can be understood that the effective feature interaction of the above-mentioned modal attention module is the key to modeling the complementary features between the available modalities. Although the reverse denoising generation module uses the correlation between the available modalities and the missing modalities to model the missing modalities, since each modality is a different morphological annotation of the same sample, there is a large amount of redundant feature information in the information of each modality. In order to further extract effective relevant features and suppress redundant noise, the reverse denoising generation module adds a modal attention module before each U-Net encoding-decoding module from time steps t to t-1. It uses the adaptive characteristics of graph structures in feature learning to further extract relevant information between different modalities to obtain more useful feature maps and suppress redundant noise.

[0074] Specifically, the modality attention module is a graph structure , where V represents the graph nodes corresponding to the four modal features, and E represents the adjacency edge matrix representing the relationship between nodes. In graph edge computing, we will arrive The message passed is defined as the node characteristic and edge weights The product of .

[0075] The edge weight The features used to learn complementary nodes are expressed as: in, Indicates that the modal is missing status code With the original feature map The result after splicing is . is the state vector of the four original modal images, i.e., the missing mode / available mode of modal information. The same logic applies. Indicates that a parameter is The nonlinear function consists of two linear mapping layers and a LeakyReLU activation function layer, which is used to estimate the attention weights between each pair of modalities in G. Based on the edge weights obtained by G, the softmax function is used to merge the information of all available modalities and update the node Feature Map , updated features It can be expressed as: , Among them, N=4.

[0076] when When t, based on the t-th new feature map of the four modal images in the i-th sample and under the guidance of the low-frequency information of the i-th sample, reconstruct the available modality and denoise the missing modality to obtain the reconstructed image of the available modality and the denoised image of the missing modality at time t; when When , based on the t-th new feature map of the four modal images in the i-th sample and under the guidance of the high-frequency information of the i-th sample, the reconstruction generation of the available modality and the denoising generation of the missing modality are performed to obtain the reconstructed image of the available modality and the denoised image of the missing modality at time t, wherein, if the true value of T is an odd number, when obtaining When the value of is , the value of T is increased by 1.

[0077] Specifically, corresponding to the forward noise diffusion module, T noise images of the missing modes FLAIR and T2 are generated, and noise reduction is performed on each of them. The noise reduction of one of them (T2) is still taken as an example.

[0078] The core of the specific generation process is to generate a network through modal attention Prediction noise , and combined with the diffusion coefficient of the forward process 、 Compute the mean of the conditional distribution of the generating process ,variance .

[0079] Assume that the condition at time t is , then the image at time t Perform denoising operation to generate the image of the previous time step The conditional probability distribution of can be expressed as: , in, The noise mean predicted by the modal attention generation network of the reverse denoising module is expressed as: in, for The variance of 、 and Calculation yields: , Afterwards, according to exist Probability distribution formula under conditions , the generation process can be expressed as: , in, is the generated noise component obtained by the modal attention generation module at each time step.

[0080] In addition, if Figure 1 As shown, in the reverse denoising generation module based on modal attention, the value of t in the subscript is from T to 0, and in the forward noise diffusion module, the value of t is from 0 to T. In this application, when the t moment of the reverse denoising generation module based on modal attention and the forward noise diffusion module are expressed at the same time, it is as follows Figure 1 The corresponding moments above and below.

[0081] In step S103, the training set is input into the model for training, and a joint loss function is set. The parameters of the model are optimized by the joint loss function of each sample to obtain an optimized model.

[0082] The joint loss function consists of the learning loss of the learnable frequency domain module and the inverse denoising generation loss based on modality attention.

[0083] The learning loss of the learnable frequency domain module is: , in, represents the learning loss of the learnable frequency domain module for the i-th sample, and Represent the low-frequency amplitude spectrum and high-frequency amplitude spectrum of the i-th sample respectively, and They are respectively the preset low-frequency amplitude spectrum and the i-th preset high-frequency amplitude spectrum of the i-th sample divided according to the preset fixed threshold.

[0084] The inverse denoising generation loss based on modality attention is: , Among them, the represents the inverse denoising generation loss based on modality attention of the i-th sample, The noise image of the missing mode is obtained by the forward noise diffusion module at time t. is the denoised image of the missing modality obtained by the reverse denoising generation module based on modality attention at time t (it should be noted that and At time t, Figure 1 The corresponding moments above and below); is the high-frequency information or low-frequency information of the i-th sample at time t; represents a joint random variable 、 expectations; Represents the Euclidean distance; represents the predicted noise of the inverse denoising generation module based on modality attention and the noise of the forward noise diffusion module The m represents multimodality and has no specific value.

[0085] The joint loss function is: , in, represents the joint loss function of the i-th sample, and λ represents the preset hyperparameter.

[0086] It should be noted that the joint loss function of each sample is obtained above, and the model can be optimized one by one, or the joint loss function of all samples in the training set can be added together to optimize the model. The specific optimization depends on the actual situation and is not limited here.

[0087] Step S104: Input the multimodal MRI images in the test set into the optimized model to obtain generated images corresponding to the missing modalities.

[0088] In summary, the missing modality brain tumor modality generation network model in this application is guided by the high-frequency information and low-frequency information obtained by the learnable frequency domain module, and the modal attention module introduced in the modal attention-based reverse denoising generation module, to learn to capture the relevant features between different modalities and promote the complementarity of feature information; then, the noisy image obtained by the forward denoising diffusion module and the denoised image obtained by the modal attention-based reverse denoising generation module are optimized using the set loss function, thereby improving the accuracy and robustness of the missing modality brain tumor modality generation network model for missing modality generation, and only a single model is required to process different modality combinations.

[0089] Combination of the above Figure 1-Figure 3 The MRI missing modality generation method based on a learnable frequency domain module provided in an embodiment of the present application is described in detail. The following details the MRI missing modality generation system based on a learnable frequency domain module provided in an embodiment of the present application.

[0090] The system specifically includes: data acquisition module, model building module, training module and generation module, as shown below.

[0091] a data acquisition module, configured to acquire a training set and a test set of multimodal MRI images of brain tumors, wherein the training set and the test set include a plurality of samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality; A model construction module is used to construct a missing modality brain tumor generation network model, wherein the missing modality brain tumor generation network model includes a learnable frequency domain module and a conditional diffusion module based on modal attention; the learnable frequency domain module is used to extract the frequency domain information of the available modality in the i-th sample, and perform adaptive separation of high-frequency features and low-frequency features to obtain the high-frequency information and low-frequency information of the i-th sample; the conditional diffusion module based on modal attention includes a forward denoising diffusion module and a reverse denoising generation module based on modal attention, the forward denoising diffusion module is used to gradually add Gaussian noise to the missing modality in the i-th sample until T noise images are generated; the reverse denoising generation module based on modal attention is used to gradually denoise the T-th noise image generated by the forward denoising diffusion module under the guidance of the high-frequency information and low-frequency information of the i-th sample until T denoised images are generated, wherein T is a positive integer greater than 1, and i is a positive integer greater than or equal to 1; The training module is used to input the training set into the model for training and set the joint loss function. The model parameters are optimized through the joint loss function of each sample to obtain the optimized model. The generation module is used to input the multimodal MRI images in the test set into the optimized model to obtain the generated images corresponding to the missing modalities.

[0092] In addition, for the above-mentioned specific implementation of the system, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation. Moreover, it should be noted that in each module of the system of the present application, the components therein are logically divided according to the functions to be implemented, but the present application is not limited thereto, and the components can be re-divided or combined as needed.

[0093] The above describes the MRI missing modality generation method and system based on the learnable frequency domain module provided by the embodiment of the present application. Figure 4-Figure 5 The invention provides a detailed description of the MRI missing modality generation device based on the learnable frequency domain module provided in the embodiments of the present application.

[0094] Figure 4 This is a schematic diagram of the structure of the MRI missing mode generation device based on the learnable frequency domain module provided in the embodiment of the present application. Figure 1 For example, Figure 4 As shown, the MRI missing modality generation device 400 based on the learnable frequency domain module includes: a transceiver module 401 and a processing module 402. For ease of description, Figure 4 Only the main components of the MRI missing modality generation device based on the learnable frequency domain module are shown.

[0095] Among them, the transceiver module 401 is used to perform the transceiver function of the above-mentioned MRI missing modality generation method based on the learnable frequency domain module, and the processing module 402 is used to perform other functions of the above-mentioned MRI missing modality generation method based on the learnable frequency domain module except the transceiver function.

[0096] Optionally, the transceiver module 401 may include a sending module ( Figure 4 Not shown) and the receiving module ( Figure 4 (not shown). The sending module is used to implement the sending function of the MRI missing modality generation device 400 based on the learnable frequency domain module, and the receiving module is used to implement the receiving function of the MRI missing modality generation device 400 based on the learnable frequency domain module.

[0097] Optionally, the MRI missing modality generation device 400 based on the learnable frequency domain module may further include a storage module ( Figure 4 (not shown) the storage module stores a program or instruction. When the processing module 402 executes the program or instruction, the apparatus 400 for generating missing MRI modalities based on a learnable frequency domain module can perform the method for generating missing MRI modalities based on a learnable frequency domain module according to the embodiment of the present application.

[0098] The following combination Figure 5Each component of the MRI missing modality generation device 500 based on a learnable frequency domain module is described in detail: The processor 501 is the control center of the MRI missing modality generation device 500 based on a learnable frequency domain module, and can be a single processor or a collective term for multiple processing elements. For example, the processor 501 can be one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0099] Optionally, the processor 501 can execute various functions of the MRI missing modality generation device 500 based on a learnable frequency domain module by running or executing a software program stored in the memory 502 and calling data stored in the memory 502, such as executing the MRI missing modality generation method based on a learnable frequency domain module in an embodiment of the present application.

[0100] In a specific implementation, as an embodiment, the processor 501 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0101] In a specific implementation, as an embodiment, the MRI missing modality generation device 500 based on the learnable frequency domain module may also include multiple processors, such as Figure 5 The processor 501 and processor 504 shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). The memory 502 is used to store the software program that executes the solution of the present application, and the execution is controlled by the processor 501. The specific implementation method can refer to the above-mentioned method embodiment and will not be repeated here.

[0102] Alternatively, the memory 502 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 502 may be integrated with the processor 501 or exist independently and accessed through the interface circuit ( Figure 5 (not shown) is coupled to the processor 501, which is not specifically limited in this embodiment of the present application.

[0103] The transceiver 503 is used for communicating with other communication devices. For example, the MRI missing modality generation device 500 based on a learnable frequency domain module is a first device, and the transceiver 503 can be used for communicating with a second device or a third device.

[0104] Optionally, the transceiver 503 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0105] Optionally, the transceiver 503 may be integrated with the processor 501 or may exist independently and communicate with the MRI missing modality generation device 500 based on a learnable frequency domain module through an interface circuit ( Figure 5 (not shown) is coupled to the processor 501, which is not specifically limited in this embodiment of the present application.

[0106] It is understandable that Figure 5 The structure of the MRI missing modality generation device 500 based on a learnable frequency domain module shown in the figure does not constitute a limitation of the MRI missing modality generation device based on a learnable frequency domain module. The actual MRI missing modality generation device based on a learnable frequency domain module may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0107] In addition, the technical effects of the MRI missing modality generation device 500 based on the learnable frequency domain module can refer to the technical effects of the method described in the above method embodiment, and will not be repeated here.

[0108] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), but may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0109] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0110] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in the embodiments of this application are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

Claims

1. A method for generating MRI missing modalities based on a learnable frequency domain module, characterized in that: The method comprises: Acquire a training set and a test set of multimodal MRI images of brain tumors, wherein the training set and the test set include a plurality of samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality; A missing modality brain tumor generation network model is constructed, wherein the missing modality brain tumor generation network model includes a learnable frequency domain module and a conditional diffusion module based on modal attention; the learnable frequency domain module is used to extract the frequency domain information of the available modality in the i-th sample, and perform adaptive separation of high-frequency features and low-frequency features to obtain high-frequency information and low-frequency information of the i-th sample; the conditional diffusion module based on modal attention includes a forward denoising diffusion module and a reverse denoising generation module based on modal attention, wherein the forward denoising diffusion module is used to gradually add Gaussian noise to the missing modality in the i-th sample until T noise images are generated; the reverse denoising generation module based on modal attention is used to gradually denoise the T-th noise image generated by the forward denoising diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T denoised images are generated, wherein T is a positive integer greater than 1, and i is a positive integer greater than or equal to 1; Inputting the training set into the model for training, setting a joint loss function, and optimizing the parameters of the model through the joint loss function of each sample to obtain an optimized model; The multimodal MRI images in the test set are input into the optimized model to obtain generated images corresponding to the missing modalities.

2. The method for generating MRI missing modalities based on a learnable frequency domain module according to claim 1, wherein: The training set and test set for obtaining multimodal MRI images of brain tumors include: Acquire a multimodal MRI image dataset of a brain tumor, wherein the multimodal MRI image dataset includes a plurality of samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality; Preprocessing the multimodal MRI image dataset to obtain a two-dimensional multimodal MRI slice image dataset, and dividing the two-dimensional multimodal MRI slice image dataset into a first dataset and a second dataset according to a preset ratio; performing data enhancement on the two-dimensional multimodal MRI slice images in the first data set to obtain a third data set; generating a state code for the data-enhanced two-dimensional multimodal MRI slice image in the third data set to obtain a corresponding state code, and concatenating the state code with the corresponding original modality image in the multimodal MRI image data set to obtain a fourth data set; The fourth data set is a training set of multimodal MRI images of brain tumors, and the second data set is a test set of multimodal MRI images of brain tumors.

3. The method for generating MRI missing modalities based on a learnable frequency domain module according to claim 2, wherein: The data enhancement processing includes: random rotation, random flipping, random cropping and random addition of noise to the image.

4. The method for generating MRI missing modalities based on a learnable frequency domain module according to claim 1, wherein: The learnable frequency domain module is used to extract the frequency domain information of the available modes in the i-th sample and perform adaptive separation of high-frequency features and low-frequency features to obtain the high-frequency information and low-frequency information of the i-th sample, including: Performing a Fourier transform on the available modes in the i-th sample, transforming the spatial domain information into the frequency domain, and obtaining the frequency domain information of the available modes in the i-th sample. If there is more than one available mode in the i-th sample, the available mode with the highest priority is selected for processing according to the priority, and the frequency domain information of the i-th sample is obtained. The priority is preset; Based on the frequency domain information of the i-th sample, obtaining the amplitude spectrum and phase spectrum of the i-th sample; Downsampling the amplitude spectrum of the i-th sample and normalizing it to a preset range to obtain a processed amplitude spectrum of the i-th sample; Inputting the processed amplitude spectrum of the i-th sample into a dynamic mask generation module to obtain a frequency band mask of the i-th sample; Using the frequency band mask of the i-th sample for weighting, the low-frequency amplitude spectrum and the high-frequency amplitude spectrum of the i-th sample are separated; The low-frequency amplitude spectrum and the high-frequency amplitude spectrum of the i-th sample are respectively combined with the phase spectrum of the i-th sample, and the high-frequency information and the low-frequency information of the i-th sample are obtained by inverse fast Fourier transform.

5. The method for generating MRI missing modalities based on a learnable frequency domain module according to claim 2, wherein: The modal attention-based reverse denoising generation module is configured to gradually denoise the T-th noise image generated by the forward denoising diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T denoised images are generated, including: The modality attention-based reverse denoising generation module is composed of T sub-generation modules, each of which includes a modality attention module and a U-Net encoding-decoding module; The modality attention module is configured to obtain, at time t, a t-th embedded feature map of the four modal images in the ith sample based on the four modal images in the ith sample, determine a t-th attention weight between any two modalities in the ith sample based on the t-th embedded feature map of the four modal images in the ith sample and the state code of the ith sample, and obtain a t-th new feature map of the four modal images in the ith sample based on the t-th attention weight between any two modalities in the ith sample; The U-Net encoding-decoding module is configured to, at time t, reconstruct the available modality and denoise the missing modality based on the t-th new feature map of the four modal images in the ith sample and under the guidance of the high-frequency information or low-frequency information of the ith sample, thereby obtaining a reconstructed image of the available modality and a denoised image of the missing modality at time t; Wherein, when t is T, the four modal images are the original modal image of the available modality and the Tth noise image of the missing modality; when t is not T, the four modal images are the reconstructed image of the available modality at the previous moment and the denoised image of the missing modality, and t is a positive integer greater than or equal to 1 and less than or equal to T.

6. The method for generating MRI missing modalities based on a learnable frequency domain module according to claim 5, wherein: The method of reconstructing the available modality and denoising the missing modality at time t based on the t-th new feature map of the four modal images in the i-th sample and under the guidance of the high-frequency information or low-frequency information of the i-th sample to obtain the reconstructed image of the available modality and the denoised image of the missing modality at time t includes: when When t, based on the t-th new feature map of the four modal images in the i-th sample and under the guidance of the low-frequency information of the i-th sample, reconstruct the available modality and denoise the missing modality to obtain the reconstructed image of the available modality and the denoised image of the missing modality at time t; when When , based on the t-th new feature map of the four modal images in the i-th sample and under the guidance of the high-frequency information of the i-th sample, the reconstruction generation of the available modality and the denoising generation of the missing modality are performed to obtain the reconstructed image of the available modality and the denoised image of the missing modality at time t, wherein, if the true value of T is an odd number, when obtaining When the value of is , the value of T is increased by 1.

7. The method for generating MRI missing modalities based on a learnable frequency domain module according to claim 4, wherein: The joint loss function consists of the learning loss of the learnable frequency domain module and the inverse denoising generation loss based on modal attention; The learning loss of the learnable frequency domain module is: ， Among them, the Represents the learning loss of the learnable frequency domain module of the i-th sample, and stated Represent the low-frequency amplitude spectrum and high-frequency amplitude spectrum of the i-th sample respectively, and stated are respectively the preset low-frequency amplitude spectrum and the preset high-frequency amplitude spectrum of the i-th sample divided according to the preset fixed threshold; The modality-attention-based reverse denoising generation loss is: ， Among them, the represents the inverse denoising generation loss based on modality attention of the i-th sample, The noise image of the missing mode is obtained by the forward noise diffusion module at time t. is the denoised image of the missing modality obtained by the reverse denoising generation module based on modality attention at time t; is the high-frequency information or low-frequency information of the i-th sample at time t; represents a joint random variable 、 expectations; Represents the Euclidean distance; represents the predicted noise of the inverse denoising generation module based on modality attention and the noise of the forward noise diffusion module The mean square error of ; the m represents multimodality and has no specific value; The joint loss function is: ， Among them, the represents the joint loss function of the i-th sample, and λ represents a preset hyperparameter.

8. A system for generating MRI missing modalities based on a learnable frequency domain module, characterized in that: The system comprises: a data acquisition module, configured to acquire a training set and a test set of multimodal MRI images of brain tumors, wherein the training set and the test set include a plurality of samples, and each sample includes four original modality images, and each original modality image is an available modality or a missing modality; A model construction module is used to construct a missing modality brain tumor generation network model, wherein the missing modality brain tumor generation network model includes a learnable frequency domain module and a conditional diffusion module based on modal attention; the learnable frequency domain module is used to extract the frequency domain information of the available modality in the i-th sample, and perform adaptive separation of high-frequency features and low-frequency features to obtain high-frequency information and low-frequency information of the i-th sample; the conditional diffusion module based on modal attention includes a forward denoising diffusion module and a reverse denoising generation module based on modal attention, wherein the forward denoising diffusion module is used to gradually add Gaussian noise to the missing modality in the i-th sample until T noise images are generated; the reverse denoising generation module based on modal attention is used to gradually denoise the T-th noise image generated by the forward denoising diffusion module under the guidance of the high-frequency information or low-frequency information of the i-th sample until T denoised images are generated, wherein T is a positive integer greater than 1, and i is a positive integer greater than or equal to 1; A training module, configured to input the training set into the model for training, set a joint loss function, optimize the parameters of the model through the joint loss function of each sample, and obtain an optimized model; A generation module is used to input the multimodal MRI images in the test set into the optimized model to obtain generated images corresponding to the missing modalities.

9. An MRI missing modality generation device based on a learnable frequency domain module, characterized in that: The apparatus comprises: a module for executing the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or instructions, which, when executed, causes the method according to any one of claims 1 to 7 to be performed.

Images