Cross-modal medical image synthesis method, system and device based on Fourier features

Abstract

The present invention provides a cross-modal medical image synthesis method, system and device based on Fourier features, which belongs to the technical field of medical image synthesis in computer vision. It includes: using a pseudo target modality generator PTMG to generate a source modality image I source Pseudo target modality image I pt ; Style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the multi-layer perceptron module MLP processes the style feature x style Get the style feature #imgabs0# based on Shapley value, and the frequency selection module FSS based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to analyze the structural information T s Adjustment; structural feature extractor E from the source modality image I source Extract structural features x struct ; The structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain the enhanced structural feature #imgabs1# generator G to synthesize the style feature #imgabs2# and the enhanced structural feature #imgabs3# to generate the target modality image I target , #imgabs4#

Description

Technical Field

[0001] The present invention relates to the technical field of medical image synthesis of computer vision, and in particular to a method, system and device for cross-modal medical image synthesis based on Fourier features. Background Art

[0002] Cross-modal medical image synthesis aims to synthesize target modality medical images by learning the relationship between given source modality images and target modality images. It plays a very important role in modern medical diagnosis, because medical images of different modalities can highlight the characteristics of different tissues. Combining multiple modality medical image information for accurate disease assessment and diagnosis has become a popular method for intelligent medical diagnosis. The wide application of cross-modal medical image synthesis tasks includes: using cross-modal synthesis methods to generate missing or damaged images, using healthy modality images to synthesize pseudo-healthy images for lesion segmentation, and using images that take less time to scan to generate images that take a lot of time to scan.

[0003] In the early days, traditional machine learning methods were used for cross-modal medical image synthesis, such as atlas-based methods, which performed poorly on images with obvious abnormalities. Recently, cross-modal medical image synthesis methods based on deep neural network modeling have become mainstream, such as using conditional GANs for multi-contrast MRI image synthesis and further improving the synthesis quality by adding adjacent cross-sectional images. However, existing work aims to minimize the difference in pixel or voxel intensity. This pixel-by-pixel training mode may cause abnormal anatomical structure generation. For example, IMT, which adds a penalty for mismatch between input and output, ensures the quantization of low-level textures, but ignores the structural content in the image, such as texture or shape.

[0004] The synthesis of cross-modal medical images is a challenging task because it requires the model to keep the pathological information of the original image unchanged during the conversion process. Most of the previous cross-modal image synthesis algorithms cannot highlight more important disease-related areas and local detailed anatomical structures. Even if the generated images have considerable image quality evaluation indicators (PSNR, SSIM, etc.), the generation accuracy problem (the generated tissue structure is wrong compared to the real image) will inevitably cause great confusion to the radiologist's diagnosis. For example, the generated image anatomical structure is distorted, subtle structures are missing, non-existent structures are generated, the generated structures are discontinuous, regional features are distorted, and white noise appears. In summary, the existing synthesis algorithms have the problem of ignoring image structure, and their performance on images is not satisfactory. Summary of the invention

[0005] In view of this, the present invention provides a cross-modal medical image synthesis method, system and device based on Fourier features, which improve the abnormal problems occurring in the anatomical structure generation of cross-modal medical images and improve the accuracy of medical images.

[0006] The technical solution adopted by the embodiment of the present invention to solve the technical problem is:

[0007] A cross-modal medical image synthesis method based on Fourier features, comprising:

[0008] Step S1, using a pseudo target modality generator PTMG to generate a source modality image I source Pseudo target modality image I pt , the source modality image I source For images with anatomical structures;

[0009] Step S2, style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the style feature x is processed by the multi-layer perceptron module MLP style Get style features

[0010] Step S3, based on the Shapley value, the frequency selection module FSS selects the pseudo target modal image I p t and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to s Make adjustments;

[0011] Step S4: a structural feature extractor E extracts the structure feature from the source modality image I source Extract structural features x struct ;

[0012] Step S5, the structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features

[0013] Step S6: Generator G synthesizes the style features And the enhanced structural features Generate target modality image I target ,

[0014] Preferably, the frequency selection module FSS in step S3 is based on the pseudo target modal image I pt and the source modality image I sourceExtract consistent structural information T s include:

[0015] Step S31: input the source modality image I to the selection module FSS. source and the pseudo target modality image I pt , where I source with I pt The image size is d1×d2;

[0016] Step S32: Create a frequency domain component set and

[0017]

[0018]

[0019] Wherein, f represents two-dimensional cosine transform DCT, the pseudo target modal image I pt Transformed to the frequency domain, it is expressed as f(I pt ), the source modality image I source Transformed to the frequency domain, it is expressed as f(I source ), f(I)(·,·) represents the frequency component of image I, and the component corresponding to the mean value of the image is removed from the frequency domain component set f(I)(0,0). Let The frequency domain components are expressed as:

[0020]

[0021]

[0022]

[0023] in, Respectively represent two-dimensional DCT images; Respectively represent the input images;

[0024] Step S33, calculating the pseudo target modality image I pt and the source modality image I source Structural similarity R(S):

[0025] R(S)=SSIM(f -1 (M s ⊙f(I source )),f -1 (M s ⊙f(I pt )))

[0026] Among them, f -1represents the inverse operation of two-dimensional discrete cosine transform DCT, ⊙ represents Hadamard product, M s is the mask generated based on the alliance S;

[0027] The calculation formula of SSIM is:

[0028]

[0029] Where μ1 is f -1 (M s ⊙f(I source )) is the average value, μ2 is f -1 (M s ⊙f(I pt )) is the average value, σ1 is f -1 (M s ⊙f(I source )) is the variance of f -1 (M s ⊙f(I pt ))’s variance, σ 1，2 Yes -1 (M s ⊙f(I source )) and f -1 (M s ⊙f(I pt )) covariance, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability; L is the dynamic range of the input image pixel value, k1 = 0.01, k2 = 0.03;

[0030] M S The calculation formula is:

[0031] M s (u, v) = II[f(I)(u, v)]∈S

[0032] Where u is the horizontal position, v is the vertical position, II(·) represents the indicator function, which takes the value of 1 when f(I)(u, v)∈S is satisfied, and takes the value of 0 when f(I)(u, v)∈S is not satisfied;

[0033] Step S34, calculating the Shapley value of each frequency domain component i according to the R(S)

[0034]

[0035]

[0036]

[0037] in, Is in the collection The average marginal contribution of each frequency domain component i added to the combination under different arrangements π, is the frequency domain component set The set of all permutations of , π(i) represents the frequency domain component The position in the arrangement π; represents the set of frequency domain components that precede frequency domain component i in the arrangement π;

[0038] The reward obtained by Alliance S is defined as:

[0039]

[0040] Step S35: select the first k numbers in descending order. The corresponding k frequency domains are converted into spatial domains to obtain a structural consistency image denoted as T s .

[0041] Preferably, after step S6, the method further includes:

[0042] Step S7: the discriminator D uses the target modality image I target The loss result is obtained by making a judgment, and the loss result is used to measure the image restoration effect.

[0043] The present invention also provides a cross-modal medical image synthesis system based on Fourier features, characterized in that it includes:

[0044] The generation module uses the pseudo target modality generator PTMG to generate the source modality image I source Pseudo target modality image I pt , the source modality image I source For images with anatomical structures;

[0045] Extraction module, using style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the style feature x is processed by the multi-layer perceptron module MLP style Get style features

[0046] The extraction module, based on the Shapley value, uses the frequency selection module FSS based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to sMake adjustments;

[0047] The extraction module uses a structural feature extractor E to extract the source modality image I source Extract structural features x struct ;

[0048] The calculation module converts the structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features

[0049] The generation module, the generator G synthesizes the style features And the enhanced structural features Generate target modality image I target ,

[0050] It can be seen from the above technical solution that the cross-modality medical image synthesis method based on Fourier features provided by the embodiment of the present invention first uses a pseudo target modality generator PTMG to generate a source modality image I source Pseudo target modality image I pt , source modality image I source For an image with anatomical structure; style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the multi-layer perceptron module MLP processes the style feature x style Get style features Based on the Shapley value, the frequency selection module FSS is based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to analyze the structural information T s Adjustment; structural feature extractor E from the source modality image I source Extract structural features x struct ; The structural feature x strucl With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features Generator G synthesizes style features And enhance the structural features Generate target modality image I target , The present invention uses Shapley values ​​to quantify the contribution of each frequency component to the brightness, contrast and structure of different modal MRI image pairs. At the same time, based on the medical prior that MRI images of different modalities should have the same anatomical structure, the frequency of MRI images of different modes expressing the same structure is selected according to the quantified contribution value of each frequency, and then the anatomical structure generated by the target mode is iteratively adjusted to generate a reasonable anatomical structure image. Therefore, the solution of the present invention can improve the abnormal problems in the anatomical structure generation of cross-modal medical images and improve the accuracy of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a system architecture diagram for implementing the cross-modal medical image synthesis method based on Fourier features of the present invention.

[0052] Figure 2 This is a visualization of the structural features selected based on this method.

[0053] Figure 3 Run the main interface for the system.

[0054] Figure 4 Generates an interface for system target images.

[0055] Figure 5 Generates an interface for system target images.

[0056] Figure 6 Generate images for T2 to FLAIR to visualize experimental results.

[0057] Figure 7 Generate images for FLAIR to T2 to visualize experimental results.

[0058] Figure 8 The present invention is a block diagram of an electronic device for implementing the cross-modal medical image synthesis method based on Fourier features according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] The technical scheme and technical effects of the present invention are further elaborated in detail below in conjunction with the accompanying drawings of the present invention.

[0060] In order to ensure the consistency of pathological information during the transformation process and generate reasonable anatomical structure images, we proposed a frequency selection method based on the FSS method of Shapley value, that is, a cross-modal medical image synthesis method based on Fourier characteristics. The image processing of the present invention can be applied to nuclear magnetic resonance images and can also be applied to other types of medical equipment imaging.

[0061] refer to Figure 1 The image processing flow shown in FIG. 1 is a cross-modal medical image synthesis method based on Fourier features of the present invention, and the steps include:

[0062] Step S1, using a pseudo target modality generator PTMG to generate a source modality image I source Pseudo target modality image I pt , source modality image I source An image with anatomical structure (Resnet-like and U-Net-like structures are considered PTMG);

[0063] Step S2, style feature extractor E style From the pseudo target modality image I p tExtract style features x style , and then the multi-layer perceptron module MLP processes the style feature x style Get style features

[0064] Step S3, based on the Shapley value, the frequency selection module FSS is based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to analyze the structural information T s Make adjustments;

[0065] Step S4: the structural feature extractor E extracts the image I from the source modality. source Extract structural features x struct ;

[0066] Step S5, the structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features

[0067] Step S6, generator G synthesizes style features And enhance the structural features Generate the target modality image Itarget,

[0068] Based on the above-mentioned cross-modal medical image synthesis method and model based on Fourier features, a corresponding cross-modal medical image synthesis system was designed and developed. The system development environment is shown in Table 1.

[0069] Table 1 Development environment of cross-modal medical image synthesis system

[0070]

[0071]

[0072] like Figure 1As shown in Figure 1, it is observed that different modality images are multiple different representations of the same tissue or descriptions with different significant features, and these different descriptions constitute different modality images of the same tissue. The model aims to extract the same structural information contained in different modality images, and then assist the model to generate a target modality image with a reasonable anatomical structure. Specifically, for the input image I source , the structural feature extractor E extracts image I source The structural features of x struct .x struct The underlying information is a set of 64×64 feature maps with a total of 256 channels. Since the target modality image is not available during the test phase, we cannot directly extract the style information of the target modality during the test phase. Therefore, we refer to the work of Zhan et al. (Decoupled dual feature extraction based GAN for MRI image synthesis) and establish a pseudo target modality generator PTMG to generate the pseudo target modality I pt In the style feature extractor E style Extract the style information of the pseudo target modality image x style Based on the Shapley value, we use the frequency selection of the FSS (Frequency Selection Based on Shapley Value) module to extract consistent structural information from images of different modalities. Finally, the structural feature X struct and style features It is provided to the generator G to generate target modality images with reasonable anatomical structure.

[0073] Furthermore, in order to achieve reasonable anatomical structure image generation, the image is converted to the frequency domain, and the contribution of each frequency to the brightness, contrast and structure of the different modality MRI image pairs is quantified using the Shapley value. Then, the frequencies that express the same brightness, contrast and structure of the different modality MRI images are selected, and the anatomical structure generated by the target modality is iteratively adjusted. Specifically, step S3 frequency selection module FSS is based on the pseudo target modality image I pt and the source modality image I source Extract consistent structural information T s The specific implementation process includes:

[0074] Step S31: input source modality image I to selection module FSS source and pseudo target modality image I pt , where I source with I pt The image size is d l ×d2;

[0075] Step S32: Create a frequency domain component set and

[0076]

[0077]

[0078] Where f represents the two-dimensional cosine transform DCT, the pseudo target modal image I pt Transformed to the frequency domain, it is expressed as f(I pt ), source modality image I source Transformed to the frequency domain, it is expressed as f(I source ), f(I)(·,·) represents the frequency component of image I. In the frequency domain component set, f(I)(0,0) is excluded from the frequency domain component set, and the image mean is kept unchanged to ensure fair calculation of the marginal contribution component of the frequency. Let The frequency domain components are expressed as:

[0079]

[0080]

[0081]

[0082] The constraints include: Respectively represent two-dimensional DCT images; Respectively represent the input image; H represents the height of the image, W represents the width of the input image, It is the basic equation of two-dimensional cosine transform;

[0083] Step S33, calculating the pseudo target modality image I p t and source modality image I source Structural similarity R(S):

[0084] R(S)=SSIM(f -1 (M s ⊙f(I source )),f -1 (M s ⊙f(I pt )))

[0085] Among them, f -1 represents the inverse operation of two-dimensional discrete cosine transform DCT, ⊙ represents Hadamard product, M s is the mask generated by the union S, for two input images I pt and I source , using the same S;

[0086] The calculation formula of SSIM is:

[0087]

[0088] Where μ1 is f -1 (M s ⊙f(I source )) is the average value, μ2 is f -1 (M s ⊙f(I pt )) is the average value, σ1 is f -1 (M s ⊙f(I source )) is the variance of f -1 (M s ⊙f(I pt ))’s variance, σ 1，2 Yes -1 (M s ⊙f(I source )) and f -1 (M s ⊙f(I pt )) covariance, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability; L is the dynamic range of the input image pixel value, k1 = 0.01, k2 = 0.03;

[0089] M S The calculation formula is:

[0090]

[0091] Among them, u is the horizontal position, v is the vertical position, represents the indicator function, which takes the value 1 when f(I)(u, v)∈S is satisfied, and takes the value 0 when f(I)(u, v)∈S is not satisfied; I p t and I source Substituting the Ms function into different values, we can make Hadamard product with the F function.

[0092] Step S34, calculating the Shapley value of each frequency domain component i according to R(S)

[0093]

[0094]

[0095]

[0096] in, Is in the collection The average marginal contribution of each frequency domain component i added to the combination under different arrangements π, is the frequency domain component set The set of all permutations of , π(i) represents the frequency domain component The position in the arrangement π, ( is the sum of products); represents the set of frequency domain components that precede frequency domain component i in the arrangement π;

[0097] The reward obtained by Alliance S is defined as:

[0098]

[0099] Step S35: select the first k numbers in descending order. The corresponding k frequency domains are converted into spatial domains to obtain a structural consistency image denoted as T s .

[0100] In the above step S5, the structural information T adjusted by the spatial attention module SAM s Expressed as SAM(T S ), structural feature x struct With the adjusted structural information T s Element-wise multiplication is represented as: in, Represents element-wise multiplication.

[0101] Figure 2 The frequencies of brightness, contrast and structure consistency selected by Shapley value quantification are visualized in Figure 1. Compared with the work of minimizing the difference in pixel or voxel intensity, the cross-modal medical image synthesis model based on Fourier features can significantly reduce the abnormal problems in the generation of anatomical structures between the source MRI modality and the target MRI modality, and is therefore more suitable for the field of cross-modal medical image generation.

[0102] Figure 3 For execution Figure 1 The main interface diagram of the system operation method shown in the figure is as follows: Figure 1 The processing flow shown in Figure 2 generates the target modality image as shown in Figure 2. Figure 4 and Figure 5 shown.

[0103] Furthermore, after executing step S6, the method of the present invention may further include:

[0104] Step S7: the discriminator D uses the target modality image I target A judgment is made to obtain a loss result, wherein the loss result can be used to measure the image restoration effect.

[0105] like Figure 6 , 7 , in order to visualize representative examples of results, an ablation study is presented to address the proposed FSS module. First, the target modality images are generated by removing the FSS module while keeping other modules fixed. No FSS produces significant anatomical abnormalities when generating the target modality images. Once the FSS module is added, we can see significant changes in the anatomical structures from the source modality images to the target modality images. FSS without generates target modality images that contain style transfer, but fails in terms of generated structural details. In contrast, wFSS generates target modality images that contain style transfer while ensuring the structural plausibility of the generated target modality images.

[0106] This paper proposes a new framework to synthesize MR images of the target modality to reduce the generation of abnormal anatomical structures. Based on the Shapley value, the present invention uses the frequency selection of the FSS (Frequency Domain Selection Based on Shapley Value) module to extract consistent structural information from images of different modalities, and uses the Shapley value to quantify the contribution of each frequency component to the brightness, contrast and structure of MRI images of different modalities, thereby achieving reasonable anatomical structure generation. Finally, the state-of-the-art results were achieved on the BraTS2020 dataset, and its superior performance proved the effectiveness of the method.

[0107] The present invention also provides a cross-modal medical image synthesis system based on Fourier features, characterized in that it includes:

[0108] The generation module uses the pseudo target modality generator PTMG to generate the source modality image I source Pseudo target modality image I pt , source modality image I source For images with anatomical structures;

[0109] Extraction module, using style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the multi-layer perceptron module MLP processes the style feature x style Get style features

[0110] The extraction module, based on the Shapley value, uses the frequency selection module FSS based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to analyze the structural information T s Make adjustments;

[0111] The extraction module uses the structural feature extractor E to extract the source modality image I source Extract structural features x struct ;

[0112] The calculation module converts the structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features

[0113] Generation module, generator G synthesizes style features And enhance the structural features Generate target modality image I target ,

[0114] The identification module uses the discriminator D to identify the target modality image I target The loss result is obtained by making a judgment, and the loss result is used to measure the image restoration effect.

[0115] The extraction module includes:

[0116] Input unit, input source modality image I to the selection module FSS source and pseudo target modality image I pt , where I source with I pt The image size is d1×d2;

[0117] Create a unit and create a set of frequency domain components and

[0118]

[0119]

[0120] Where f represents the two-dimensional cosine transform DCT, the pseudo target modal image I pt Transformed to the frequency domain, it is expressed as f(I pt ), source modality image I source Transformed to the frequency domain, it is expressed as f(I source ), f(I)(·,·) represents the frequency component of image I, and the component corresponding to the mean value of the image is removed from the frequency domain component set f(I)(0,0). Let The frequency domain components are expressed as:

[0121]

[0122]

[0123]

[0124] in, Respectively represent two-dimensional DCT images; Respectively represent the input images;

[0125] Computation unit, calculates pseudo target modality image I pt and the source modality image I source Structural similarity R(S):

[0126] R(S)=SSIM(f -1 (M s ⊙f(I source )),f -1 (M s ⊙f(I pt )))

[0127] Among them, f -1 represents the inverse operation of two-dimensional discrete cosine transform DCT, ⊙ represents Hadamard product, M s is the mask generated based on the alliance S;

[0128] The calculation formula of SSIM is:

[0129]

[0130] Where μ1 is f -1 (M s ⊙f(I source )) is the average value, μ2 is f -1 (M s ⊙f(I pt )) is the average value, σ1 is f -1 (M s ⊙f(I source )) is the variance of f -1 (M s ⊙f(I pt ))’s variance, σ 1，2 Yes -1 (M s ⊙f(I source )) and f -1 (M s ⊙f(I pt )) covariance, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability; L is the dynamic range of the input image pixel value, k1 = 0.01, k2 = 0.03;

[0131] M SThe calculation formula is:

[0132]

[0133] in, represents the indicator function, which takes the value of 1 when f(I)(u, v)∈S is satisfied, and takes the value of 0 when f(I)(u, v)∈S is not satisfied;

[0134] The calculation unit calculates the Shapley value of each frequency domain component i according to R(S)

[0135]

[0136]

[0137]

[0138] in, Is in the collection The average marginal contribution of each frequency domain component i added to the combination under different arrangements π, is the frequency domain component set The set of all permutations of , π(i) represents the frequency domain component The position in the arrangement π; represents the set of frequency domain components that precede frequency domain component i in the arrangement π;

[0139] The reward obtained by Alliance S is defined as:

[0140]

[0141] Conversion unit, select the first k in descending order The corresponding k frequency domains are converted into spatial domains to obtain a structural consistency image denoted as T s .

[0142] The present invention converts MRI images into frequency domain and proposes a cross-modal medical image synthesis method based on Fourier features. Compared with the existing method of simply dividing the frequency domain into low-frequency and high-frequency parts, the Shapley value is used to quantify the contribution of each frequency component to the brightness, contrast and structure of different modal MRI image pairs. At the same time, based on the medical prior that MRI images of different modalities should have the same anatomical structure, the frequency of MRI images of different modes expressing the same structure is selected according to the quantized contribution value of each frequency, and then the anatomical structure generated by the target mode is iteratively adjusted to generate a reasonable anatomical structure image. Compared with the work of minimizing the difference in pixel or voxel intensity, the cross-modal medical image synthesis method based on Fourier features can significantly reduce the abnormal problems in the generation of anatomical structures between the source MRI modality and the target MRI modality, and is therefore more suitable for the field of cross-modal medical image generation.

[0143] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0144] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0145] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0146] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0147] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a cross-modal medical image synthesis method based on Fourier features. For example, in some embodiments, the cross-modal medical image synthesis method based on Fourier features may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the cross-modal medical image synthesis method based on Fourier features described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the Fourier feature-based cross-modal medical image synthesis method in any other appropriate manner (for example, by means of firmware).

[0148] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0150] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0153] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0154] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0155] What is disclosed above is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A cross-modal medical image synthesis method based on Fourier features, characterized in that: include: Step S1, using a pseudo target modality generator PTMG to generate a source modality image I source Pseudo target modality image I pt , the source modality image I source For images with anatomical structures; Step S2, style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the style feature x is processed by the multi-layer perceptron module MLP style Get style features Step S3, based on the Shapley value, the frequency selection module FSS selects the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to s Make adjustments; Step S4: a structural feature extractor E extracts the structure feature from the source modality image I source Extract structural features x struct ; Step S5, the structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features Step S6: Generator G synthesizes the style features And the enhanced structural features Generate target modality image I target , In step S3, the frequency selection module FSS is based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s include: Step S31: input the source modality image I to the selection module FSS. source and the pseudo target modality image I pt , where I source with I pt The image size is d1×d2; Step S32: Create a frequency domain component set and Wherein, f represents two-dimensional cosine transform DCT, the pseudo target modal image I pt Transformed to the frequency domain, it is expressed as f(I pt ), the source modality image I source Transformed to the frequency domain, it is expressed as f(I source ), f(I)(·,·) represents the frequency component of image I, and the component corresponding to the mean value of the image is removed from the frequency domain component set f(I)(0,0). Let The frequency domain components are expressed as: in, Respectively represent the two-dimensional DCT spectrum; Respectively represent the input images; Step S33, calculating the pseudo target modality image I pt and the source modality image I source Structural similarity R(S): R(S)=SSIM(f -1 (M s ⊙f(I source )),f -1 (M s ⊙f(I pt ))) Among them, f -1 represents the inverse operation of two-dimensional discrete cosine transform DCT, ⊙ represents Hadamard product, M s is the mask generated based on the alliance S; The calculation formula of SSIM is: Where μ1 is f -1 (M s ⊙f(I source )) is the average value, μ2 is f -1 (M s ⊙f(I pt )) is the average value, σ1 is f -1 (M s ⊙f(I source )) is the variance of f -1 (M s ⊙f(I pt ))’s variance, σ 1,2 Yes -1 (M s ⊙f(I source )) and f -1 (M s ⊙f(I pt )) covariance, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability; L is the dynamic range of the input image pixel value, k1 = 0.01, k2 = 0.03; M S The calculation formula is: Among them, u is the horizontal position, v is the vertical position, represents the indicator function, which takes the value of 1 when f(I)(u,v)∈S is satisfied, and takes the value of 0 when f(I)(u,v)∈S is not satisfied; Step S34, calculating the Shapley value of each frequency domain component i according to the R(S) in, It is in the collection The average marginal contribution of each frequency domain component i added to the combination under different arrangements π, is the frequency domain component set The set of all permutations of , π(i) represents the frequency domain component The position in the arrangement π; represents the set of frequency domain components that precede frequency domain component i in the arrangement π; The reward obtained by Alliance S is defined as: Step S35: select the first k numbers in descending order. The corresponding k frequency domains are converted into spatial domains to obtain a structural consistency image denoted as T s .

2. The cross-modal medical image synthesis method based on Fourier features according to claim 1, characterized in that: After step S6, the method further includes: Step S7: the discriminator D uses the target modality image I target The loss result is obtained by making a judgment, and the loss result is used to measure the image restoration effect.

3. A cross-modal medical image synthesis system based on Fourier features, characterized in that: include: The generation module uses the pseudo target modality generator PTMG to generate the source modality image I source Pseudo target modality image I pt , the source modality image I source For images with anatomical structures; Extraction module, using style feature extractor E style From the pseudo target modality image I pt Extract style features x style , and then the style feature x is processed by the multi-layer perceptron module MLP style Get style features The extraction module, based on the Shapley value, uses the frequency selection module FSS based on the pseudo target modal image I pt and the source modality image I source Extract consistent structural information T s , and use the spatial attention module SAM to s Make adjustments; The extraction module uses a structural feature extractor E to extract the source modality image I source Extract structural features x struct ; The calculation module converts the structural feature x struct With the adjusted structural information T s Perform element-wise multiplication to obtain enhanced structural features The generation module, the generator G synthesizes the style features And the enhanced structural features Generate target modality image I target , The extraction module comprises: An input unit, inputting the source modality image I to the selection module FSS source and the pseudo target modality image I pt , where I source with I pt The image size is d1×d2; Create a unit and create a set of frequency domain components and Wherein, f represents two-dimensional cosine transform DCT, the pseudo target modal image I pt Transformed to the frequency domain, it is expressed as f(I pt ), the source modality image I source Transformed to the frequency domain, it is expressed as f(I source ), f(I)(·,·) represents the frequency component of image I, and the component corresponding to the mean value of the image is removed from the frequency domain component set f(I)(0,0). Let The frequency domain components are expressed as: in, Respectively represent the two-dimensional DCT spectrum; Respectively represent the input images; A calculation unit, calculating the pseudo target modality image I pt and the source modality image I source Structural similarity R(S): R(S)=SSIM(f -1 (M s ⊙f(I source )),f -1 (M s ⊙f(I pt ))) Among them, f -1 represents the inverse operation of two-dimensional discrete cosine transform DCT, ⊙ represents Hadamard product, M s is the mask generated based on the alliance S; The calculation formula of SSIM is: Where μ1 is f -1 (M s ⊙f(I source )) is the average value, μ2 is f -1 (M s ⊙f(I pt )) is the average value, σ1 is f -1 (M s ⊙f(I source )) is the variance of f -1 (M s ⊙f(I pt ))’s variance, σ 1,2 Yes -1 (M s ⊙f(I source )) and f -1 (M s ⊙f(I pt )) covariance, c1=(k1L) 2 , c2=(k2L) 2 is a constant used to maintain stability; L is the dynamic range of the input image pixel value, k1 = 0.01, k2 = 0.03; M S The calculation formula is: in, represents the indicator function, which takes the value of 1 when f(I)(u,v)∈S is satisfied, and takes the value of 0 when f(I)(u,v)∈S is not satisfied; The calculation unit calculates the Shapley value of each frequency domain component i according to R(S) in, It is in the collection The average marginal contribution of each frequency domain component i added to the combination under different arrangements π, is the frequency domain component set The set of all permutations of , π(i) represents the frequency domain component The position in the arrangement π; represents the set of frequency domain components that precede frequency domain component i in the arrangement π; The reward obtained by Alliance S is defined as: Conversion unit, select the first k in descending order The corresponding k frequency domains are converted into spatial domains to obtain a structural consistency image denoted as T s .

4. The cross-modal medical image synthesis system based on Fourier features according to claim 3, characterized in that: Also includes: The identification module uses the discriminator D to identify the target modality image I target The loss result is obtained by making a judgment, and the loss result is used to measure the image restoration effect.

5. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 2.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-2.

Images