Low-resolution image domain migration network construction method and device, and non-paired MR image super-resolution method and device based on domain migration

Abstract

The application relates to a low-resolution image domain migration network construction method and device and a non-paired MR image super-resolution method and device based on domain migration, and relates to the super-resolution field of a low-resolution nuclear magnetic resonance (MR) image collected in the field of nuclear magnetic resonance imaging. In view of the problems in the prior art that the degradation process of a real MR image is quite different from the bicubic down-sampling process, and the traditional MR image super-resolution technology based on a single-image super-resolution method cannot meet actual requirements, the technical scheme provided by the application comprises the following steps: collecting a low-resolution MR image set and a down-sampled high-resolution MR image set of the image set; performing preliminary feature extraction on the low-resolution MR image set and the down-sampled high-resolution MR image set; taking the low-resolution features of the extracted low-resolution MR image set and the high-resolution features of the extracted down-sampled high-resolution MR image set as inputs to construct a network. The application is suitable for application in improving the resolution of a low-resolution nuclear magnetic resonance image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of super-resolution of low-resolution magnetic resonance images acquired under low imaging conditions in the field of magnetic resonance imaging, in particular to improving the resolution of low-resolution magnetic resonance images. BACKGROUND

[0002] Magnetic resonance imaging (MRI) is an important medical imaging technology that can provide clear information about tissue structure and function without causing ionizing radiation. High-field magnetic resonance imaging devices can image more hydrogen protons than low-field magnetic resonance imaging devices, especially the excitation of few hydrogen protons, to improve the clarity of magnetic resonance imaging. However, although no harm to the human body has been found in clinical applications, some magnetic resonance imaging doctors still have symptoms such as dizziness and fatigue. At present, it is generally believed that people are safe in a static magnetic field below 3.0T and will not cause irreversible damage to the human body. Excessive radiation from higher field strength imaging devices can cause the body to overheat, and in severe cases, protein inactivation. Therefore, the use of high-field imaging devices to obtain clearer magnetic resonance images (hereinafter referred to as MR images) is limited in clinical applications, and higher clinical MR image resolution is usually achieved by two methods, namely hardware methods and super-resolution methods, while hardware methods such as compressed sensing technology, multi-coil long-time imaging, etc. often accompanied by higher costs, it is difficult to make high-resolution MR images popular in clinical practice, therefore, using super-resolution technology to improve the resolution of clinical MR images has research value, the present application improves the existing MR image super-resolution technology.

[0003] The current mainstream image super-resolution technology mainly includes two methods based on signal processing and learning. The image super-resolution technology based on signal processing has clear principles, simple implementation and high computational efficiency, and has been applied in practical scenarios. However, since it does not fully consider the degradation mechanism of natural images, the interpolated image is far from the expected effect. The learning-based method can fully utilize prior knowledge to make the super-resolution reconstruction better.

[0004] However, the current learning-based super-resolution method for MR images is a single-image super-resolution reconstruction method, that is, under experimental conditions, a bicubic down-sampling is applied to the obtained high-resolution image to obtain a low-resolution image, the obtained low-high resolution paired images are taken as the input and output of the network, and the pixel-level difference between the output of the network and the real high-resolution image is taken as the loss function to train the network. Such a single-image super-resolution algorithm can obtain excellent performance, but this single-image super-resolution method is closer to solving the inverse mapping of the bicubic down-sampling process. In actual scene applications, the degradation process of the real MR image is quite different from the bicubic down-sampling process. The super-resolution image obtained by the network model trained under experimental conditions from the natural and real low-resolution MR image often has distortion, blur, amplification of unknown noise and other problems. Especially for MR images, which require high resolution and have very low tolerance for additional details caused by algorithms, the traditional MR image super-resolution technology based on single-image super-resolution method cannot meet the actual demand. SUMMARY

[0005] In view of the problems in the prior art that the single-image super-resolution method is closer to solving the inverse mapping of the bicubic down-sampling process, in actual scene applications, the degradation process of the real MR image is quite different from the bicubic down-sampling process, the super-resolution image obtained by the network model trained under experimental conditions from the natural and real low-resolution MR image often has distortion, blur, amplification of unknown noise and other shortcomings, and the traditional MR image super-resolution technology based on single-image super-resolution method cannot meet the actual demand, the technical scheme provided by the present application is as follows:

[0006] The method comprises the following steps:

[0007] Step 1: collecting a low-resolution MR image set and a down-sampled high-resolution MR image set of the image set;

[0008] Step 2: performing preliminary feature extraction on the low-resolution MR image set and the down-sampled high-resolution MR image set;

[0009] Step 3: taking the extracted low-resolution features of the low-resolution MR image set and the extracted high-resolution features of the down-sampled high-resolution MR image set as inputs to construct a network.

[0010] Further, a preferred embodiment is provided, wherein in step 3, the process of constructing the network further comprises:

[0011] Step 31: introducing degradation characteristics to the low-resolution features;

[0012] Step 32: performing degradation reconstruction on the low-resolution features with introduced degradation characteristics.

[0013] Further, a preferred embodiment is provided, wherein the step 31 is specifically:

[0014] The encoder-decoder of the parallel Transformer gradually fuses the random noise features on the basis of retaining the low-resolution features.

[0015] Further, a preferred embodiment is provided, wherein the step 32 is specifically:

[0016] The encoder-decoder of the parallel Transformer gradually fuses different dimension features of the low-resolution features on the basis of retaining the low-resolution features.

[0017] Further, a preferred embodiment is provided, wherein the down-sampled high-resolution MR image set of the image set is obtained by bi-cubic down-sampling of the low-resolution MR image set.

[0018] Based on the same inventive concept, the application further provides a low-resolution image domain migration network construction device, which comprises:

[0019] Module 1: for collecting a low-resolution MR image set and a down-sampled high-resolution MR image set of the image set;

[0020] Module 2: for performing preliminary feature extraction on the low-resolution MR image set and the down-sampled high-resolution MR image set;

[0021] Module 3: for constructing a training network by taking the extracted low-resolution features of the low-resolution MR image set as input and the extracted high-resolution features of the down-sampled high-resolution MR image set as output.

[0022] Based on the same inventive concept, the application further provides a domain migration based non-paired MR image super-resolution method, which comprises:

[0023] Step 4: collecting a low-resolution MR image set;

[0024] Step 5: training a single-image super-resolution network by the low-resolution MR image set;

[0025] Step 6: performing up-sampling operation on the low-resolution MR image set by the single-image super-resolution network after migration by the low-resolution image domain migration network constructed according to the low-resolution image domain migration network construction method, to obtain a large-scale high-resolution MR image.

[0026] Based on the same inventive concept, the application further provides a domain migration based non-paired MR image super-resolution device, which comprises:

[0027] Module 4: for collecting a low-resolution MR image set;

[0028] Module 5: a single-image super-resolution network for training by the low-resolution MR image set;

[0029] Module 6: after migration of the low-resolution MR image set by the low-resolution image domain migration network construction device, the network constructed by the low-resolution image domain migration network construction device is used for upsampling operation by the single-image super-resolution network to obtain a large-scale high-resolution MR image.

[0030] Based on the same inventive concept, the application further provides a computer storage medium for storing a computer program, and when the computer program stored in the storage medium is read by a processor of a computer, the computer executes the low-resolution image domain migration network construction method or the non-paired MR image super-resolution method based on domain migration.

[0031] Based on the same inventive concept, the application further provides a computer including a processor and a storage medium, and the storage medium is used for storing a computer program, characterized in that when the processor processes the computer program, the computer executes the low-resolution image domain migration network construction method or the non-paired MR image super-resolution method based on domain migration.

[0032] Compared with the prior art, the application has the following advantages:

[0033] The non-paired MR image super-resolution method based on domain migration provided by the application directly takes original low-resolution and high-resolution MR images as input, solves the image distortion problem in the real MR image super-resolution process by improving the blind super-resolution method framework, introducing the Transformer network structure, and applying the k-space loss function and other methods.

[0034] The non-paired MR image super-resolution method based on domain migration provided by the application is optimized and trained by a joint loss function, and additional details generated in the domain migration process are reduced.

[0035] The application provides a non-paired MR image super-resolution method based on domain migration, which splits the super-resolution process of real MR images into two parts of domain migration process by designing a non-paired image super-resolution network, essentially realizes decoupling of the non-paired MR image super-resolution task, can effectively map the input real low-resolution MR image to the down-sampling high-resolution MR image domain, and generates the high-resolution MR image through a complete single-image super-resolution network, and realizes the super-resolution process without the direct pixel loss constraint condition of high and low resolution MR images. In addition, the application proposes to use a multi-level Transformer structure to jointly embed high and low dimensional features, compared with the previous network structure, the random noise embedding can be realized by modifying only the single branch input, and better network fitting performance can be obtained. The acquisition domain of the MR image is the frequency domain, and the frequency domain characteristics of the image contain more information than the spatial domain characteristics, and the use of a k-space loss function to constrain the domain migration process can ensure that the domain migration process does not introduce too many additional details, and has better representation performance.

[0036] Suitable for application in work for improving the resolution of low-resolution nuclear magnetic resonance images. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The overall structure schematic diagram of the non-paired MR image super-resolution network based on domain migration mentioned in the eleventh embodiment;

[0038] Figure 2 The network structure schematic diagram of the noise embedding network NEN mentioned in the eleventh embodiment;

[0039] Figure 3 The network structure schematic diagram of the degradation reconstruction network DRN mentioned in the eleventh embodiment;

[0040] Figure 4 The structure schematic diagram of the random feature extraction module RFEM mentioned in the eleventh embodiment;

[0041] Figure 5 The structure schematic diagram of the Transformer encoder-decoder mentioned in the eleventh embodiment. DETAILED DESCRIPTION

[0042] In order to make the advantages and beneficial effects of the technical solutions provided by the application more clear, the technical solutions provided by the application are further described in detail in combination with the drawings, and the specific:

[0043] Embodiment one, the embodiment provides a low-resolution image domain migration network construction method, the method comprises:

[0044] Step 1: collect a low-resolution MR image set and a down-sampling high-resolution MR image set of the image set;

[0045] Step 2: performing preliminary feature extraction on the low-resolution MR image set and the down-sampled high-resolution MR image set;

[0046] Step 3: constructing a network by taking the extracted low-resolution features of the low-resolution MR image set and the extracted high-resolution features of the down-sampled high-resolution MR image set as inputs.

[0047] In this embodiment, a low-resolution image domain migration network LDMN is constructed by taking the low-resolution MR image set and the down-sampled high-resolution MR image set as inputs. The low-resolution image domain migration network LDMN includes a CycleGAN structure composed of a noise embedding network NEN and a degradation reconstruction network DRN. The down-sampled high-resolution MR image set and the low-resolution MR image set are respectively taken as inputs of the noise embedding network NEN and the degradation reconstruction network DRN. The entire network is optimized and trained through a joint loss function. In this process, a k-space loss function is introduced as a joint loss function to jointly constrain the optimization process and reduce the generation of additional details in the domain migration process.

[0048] Embodiment Two, Combination Figures 2-5 This embodiment is a further limitation of the low-resolution image domain migration network construction method provided in Embodiment One. In Step 3, the process of constructing the network further includes:

[0049] Step 31: introducing degradation characteristics to the low-resolution features;

[0050] Step 32: performing degradation reconstruction on the low-resolution features with introduced degradation characteristics.

[0051] Specifically, a residual feature extraction module RFEM composed of multiple residual blocks is used to perform preliminary feature extraction on the input down-sampled high-resolution MR image and low-resolution MR image. The obtained feature maps are taken as the original inputs of the noise embedding network NEN and the degradation reconstruction network DRN. For the noise embedding network NEN, an encoder-decoder structure based on parallel Transformer is used. Multiple encoding-decoding modules are used to gradually fuse random noise features while preserving the original features of the input feature maps. Specifically, the encoder is used to embed the features of the input low-resolution MR image and random noise, and the fused encoding is reconstructed in the decoder. This introduces degradation characteristics to the down-sampled high-resolution MR image. This network simultaneously fuses the spatial domain characteristics of random noise and the down-sampled high-resolution MR image, and preserves the prior features of the down-sampled high-resolution MR image.

[0052] For the degradation reconstruction network DRN, a parallel Transformer encoder-decoder structure similar to the noise embedding network is adopted, and a multi-level codec is used to gradually fuse the feature maps output by the previous level Transformer decoder on the basis of preserving the original features of the input feature map. Specifically, the encoder is used to self-embed the feature map of the input low-resolution MR image, and the encoding embedding of different dimensional features is fused in the decoder to generate a low-resolution MR image with degradation features. The network simultaneously fuses the multi-level features of the input low-resolution MR image and preserves the prior features of the input low-resolution MR image.

[0053] The noise embedding network NEN and the degradation reconstruction network DRN are combined as generator networks with the discriminator network in the manner of GAN network, and the two GAN networks are connected end to end in the manner of CycleGAN network structure to form a low-resolution image domain migration network LDMN. As a preferred, the discriminator in the GAN network uses the PatchGAN strategy, that is, the quality of each small part of the image is evaluated and output as a probability matrix, instead of the original GAN network discriminator output which represents the overall image quality as a probability. The low-resolution image domain migration network composed of two parts of GAN network is jointly trained.

[0054] Preferably, the loss function of the low-resolution image domain migration network LDMN during training contains three parts: adversarial loss function, consistency loss function and k-space loss function, and the optimization function of the entire low-resolution image domain migration network is jointly constrained by the three parts:

[0055] ;

[0056] Among them the total loss function is represented by Ltotal, the adversarial loss function is represented by Ladv, the consistency loss function is represented by Lconsis, the k-space loss function is represented by Lk, , , and the weight values of the three loss functions are respectively represented by a, b and c.

[0057] Specifically, the adversarial loss function Ladv is the optimization function of the GAN network, including the adversarial loss function Ladv_NEN of the noise embedding network and the adversarial loss Ladv_DRN of the degradation reconstruction network, and the weight proportion of the two is the same, which is specifically represented as:

[0058] ,

[0059] ,

[0060] wherein, and denote generator and discriminator networks constituting the noise-embedded network NEN, denote a down-sampled high-resolution MR image set obtained by down-sampling the high-resolution MR image set, X denotes a low-resolution MR image, E(*) denotes an expectation value of a distribution function, denote a real low-resolution MR image sample distribution, denote a generated low-resolution MR image sample distribution containing degradation characteristics; and denote generator and discriminator networks constituting the degradation reconstruction network DRN, denote a low-resolution image, denote a high-resolution image.

[0061] Consistency loss function for preserving the consistency of input and output color composition, avoiding the generator to modify the color tone of the image autonomously, denoted as:

[0062] ;

[0063] Finally, the k-space loss function maps each pixel of the input MR image to k-space, i.e. the frequency domain, through a Fourier transform, and calculates the Euclidean distance between the reconstructed image and the original image in the frequency domain after the image passes through the cycle network, in order to preserve the detail information before and after the image is generated, which is specifically denoted as:

[0064] ,

[0065] wherein, denotes the Euclidean distance, denotes the Fourier transform process.

[0066] The low-resolution image domain migration network LDMN is optimized by joint constraint of the three-part loss function, and the weight of the k-space loss function in the overall loss function is changed according to the degradation degree of the MR image, so as to decide to preserve more or less original information.

[0067] Embodiment three, this embodiment is a further limitation of the low-resolution image domain migration network construction method provided in embodiment two, and the step 31 is specifically:

[0068] An encoder-decoder using parallel Transformer is adopted to gradually fuse random noise features on the basis of preserving the low-resolution features.

[0069] Embodiment four, this embodiment is a further limitation of the low-resolution image domain migration network construction method provided in embodiment two, and the step 32 is specifically:

[0070] The encoder-decoder using the parallel Transformer fuses different dimensional features of the low-resolution features step by step on the basis of preserving the low-resolution features.

[0071] Embodiment five, this embodiment is a further limitation of the low-resolution image domain migration network construction method provided in embodiment one, the down-sampled high-resolution MR image set of the image set is obtained by double cubic down-sampling of the low-resolution MR image set.

[0072] Embodiment six, this embodiment provides a low-resolution image domain migration network construction device, the device comprises:

[0073] Module 1: for collecting a low-resolution MR image set and a down-sampled high-resolution MR image set of the image set;

[0074] Module 2: for performing preliminary feature extraction on the low-resolution MR image set and the down-sampled high-resolution MR image set;

[0075] Module 3: for constructing a training network by taking the extracted low-resolution features of the low-resolution MR image set as input and the extracted high-resolution features of the down-sampled high-resolution MR image set as output.

[0076] Embodiment seven, in combination with Figure 1 This embodiment is described, this embodiment provides a non-paired MR image super-resolution method based on domain migration, the method comprises:

[0077] Step 4: collect a low-resolution MR image set;

[0078] Step 5: train a single-image super-resolution network by the low-resolution MR image set;

[0079] Step 6: according to the network constructed by the low-resolution image domain migration network construction method provided in any one of embodiments one to five, migrate the low-resolution MR image set, and then perform up-sampling operation by the single-image super-resolution network to obtain a large-scale high-resolution MR image.

[0080] This embodiment trains a single-image super-resolution network by using a high-resolution MR image set and a corresponding double cubic down-sampling low-resolution MR image set generated by the single-image super-resolution network, the single-image super-resolution network comprises a plurality of super-resolution networks, the input of each super-resolution network is a down-sampled high-resolution MR image, and finally the network with the optimal output performance is selected as a trained single-image super-resolution network for subsequent up-sampling.

[0081] Specifically: using the MR image training set to train each single-image super-resolution network respectively, optimizing the parameters of each single-image super-resolution network, completing the training of each super-resolution network, taking the low-resolution MR image as the input, respectively obtaining the super-resolution results of each super-resolution network after training, comparing and calculating the super-resolution image with the original high-resolution MR image, measuring the performance of each single-image super-resolution network, and finally selecting the network with the best performance as the up-sampling network to complete the domain migration of the down-sampled high-resolution MR image to the high-resolution MR image.

[0082] In the embodiment, the training method of each super-resolution network is as follows:

[0083] The low-resolution MR image in the MR image set is reconstructed into a high-resolution MR image by using the super-resolution network, the error between the high-resolution MR image and the corresponding high-resolution MR image in the MR image training set is calculated, the loss function when calculating the error is the l1 loss function, that is, the absolute difference between the corresponding pixels of the images, and the parameters of the super-resolution network are updated according to the calculated error. The training process uses the Adam optimizer to optimize the calculated error to obtain the optimized parameters of the super-resolution network.

[0084] Step 6 specifically includes: for each original low-resolution MR image, inputting it into the trained low-resolution image domain migration network LDMN, performing feature extraction and degradation reconstruction, and obtaining a domain migration result image at the output end of the degradation reconstruction network DRN, inputting the result image into the trained up-sampling network, and outputting a specific magnification high-resolution MR image after up-sampling at the terminal of the up-sampling network.

[0085] Embodiment eight, the embodiment provides a non-paired MR image super-resolution device based on domain migration, the device comprises:

[0086] Module 4: used for collecting a low-resolution MR image set;

[0087] Module 5: used for training a single-image super-resolution network through the low-resolution MR image set;

[0088] Module 6: used for performing migration on the low-resolution MR image set through the network constructed by the low-resolution image domain migration network construction device provided in embodiment six, and then performing up-sampling operation through the single-image super-resolution network to obtain a large-scale high-resolution MR image.

[0089] Embodiment nine, the embodiment provides a computer storage medium for storing a computer program, when the computer program stored in the storage medium is read by the processor of the computer, the computer executes the low-resolution image domain migration network construction method provided in any one of embodiments one to five or the non-paired MR image super-resolution method based on domain migration provided in embodiment seven.

[0090] Implementation Method 10: This implementation method provides a computer, including a processor and a storage medium, wherein the storage medium is used to store a computer program. When the processor processes the computer program, the computer executes the low-resolution image domain migration network construction method provided in any one of Implementation Methods 1 to 5 or the domain migration-based unpaired MR image super-resolution method provided in Implementation Method 7.

[0091] Implementation Method Eleven: Combination Figures 1-5 This embodiment describes a specific example of the domain-transfer-based unpaired MR image super-resolution method provided in Embodiment Seven. It demonstrates the advantages and benefits of the domain-transfer-based unpaired MR image super-resolution method provided by this invention, and also explains the above embodiments. Specifically:

[0092] Step 1: Obtain the MR image training set, including a low-resolution MR image set and a high-resolution MR image set with no corresponding relationship;

[0093] Step 2: Train the single-image super-resolution network using a high-resolution MR image set and the corresponding bicubic downsampled low-resolution MR image set. The single-image super-resolution network consists of multiple super-resolution networks, each receiving a downsampled high-resolution MR image as its input. The network with the best output performance is selected as the fully trained single-image super-resolution network for subsequent upsampling. This single-image super-resolution network serves as the High-Resolution Image Domain Migration Network (HDMN), and its network structure is as follows: Figure 1 As shown in the right half.

[0094] Step 3: Using the low-resolution MR image set and the downsampled high-resolution MR image set as input, train the Low-Resolution Image Domain Migration Network (LDMN). The LDMN network structure is as follows: Figure 1 The left half is shown. The Low-Resolution Image Domain Transfer Network (LDMN) includes a CycleGAN structure composed of a Noise Embedding Network (NEN) and a Degradation Reconstruction Network (DRN). Downsampled high-resolution MR image sets and low-resolution MR image sets are used as inputs to the NEN and DRN networks, respectively. The network structures of the NEN and DRN networks are shown below. Figure 2 , 3The entire network is optimized and trained by a joint loss function, and a k-space loss function is introduced as a joint loss function to jointly constrain the optimization process and reduce the generation of additional details in the domain migration process.

[0095] Fourth step: using the trained low-resolution image domain migration network LDMN to perform domain migration on the low-resolution MR image set to obtain the domain migration result images of the low-resolution MR image set to the down-sampled high-resolution MR image set, and then using the aforementioned trained single-image super-resolution network to up-sample the domain migration result images to complete the entire unpaired MR image super-resolution based on domain migration. The complete domain migration super-resolution process is shown in Figure 1 .

[0096] In the preferred embodiment, the second step includes using high-performance single-image super-resolution methods including Enhanced Deep Residual Network (EDSR), Deep Back-Projection Networks for Super-Resolution (DBPN), Residual Dense Network (RDN), and Feedback Network for Image Super-Resolution (SRFBN). The high-resolution MR image set obtained in the first step and the low-resolution MR image set generated by bicubic down-sampling are used to form a training pair, and the four single-image super-resolution networks are trained. The training settings are as follows:

[0097] EDSR: feature number 256, enhanced residual block number 32, residual factor 0.1;

[0098] DBPN: feature number 64, back-projection block number 7;

[0099] RDN: feature number 64, dense residual block number 16, dense residual layer number 8;

[0100] SRFBN: feature number 32, feedback step number 4, return group number 3.

[0101] All networks are trained using ADAM optimizer with learning rate set to 0.0001, and a total of 1000 rounds of training are performed, and the learning rate is halved at the 200th, 400th, 600th and 800th round respectively, and the respective optimal performance model is taken for comparison, and the performance test uses indicators including peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), and the peak signal-to-noise ratio can be expressed as:

[0102] ;

[0103] wherein, represents the maximum value of the color of the image point, for example, if each sampling point is represented by 8 bits, it should be 255, and MSE represents the mean square error, which can be expressed as:

[0104] ;

[0105] wherein , represents the length and width of the input image, , respectively represent two input images, and respectively represent two process quantities. The structural similarity SSIM can be expressed as:

[0106] ;

[0107] wherein, , , respectively represent brightness, contrast and structural comparison, , and respectively represent the proportion of the three characteristics in the SSIM measurement, which is generally 1:1:1.

[0108] The peak signal-to-noise ratio PSNR and the structural similarity SSIM are both larger, which represents that the network super-resolution performance is better. Taking PSNR and SSIM as indicators, the optimal performance super-resolution feedback network SRFBN is finally selected as the single-image super-resolution network in the high-resolution MR image super-resolution network HDMN.

[0109] The third step comprises:

[0110] The down-sampling image set of the real low-resolution MR image set and the real high-resolution MR image set is cropped to have the same size, and for this embodiment, the cropping size is 48*48. The obtained image set is used as a training set to train the low-resolution image domain migration network LDMN, and the network structure diagram of the LDMN is as shown in Figure 1As shown in the left half (Low-resolution Image Domain Migration Network), the specific process of low-resolution image domain migration is as follows:

[0111] For Noise Embedded Networks (NENs), such as Figure 2 It first focuses on high-resolution MR images. Downsampling was performed using a bicubic downsampling method to obtain downsampled high-resolution MR images. The image is sized to match the input low-resolution MR image and used as input to the noise embedding network NEN, along with a random noise image n of the same size. Each image first passes through a convolutional layer to obtain a primary feature map. Here, the feature number is set to 64. For downsampled high-resolution MR images... The feature map is further processed using the Residual Feature Extraction Module (RFEM). The structure of the Residual Feature Extraction Module (RFEM) is as follows: Figure 4 As shown, it is composed of multiple residual blocks connected in a residual form. Each residual block includes a convolution-ReLU-convolution connection structure, and the two blocks are connected end to end to realize residual connection. Each residual block is also connected in a residual form. In this embodiment, a total of 5 residual blocks ResBlock are used to form each residual feature extraction module RFEM.

[0112] In this embodiment, three parallel residual feature extraction modules (RFEM) are used to process downsampled high-resolution MR images. The feature maps are further used for feature extraction and then used as feature inputs for the parallel Transformer codec. The structure of the parallel Transformer codec is as follows: Figure 5 As shown, it first flattens the input image by mapping the input feature map to a 1024-dimensional vector for each 8*8*16 feature map. Then, it passes through a linear layer to map the 1024-dimensional vector to a 512-dimensional vector, which is used as the input to the Transformer encoder. In the Noise Embedding Network (NEN), the high-resolution MR image is first downsampled... The feature map of the input random noise is flattened and used as input to Encoder-1 and Encoder-n in the first-stage Transformer codec to downsample the high-resolution MR image. The Encoder output of the feature map is taken as the Q, K value of the Transformer decoder multi-head attention module, and the Encoder output of the random noise feature map is taken as the V value of the decoder multi-head attention module, so as to obtain the output feature map of the first-level Transformer encoder-decoder. And again, it is flattened as the Q, K value of the second-level Transformer decoder, and the feature map of the down-sampled high-resolution MR image is input into the encoder Encoder-2 of the second-level Transformer encoder-decoder, and the output thereof is taken as the V value of the decoder multi-head attention module, so as to obtain the output feature map of the second-level Transformer encoder-decoder. And it is flattened as the Q, K value of the third-level Transformer decoder, and the feature map of the down-sampled high-resolution MR image is input into the encoder Encoder-3 of the third-level Transformer encoder-decoder, and the output thereof is taken as the V value of the decoder multi-head attention module, so as to obtain the output feature map of the third-level Transformer encoder-decoder. After the channel number of this feature map is restored through a convolutional layer, the output image of the noise embedding network NEN is obtained , and this process can be represented as:

[0113] ,

[0114] ,

[0115] ,

[0116] ,

[0117] wherein, , , are the output feature maps of the first, second and third-level Transformer encoder-decoders respectively, represents a convolutional operation, the three parameters in the parentheses represent the Q, K and V values in the Transformer encoder-decoder respectively.

[0118] For the degenerative reconstruction network DRN, the structure used is the same as that of the noise embedding network NEN, and the only difference is that the random noise n in the noise embedding network is replaced by the low-resolution MR image, and this process can be represented as:

[0119] ,

[0120] ,

[0121] ,

[0122] ,

[0123] Finally, the degraded reconstructed image In the low-resolution image super-resolution network LDMN, the generator network is composed of a noise embedding network NEN and a degraded reconstruction network DRN, and the discriminator is used to distinguish the low-resolution MR image and the input low-resolution MR image, and the discriminator is used to distinguish the down-sampled high-resolution MR image and the degraded reconstructed image . The two-part GAN network is connected in a CycleGAN form, and together constitutes the low-resolution image super-resolution network LDMN. In this embodiment, the training of the low-resolution image super-resolution network LDMN is set as follows:

[0124] The network is optimized using the Adam optimizer, where the bias is set to 0.5, is set to 0.999, the blur factor ε is set to 10-8, the number of selected samples for one training is 8, the initial learning rate is set to 0.0001, and thereafter the learning rate is halved every 100 training rounds. The entire network is trained for 500 rounds.

[0125] In terms of loss function, the adversarial loss function, the consistency loss function and the k-space loss function are used as joint optimization functions. As preferred, the weights of the three are set to =1, =2, =2.

[0126] The fourth step includes: inputting the real low-resolution MR image set into the trained low-resolution image domain migration network LDMN as the input of the network, obtaining the output image of the degraded reconstruction network DRN, taking the output image as the input of the high-resolution image domain migration network HDMN, and using the single-image super-resolution network SRFBN selected in the second step to perform up-sampling to obtain the output image, which is the high-definition super-resolution MR image. The process can be represented as:

[0127] ,

[0128] wherein, represents the super-resolution MR image, represents an input low-resolution MR image, LDMN and HDMN represent two parts of the domain migration network in the method respectively, a low-resolution image domain migration network and a high-resolution image domain migration network. The results of the super-resolution of the low-resolution MR image in the selected embodiment are shown in Table 1, and the optimal performance is highlighted. The experimental results prove the effectiveness of the method proposed in the embodiment.

[0129] Table 1. MR image super-resolution results of the selected embodiment using the proposed domain migration network

[0130]

[0131] The embodiment aims at the data distribution difference problem and the image distortion problem existing in the MR image super-resolution task of the super-resolution technology. First, four representative single-image super-resolution methods (the number of deep learning networks: EDSR, DBPN, RDN, and SRFBN) are used for super-resolution of MR images, and experiments are performed on three super-resolution scales of the knee part MR image set. The quantitative results show that the SRFBN network is more suitable for the MR image super-resolution task, and it is used as the high-resolution domain migration network HDMN. Second, the low-resolution domain migration network LDMN is trained using real low-resolution MR images and real high-resolution MR images. The real low-resolution MR image set is used as the input, and the trained model is used for domain migration of the real low-resolution MR image set. Finally, the domain migration output image of the low-resolution domain migration network LDMN is used as the input, and the trained high-resolution image domain migration network is used for up-sampling to obtain the super-resolution result image. The multi-scale super-resolution experimental results on the knee joint MR image dataset show that the non-paired MR image super-resolution method based on domain migration proposed in the embodiment has a significant performance improvement compared with the super-resolution method directly applied to the MR image, which proves the effectiveness of the method proposed in the embodiment.

[0132] The technical solutions provided by the present application are described in detail through several specific embodiments above. However, the above several specific embodiments are only used to highlight the advantages and benefits of the present application, and should not be used as a limitation of the present application. Any reasonable changes and improvements, reasonable combinations and replacements of embodiments, etc. based on the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing a low-resolution image domain transfer network, characterized in that, The method includes: Step 1: Acquire a set of low-resolution MR images and a set of downsampled high-resolution MR images from the acquired set; Step 2: Perform preliminary feature extraction on the low-resolution MR image set and the downsampled high-resolution MR image set; Step 3: Use the low-resolution features of the extracted low-resolution MR image set and the high-resolution features of the extracted downsampled high-resolution MR image set as input to construct a network; The method involves using a low-resolution MR image set and a downsampled high-resolution MR image set as inputs to construct a Low-Resolution Image Domain Transfer Network (LDMN). The LDMN includes a CycleGAN structure composed of a Noise Embedding Network (NEN) and a Degradation Reconstruction Network (DRN). The downsampled high-resolution MR image set and the low-resolution MR image set are used as inputs to the NEN and DRN, respectively. The entire network is optimized and trained using a joint loss function. In this process, a k-space loss function is introduced as a joint loss function to jointly constrain the optimization process. Step 3, the process of constructing the network also includes: Step 31: Introduce degradation properties to the low-resolution feature; Step 32: Perform degradation reconstruction on the low-resolution features that introduce degradation characteristics; For the Noise Embedding Network (NEN), the network uses a parallel Transformer-based encoder-decoder structure, which uses a multi-level encoder-decoder to progressively fuse random noise features while preserving the original features of the input feature map. For the Degenerate Reconstruction Network (DRN), a parallel Transformer encoder-decoder structure similar to that of the Noise Embedding Network is adopted. A multi-level encoder-decoder is used to fuse the feature maps output by the previous Transformer decoder level by level while preserving the original features of the input feature map.

2. The method for constructing a low-resolution image domain transfer network according to claim 1, characterized in that, Step 31 specifically involves: A parallel Transformer encoder and decoder are used to progressively fuse random noise features while preserving the low-resolution features.

3. The method for constructing a low-resolution image domain transfer network according to claim 2, characterized in that, Step 32 specifically involves: A parallel Transformer encoder and decoder are used to progressively fuse the different dimensions of the low-resolution features while preserving the low-resolution features.

4. The method for constructing a low-resolution image domain transfer network according to claim 1, characterized in that, The downsampled high-resolution MR image set of the image set is obtained by downsampling the low-resolution MR image set twice cubically.

5. A device for constructing a low-resolution image domain transfer network, characterized in that, The device includes: Module 1: Used to acquire a set of low-resolution MR images and a set of downsampled high-resolution MR images of the set of images; Module 2: Used to perform preliminary feature extraction on the low-resolution MR image set and the downsampled high-resolution MR image set; Module 3: Used to construct a training network by taking the low-resolution features of the extracted low-resolution MR image set as input and the high-resolution features of the extracted downsampled high-resolution MR image set as output; The method involves using a low-resolution MR image set and a downsampled high-resolution MR image set as inputs to construct a Low-Resolution Image Domain Transfer Network (LDMN). The LDMN includes a CycleGAN structure composed of a Noise Embedding Network (NEN) and a Degradation Reconstruction Network (DRN). The downsampled high-resolution MR image set and the low-resolution MR image set are used as inputs to the NEN and DRN, respectively. The entire network is optimized and trained using a joint loss function. In this process, a k-space loss function is introduced as a joint loss function to jointly constrain the optimization process. Step 3, the process of constructing the network also includes: Step 31: Introduce degradation properties to the low-resolution feature; Step 32: Perform degradation reconstruction on the low-resolution features that introduce degradation characteristics; For the Noise Embedding Network (NEN), the network uses a parallel Transformer-based encoder-decoder structure, which uses a multi-level encoder-decoder to progressively fuse random noise features while preserving the original features of the input feature map. For the Degenerate Reconstruction Network (DRN), a parallel Transformer encoder-decoder structure similar to that of the Noise Embedding Network is adopted. A multi-level encoder-decoder is used to fuse the feature maps output by the previous Transformer decoder level by level while preserving the original features of the input feature map.

6. A domain-transfer-based unpaired MR image super-resolution method, characterized in that, The method includes: Step 4: Acquire a set of low-resolution MR images; Step 5: Train a single-image super-resolution network using the low-resolution MR image set; Step 6: The network constructed according to any one of claims 1-5 is used to transfer the low-resolution MR image set, and then the single-image super-resolution network is used to perform upsampling operation to obtain a large-scale high-resolution MR image.

7. A domain-transfer-based unpaired MR image super-resolution device, characterized in that, The device includes: Module 4: Used for acquiring low-resolution MR image sets; Module 5: Used to train a single-image super-resolution network using the low-resolution MR image set; Module 6: A network constructed by the low-resolution image domain migration network construction device according to claim 6, which migrates the low-resolution MR image set and then performs upsampling operation through the single-image super-resolution network to obtain a large-scale high-resolution MR image.

8. A computer storage medium for storing computer programs, characterized in that, When the computer program stored in the storage medium is read by the computer's processor, the computer executes the low-resolution image domain transfer network construction method according to any one of claims 1-5 or the domain transfer-based unpaired MR image super-resolution method according to claim 7.

9. A computer, comprising a processor and a storage medium, said storage medium for storing computer programs, characterized in that, When the processor processes the computer program, the computer executes the low-resolution image domain transfer network construction method according to any one of claims 1-5 or the domain transfer-based unpaired MR image super-resolution method according to claim 7.

Images