DCE-MRI image generation method based on W-type network structure with spatial and temporal information

Abstract

The present invention discloses a DCE-MRI image generation method based on a W-type network structure of space and time information, comprising the following steps: obtaining a training data set; training a W-type neural network structure model according to the training data set, and collecting three-dimensional MRI image data I0 of the T0 phase before contrast agent injection, inputting the image data I0 into the trained W-type neural network structure model based on space and time information, and generating a four-dimensional DCE-MRI time-space sequence image I0 at Ti and Ti+1 after contrast agent injection. i with I i+1 Aiming at the relationship between adjacent phase DCE-MRI images, this method proposes a W-type image generation network based on spatial and temporal information. The network simultaneously extracts low-dimensional and high-dimensional features of adjacent phase images, and uses these features and their related information to generate four-dimensional DCE-MRI time-space sequence images.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a DCE-MRI image generation method based on a W-type network structure of space and time information. Background Art

[0002] Imaging examinations can not only detect the presence of lesions and locate the location of lesions through images, but also observe the physical characteristics of lesions such as size, morphology and density. Among them, plain scan magnetic resonance imaging (MRI) can detect tumors and their invasion of surrounding tissues based on the location and morphology of lesions and their relationship with surrounding tissues. However, since metabolic changes within tumor tissues may occur much earlier than their morphological changes, the diagnosis of tumors by plain scan MRI may differ from the actual situation. Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) after contrast agent injection obtains continuous dynamic enhanced images of tissues before, during and after contrast agent injection through continuous and rapid sequence imaging, thereby reflecting the physiological metabolic changes of lesion tissues.

[0003] Injection of contrast agents is considered a routine and safe examination method, but patients who are allergic to contrast agents (such as iodine) or have poor heart, liver, lung and kidney function cannot undergo DCE-MRI examinations. If DCE-MRI images can be generated based on plain MRI images using computer post-processing technology instead of multiple MRI scans after contrast agent injection, it can not only provide patients (especially those who cannot inject contrast agents due to physical conditions) with physiological metabolic changes in lesion tissues, but also eliminate the need for injection of contrast agents for secondary MRI dynamic scans. Due to the above advantages, research on computer post-processing technology for generating DCE-MRI images is becoming a hot topic for scholars from various countries.

[0004] At present, no researchers have applied deep learning methods to the generation of DCE-MRI images at different phases, but it has been widely used to generate unobstructed, higher-quality medical images. Yang et al. used cascaded CNNs in the gradient domain to generate soft tissue images without bone occlusion based on normal chest X-ray images. Professor Dinggang Shen's team at the University of North Carolina used CNNs to generate 7T brain MRI, CT, and PET images based on conventional plain scan MRI brain images. With the introduction of super-resolution image reconstruction algorithms based on CNNs, Oktaty et al. reconstructed high-resolution cardiac MRI images based on low-resolution three-dimensional MRI images. In addition, deep learning methods can not only reconstruct images with spatial grayscale information, but also infer the imaging parameters involved in generating high-resolution images. Although deep learning methods have achieved remarkable results in a variety of image generation fields, these algorithms all use CNNs to generate two-dimensional or three-dimensional images. Summary of the invention

[0005] In view of this, the present invention discloses a DCE-MRI image generation method based on a W-type network structure of spatial and temporal information, which can extract low-dimensional and high-dimensional features of adjacent phase images, and use these features and their related information to generate a four-dimensional DCE-MRI image.

[0006] The technical solution provided by the present invention is specifically a DCE-MRI image generation method based on a W-type network structure of spatial and temporal information, characterized in that the training process of the W-type neural network structure model includes:

[0007] Get the training dataset;

[0008] Training a W-type neural network structure model according to the training data set; the W-type neural network structure model is composed of a contraction path and two ascending expansion paths;

[0009] The loss function is calculated based on the difference between the outputs of the two ascending paths and the gold standard breast DCE-MRI images of two adjacent phases;

[0010] Update the values ​​of the W-type network parameters using the back propagation method according to the value of the loss function;

[0011] Through multiple iterative updates, the loss function value gradually converges, and finally the training of the W-type network model is completed.

[0012] The testing process of the W-type neural network structure model includes: collecting three-dimensional MRI image data I0 at the T0 phase before contrast agent injection, inputting the image data I0 into the trained W-type neural network structure model based on space and time information, and generating four-dimensional DCE-MRI time-space sequence images I0 at Ti and Ti+1 after contrast agent injection.i with I i+1 .

[0013] The contraction path simultaneously extracts low-dimensional features of two adjacent phase images, and the two following expansion paths respectively extract high-dimensional features of two adjacent phase images. The output of the convolutional layer on the contraction path is respectively connected to the input of the convolutional layer on the two expansion paths through jump connections, and the inputs of the convolutional layers on the two expansion paths are connected.

[0014] According to the different feature volume resolutions, the path of the W-type neural network structure is divided into different states. Each state consists of 2 blocks. Each block contains a 3×3×3 convolutional layer followed by a BN layer and a ReLu layer. On the contraction descending path, the second block is connected to a maximum downsampling layer with a kernel size of 2×2×2; the last layer of the two expansion ascending paths is a 1×1×1 convolutional layer, which outputs a set of three-dimensional MRI images respectively.

[0015] According to the training data set including training data of 5 different time phases after injection of contrast agent, 3 different W-type neural networks are trained to generate images of 5 different time phases after injection of contrast agent.

[0016] In the training of the W-type neural network structure model using the training data set, the mean absolute error MAE is used as the regression loss function, which is defined as follows:

[0017]

[0018] Where M is the number of images of adjacent phases generated simultaneously, W, H and D represent the length, width and height of the three-dimensional MRI enhanced image respectively, and V gi is the pixel value of the gold standard MRI image at the i-th phase, V pi is the pixel value of the corresponding generated image.

[0019] The present invention provides a DCE-MRI image generation method based on a W-type network structure of spatial and temporal information. The method takes into account the close relationship between MRI images of different phases and different layers of the same phase, and proposes a W-type image generation network based on spatial and temporal information for the relationship between DCE-MRI images of adjacent phases. The network simultaneously extracts low-dimensional and high-dimensional features of adjacent phase images, and uses these features and related information to generate four-dimensional DCE-MRI time-space sequence images.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a W-shaped network structure provided in an embodiment disclosed in the present invention;

[0024] Figure 2 The result representation of an image generated by applying the W-type network structure provided by the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0026] In the prior art, commonly used recurrent neural networks such as long short-term memory networks add time series information to the network, but they only use various simple linear "gates" to consider the impact of past phase inputs on the current state. When generating DCE-MRI images with complex content and uneven grayscale, the results may not be ideal. In addition, the impact of the different positions of the current phase input in the entire time series image is not considered. Since the attenuation of the contrast agent is not the same at the same time in different stages, the changes in the images between adjacent phases are also different, and the same features cannot be used to describe the difference in such changes. Accordingly, this embodiment aims at the relationship between adjacent phase DCE-MRI images, and the proposed three-dimensional W-type network structure can make full use of the spatial and temporal information of the image to generate a four-dimensional DCE-MRI image.

[0027] A DCE-MRI image generation method based on a W-type network structure of spatial and temporal information, wherein the training process of the W-type neural network structure model includes:

[0028] Get the training dataset;

[0029] like Figure 1 As shown, a W-type neural network structure model is trained according to the training data set;

[0030] The model consists of a contraction path and two ascending paths. The loss function is calculated based on the difference between the output of the two ascending paths and the gold standard breast DCE-MRI images of the two connected phases, and then the value of the W-type network parameters is updated using the back propagation method based on the value of the loss function. Through multiple iterative updates, the loss function value gradually converges. At this point, the training of the W-type network model is completed.

[0031] Collect the three-dimensional MRI image data I0 at the T0 phase before contrast agent injection, input the image data I0 into the trained W-type neural network structure model based on space and time information, and generate the four-dimensional DCE-MRI time-space sequence image I0 at Ti and Ti+1 after contrast agent injection. i with I i+1 The W-type network structure is as follows: Figure 1 As shown in the figure, the numbers next to the blocks indicate the number of convolution kernels. In the entire network, each convolution layer is followed by a BN layer and a ReLU activation layer, which are omitted in the figure.

[0032] The W-type neural network structure model includes a contraction path and two consecutive expansion paths. This symmetrical structure forms a W-type fully connected network. The contraction path extracts the low-dimensional features of the two adjacent phase images at the same time, and the two following expansion paths extract their high-dimensional features respectively. The output of the convolution layer on the contraction path is connected to the input of the convolution layer on the two expansion paths through jump connections, and the input of the convolution layer on the two expansion paths is connected, so that the network can simultaneously use the low-dimensional and high-dimensional features of the adjacent phase images to generate images.

[0033] According to the different feature volume resolutions, the path of the W-type neural network structure is divided into different states. Each state consists of 2 blocks. Each block contains a 3×3×3 convolution layer followed by a BN layer and a ReLu layer. On the contraction (expansion) path, the second block is connected to a maximum downsampling (upper convolution) layer with a kernel size of 2×2×2, so that the size of the feature volume in three directions is halved (doubled). The last layer of the two expansion paths is a 1×1×1 convolution layer, which changes the number of feature maps from 16 to 1, and outputs a set of three-dimensional MRI images respectively.

[0034] In this embodiment, three W-type neural network structure models are trained according to the training data set to ensure the generation of five sets of enhanced MRI images of different phases. At the same time, an attempt is made to increase the number of right-side expansion paths, reduce the number of training networks, and more fully utilize the mutual information between multi-phase images to simultaneously generate enhanced MRI images of more phases.

[0035] According to the training data set including training data of 5 different time phases after injection of contrast agent, training 3 different W-type neural networks can realize the generation of all DCE-MRI images. A set of DCE-MRI images includes 6 images of different time states, including 1 before injection of contrast agent, and then scanning and imaging once every 90 seconds, for a total of 5 times. A W-type network can generate images of 2 imaging after injection of contrast agent. Therefore, 3 networks can generate 5 images, realizing the generation of all images.

[0036] In the training data set for training the W-type neural network structure model, the mean absolute error MAE is used as the regression loss function. The mean absolute error (MAE) is a commonly used regression loss function, which is defined as the average of the absolute difference between the target value and the predicted value. It represents the average error amplitude of the predicted value, and the direction of the error does not need to be considered. When calculating the MAE of the W-type network, multiple groups of generated three-dimensional MRI images need to be considered simultaneously, and they are closely related. It is expected that when each group of images is generated accurately, the MAE value is the smallest and the generation result is the best. Accordingly, the loss function based on MAE is defined as

[0037]

[0038] Where M is the number of images of adjacent phases generated simultaneously, W, H and D represent the length, width and height of the three-dimensional MRI enhanced image respectively, and V gi is the pixel value of the gold standard MRI image at the i-th phase, V pi is the pixel value of the corresponding generated image.

[0039] The W-type network structure provided above was used to generate dynamic enhanced MRI images, and the following experiments were conducted: The experimental data were breast DCE-MRI T1-weighted fat-suppressed images of 60 patients (the patient's privacy information will be removed). Each patient was scanned once before the injection of contrast agent (gadopentic acid dimeglumine) to generate image I0, and scanned 5 times at an interval of 90s after the injection to generate images I1-I5. The size of each set of images was 784×784×180 pixels, and its resolution in the X, Y and Z directions was 0.45mm, 0.45mm and 1.00mm respectively. The images were trilinearly interpolated to make their resolution in the three directions 1.00mm.

[0040] Figure 2The original images and prediction generation results of a group of breast DCE-MRI images in different phases. The generated images in different phases can reflect the real breast tissue structure, and the generation results are satisfactory. Experiments were conducted on 30 groups of images in the database, and the average peak signal-to-noise ratio between the generated images and the real images was used to evaluate the image generation quality. The peak signal ratios of the generated images in the five phases can reach 11.15, 10.98, 11.04, 10.92 and 10.89 respectively. It can be seen that the method provided by this embodiment is applied, considering that MRI images in different phases and different layers of the same phase are closely related, and four-dimensional DCE-MRI time-space sequence images are generated based on three-dimensional MRI plain scan images. The method is applied to realize the use of computer post-processing technology to replace multiple MRI scanning operations after injection of contrast agents to generate DCE-MRI images, which can not only provide patients (especially patients who cannot inject contrast agents due to physical conditions) with physiological metabolic changes in lesion tissues, but also do not need to inject contrast agents for secondary MRI dynamic scanning.

[0041] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

[0042] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

Claims

1. A DCE-MRI image generation method based on a W-type network structure with spatial and temporal information, characterized in that: The training process of the W-type network structure includes: Get the training dataset; Training a W-type neural network structure model according to the training data set; The W-type neural network structure model is composed of a contraction path and two ascending expansion paths; The loss function is calculated based on the difference between the outputs of the two ascending paths and the gold standard breast DCE-MRI images of two adjacent phases; Update the values ​​of the W-type network parameters using the back propagation method according to the value of the loss function; Through multiple iterative updates, the loss function value gradually converges, and finally the training of the W-type network model is completed; The testing process of the W-type neural network structure model includes: collecting three-dimensional MRI image data I0 at the T0 phase before contrast agent injection, inputting the image data I0 into the trained W-type neural network structure model based on space and time information, and generating four-dimensional DCE-MRI time-space sequence images I0 at Ti and Ti+1 after contrast agent injection. i with I i+1 ; The contraction path simultaneously extracts low-dimensional features of two adjacent phase images, and the two following expansion paths respectively extract high-dimensional features of two adjacent phase images, and the output of the convolution layer on the contraction path is respectively connected to the input of the convolution layer on the two expansion paths through a jump connection, and the input of the convolution layer on the two expansion paths is connected; According to different feature volume resolutions, the path of the W-type neural network structure is divided into different states. Each state consists of 2 blocks. Each block contains a 3×3×3 convolutional layer followed by a BN layer and a ReLu layer. On the contraction descending path, the second block is connected to a maximum downsampling layer with a kernel size of 2×2×2. The last layer of the two expansion ascending paths is a 1×1×1 convolutional layer, which then passes through the activation function Tanh to output a set of three-dimensional MRI images. According to the training data set, three different W-type neural networks are trained to generate images of five different phases after injection of contrast agent; In the training of the W-type neural network structure model using the training data set, the mean absolute error MAE is used as the regression loss function, which is defined as follows: Where M is the number of images of adjacent phases generated simultaneously, W, H and D represent the length, width and height of the three-dimensional MRI enhanced image respectively, and V gi is the pixel value of the gold standard MRI image at the i-th phase, V pi is the pixel value of the corresponding generated image.

Images