Medical image generation method based on deep 3D network and transfer learning

Abstract

The present invention discloses a medical image generation method based on deep 3D network and transfer learning. The present invention trains the model by means of image preprocessing, data set preparation, data enhancement, model training, observation model, grid optimization, retraining, rechecking, data testing, evaluation index and result discussion, and finally uses the model to generate medical images, which relates to the technical field of medical image processing. The present invention is scientific and reasonable, safe and convenient to use. In the present model, the original image is divided into several regions, and the Softmax value of each domain is calculated to obtain the average value. The model of the present invention only needs two types of medical images and does not have too many requirements, which is meaningful for clinical practice. The model of the present invention is a conversion between two domains, and the conversion of two images can be performed at the same time. In terms of conversion efficiency, the model of the present invention has higher conversion efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a medical image generation method based on deep 3D network and transfer learning. Background Art

[0002] With the rise of deep learning, more and more deep learning technologies are applied in daily life. As for medical image generation, there are many methods in this field.

[0003] For example, the fuzzy C-means clustering method uses a unique clustering concept, which makes it difficult to generate high-quality images because the images generated by this method do not determine the attribution of a certain part of the tissue. The details of the image are difficult to present clearly, resulting in a relatively blurry image. On the other hand, with the development of GAN networks, many methods are built on deep GAN networks.

[0004] For example, Pix2Pix is ​​a GAN network based on UNet. However, this method requires two types of paired medical images for deep neural network fitting. The quality of the generated images is high, but the data requirements are very strict. The two types of medical images must be paired one by one. In other words, two types of medical images are required, and the shooting environments of the two types of images should be strictly consistent, so as to ensure that the network can learn "knowledge";

[0005] At the same time, a small number of generated images are relatively random and have a large gap with the real images. The DualGAN network is a network model based on transfer learning. Since the model cuts a 3D image into multiple 2D slices, simulates and generates each slice, and then merges the generation results of each slice, each time it generates an image, a strip-shaped "shadow mark" will be generated in the 3D image;

[0006] In order to solve the above problems and ensure the quality of image generation, this paper proposes a method based on transfer learning, which can generate images with relatively high quality. On the other hand, since this method is based on transfer learning, it does not require a one-to-one paired dataset, but only requires a dataset of two types of medical images, which has important practical significance for the application. Finally, this method can solve the problem of stripe "shadow marks" produced by DualGAN generated images. Summary of the invention

[0007] The purpose of the present invention is to provide a medical image generation method based on deep 3D network and transfer learning to solve the problems raised in the prior art.

[0008] To achieve the above object, the present invention provides the following technical solution: a medical image generation method based on deep 3D network and transfer learning, the generation method comprising the following steps:

[0009] S1. Image preprocessing: determine medical images in different domains as datasets;

[0010] S2, make data set: filter and transform the data set of S1;

[0011] S3, data enhancement: expand the data set produced by S2;

[0012] S4, model training: train the model using the data set enhanced by S3;

[0013] S5. Observation model: observe the training process of the model and determine the range of model parameters;

[0014] S6, grid optimization: determine the range of optimal values ​​of model training parameters;

[0015] S7, retraining: retraining the model using the parameters within the range of the optimal values ​​obtained in S6;

[0016] S8, recheck: observe the model training process again to determine whether the model has achieved the best fit;

[0017] S9. Data testing: Use the test set to test the trained model and check the test results.

[0018] S10, evaluation index: through the test results obtained in S9, multiple evaluation indexes are calculated using data formulas and images;

[0019] S11. Result discussion: Visually check the test results and evaluation index results.

[0020] According to the above technical solution, S1, image preprocessing: determine medical images in different domains, for example, medical images in the medical CT domain and the medical MR domain, select a number of images with clear image details and consistent size for training sets, and select a number of images as test sets, wherein the training set and the test set constitute a data set;

[0021] S2. Prepare a data set: filter the data set of S1, remove blurry and artifact unclear images, place images of different domains in different folders, mark domains of different folders for model training, and convert the original data set images into types, for example, convert the original .png images into .gif sequences, because the model of the present invention is a three-dimensional model, and convolution must be performed on the three-dimensional space;

[0022] S3, data enhancement: expand the data set produced by S2. The expansion adopts the means of rotating and folding the original data set. The purpose is to make the model learn the most and increase the generalization ability of the model.

[0023] S4, model training: Use the data set enhanced by S3 to train the model, and put images from different domains into the model as input of the model image;

[0024] S5. Observation model: observe the training process of the model and determine the range of model parameters;

[0025] S6, grid optimization: Since the model may have different effect deviations for a specific data set, grid optimization is required to determine the range of optimal values ​​of the model training parameters. The range of the optimal value has been determined in step S5. The optimal values ​​of the model parameters are gradually determined within this range, which can help determine the optimal parameter values ​​for a specific model and data set, thereby producing the optimal effect.

[0026] S7, retraining: retraining the model using the parameters within the range of the optimal values ​​obtained in S6;

[0027] S8, recheck: observe the model training process again to determine whether the model has reached the best fit. If it has not reached the best fit, readjust the model parameters;

[0028] S9. Data testing: Use the test set to test the trained model and check the test results.

[0029] S10, evaluation index: Based on the test results obtained in S9, use the data formula and the image to calculate the peak noise ratio and structural similarity;

[0030] S11. Result discussion: Visually check the test results and evaluation index results.

[0031] According to the above technical solution, the model consists of two pairs of generation networks and discrimination networks. The task of the generation network is to continuously generate forged data to deceive the discrimination network, and the task of the discrimination network is to identify the forged data of the generation network.

[0032] According to the above technical solution, the model is restricted by a loss function, and the loss function is as follows:

[0033] Loss(front)=||bG A (G B (b,x),x1)||;

[0034] Loss(after)=||aG B (G A (a,z),z1)||;

[0035] Among them, Loss(front) represents the first half of the generated network, Loss(back) represents the second half of the generated network, b represents the image input by the first image domain, and image b is transferred through the generator G through transfer learning. B Convert to x, and then pass x through the generator G A Converted to x1, on the other hand, the second image domain input image a is transferred through the generator G through transfer learning A Converted to z, and then z passes through the generator G B Convert to z1.

[0036] According to the above technical solution, the first half represents the analysis path, and the second half represents the synthesis path.

[0037] According to the above technical solution, in step S10, the peak noise ratio includes a mean square error factor, and the mean square error factor is given a clean image I and a noise image K of size m*n, and the clean image I and the noise image K represent the real image and the image generated by the model respectively. The mean square error factor (MSE) is defined as:

[0038]

[0039] The peak-to-noise ratio is a definition related to the mean square error, with the unit being db. Some parameters of the formula are explained below: is the maximum possible pixel value of the picture, and the peak noise ratio PSNR is defined as follows:

[0040]

[0041] According to the above technical solution, the structural similarity is an important indicator for evaluating images, which is composed of three measurement factors, namely brightness L, contrast C and structure S. The formulas of the three measurement factors are as follows:

[0042]

[0043]

[0044]

[0045] Among them, u x is the mean of x, and similarly, u y is the mean of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y;

[0046] The structural similarity SSIM is achieved by coupling three measurement factors. The definition of structural similarity SSIM is as follows:

[0047] SSIM(x,y)=[L(x,y) α C(x,y) β S(x,y) χ ];

[0048] If α, β, and χ are all set to 1, the original formula can be simplified. After substituting the three measurement factors, the simplified structural similarity SSIM formula can be defined as follows:

[0049]

[0050] According to the above technical solution, when rotating, from point A (x0, y0) to point B (x', y'), the length of the chord is R, the angle of rotation is a, and the angle between the chord at point A and the X-axis is b, we can get:

[0051] x0 = R*cosb;

[0052] y0=R*sinb;

[0053] x' = R*cos(ba);

[0054] y' = R*sin(ba);

[0055] Rotate the original image 45 degrees left and right to obtain a data-enhanced image;

[0056] Folding is to fold the original image A into B by folding it upside down.

[0057] By rotating and flipping, the content of the data set is expanded, making the training of the model more sufficient and making the images converted by the trained model closer to the real images.

[0058] Folding is to fold the original image A into B by folding it up and down. In the present invention, not only left-right folding but also up-down folding is adopted, that is, the original image A is folded downward once and folded left-right once.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] In traditional method theory, there are many methods for the simulation generation of medical images, but this method is different from others in the following aspects:

[0061] First, the ideas are different. Neither the fuzzy clustering method nor the classic GAN adversarial generation idea uses the idea of ​​transfer learning. Most of them are "point-to-point" algorithm generation. In practical applications, such application scope is very limited.

[0062] Secondly, the network model is different. In many medical image generation models, a 2D model is often used for image generation. The image is first cut into slices, and a simulated image is generated using a 2D network model. The generated simulated slices are synthesized into a whole 3D image. The model of this method is for a 3D model, and the result generated each time is a three-dimensional block, which is often more accurate for the model result. On the other hand, the implementation details are different. In the module of the generation network, when the Softmax value needs to be calculated, the entire image is often calculated. In this model, the original image is divided into several regions, and the Softmax values ​​of each domain are calculated to obtain the average value.

[0063] Then, the data set is inconsistent in terms of points. In traditional GAN ​​networks, paired training is often required, that is, two medical images of the same position are required, and the shooting position and angle should be strictly consistent. This is almost harsh for the data set. In contrast, the model of the present invention only requires two types of medical images, without too many requirements, which is meaningful for clinical practice.

[0064] Finally, the conversion efficiency is different. The traditional GAN ​​network can often convert CT images into MR images. This is a one-way process. That is to say, each time the model is trained, only one-way image conversion can be performed. The model of the present invention is a conversion between two domains, which can convert two images at the same time. In terms of conversion efficiency, the model of the present invention has higher conversion efficiency.

[0065] The medical image processed by the model in the present invention is based on the convolution of the three-dimensional space. Essentially, there is no need to cut and splice the three-dimensional space, which effectively solves the problem of horizontal stripes easily appearing in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a medical image generation method based on deep 3D network and transfer learning of the present invention;

[0067] Figure 2 It is a schematic diagram of the steps of the medical image generation method based on deep 3D network and transfer learning of the present invention;

[0068] Figure 3 This is a schematic diagram of the principle of the model in the medical image generation method based on deep 3D network and transfer learning of the present invention;

[0069] Figure 4A comparison chart of the peak-to-noise ratio of the model in the medical image generation method based on deep 3D network and transfer learning of the present invention and the peak-to-noise ratio of other algorithms;

[0070] Figure 5 This is a comparison chart of the structural similarity of this model in the medical image generation method based on deep 3D network and transfer learning of the present invention and the structural similarity of other algorithms.

[0071] Figure 6 A medical image of a patient's brain generated by the fuzzy clustering algorithm of the present invention;

[0072] Figure 7 A medical image of a patient's brain generated by the CycleGAN algorithm of the present invention;

[0073] Figure 8 A medical image of a patient's brain generated by the WGAN algorithm of the present invention;

[0074] Fig. 9 A medical image of a patient's brain generated by the DualDGAN algorithm of the present invention;

[0075] Fig.10 A medical image of a patient's brain generated by the Dual3D&PatchGAN model of the present invention;

[0076] Fig.11 It is a real image of the patient’s brain;

[0077] Fig.12 It is a schematic diagram of the structure rotated during data enhancement;

[0078] Fig.13 It is a schematic diagram of the flipped structure during data enhancement. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Example: Figure 1 to Figure 5 As shown, the present invention provides the following technical solution, a medical image generation method based on deep 3D network and transfer learning, the generation method comprising the following steps:

[0081] S1. Image preprocessing: determine medical images in different domains as datasets;

[0082] S2, make data set: filter and transform the data set of S1;

[0083] S3, data enhancement: expand the data set produced by S2;

[0084] S4, model training: train the model using the data set enhanced by S3;

[0085] S5. Observation model: observe the training process of the model and determine the range of model parameters;

[0086] S6, grid optimization: determine the range of optimal values ​​of model training parameters;

[0087] S7, retraining: retraining the model using the parameters within the range of the optimal values ​​obtained in S6;

[0088] S8, recheck: observe the model training process again to determine whether the model has achieved the best fit;

[0089] S9. Data testing: Use the test set to test the trained model and check the test results.

[0090] S10, evaluation index: through the test results obtained in S9, multiple evaluation indexes are calculated using data formulas and images;

[0091] S11. Result discussion: Visually check the test results and evaluation index results.

[0092] S1. Image preprocessing: determine medical images in different domains, for example, medical images in the medical CT domain and the medical MR domain, select several images with clear image details and consistent size for training sets, and select several images as test sets. The training set and the test set constitute a data set. In this embodiment, there are data of 9 patients in the data set. Each time, 8 patient data are used as training sets, and 1 patient data are used as test sets. The ratio of training set to test set is 8:1. Each patient is used as both a training set and a data set.

[0093] S2. Prepare a data set: filter the data set of S1, remove blurry and artifact unclear images, place images of different domains in different folders, mark domains of different folders for model training, and convert the original data set images into types, for example, convert the original .png images into .gif sequences, because the model of the present invention is a three-dimensional model, and convolution must be performed on the three-dimensional space;

[0094] S3, data enhancement: expand the data set produced by S2. The expansion adopts the means of rotating and folding the original data set. The purpose is to make the model learn the most and increase the generalization ability of the model.

[0095] S4, model training: the model is trained using the data set enhanced by S3, and images from different domains are put into the model as input of the model image. In this embodiment, 800 rounds of model training are performed;

[0096] S5. Observation model: observe the training process of the model and determine the range of model parameters;

[0097] S6, grid optimization: Since the model may have different effect deviations for a specific data set, grid optimization is required to determine the range of optimal values ​​of the model training parameters. The range of the optimal value has been determined in step S5. The optimal values ​​of the model parameters are gradually determined within this range, which can help determine the optimal parameter values ​​for a specific model and data set, thereby producing the optimal effect.

[0098] S7, retraining: retraining the model using the parameters within the range of the optimal values ​​obtained in S6;

[0099] S8, recheck: observe the model training process again to determine whether the model has reached the best fit. If it has not reached the best fit, readjust the model parameters;

[0100] S9. Data testing: Use the test set to test the trained model and check the test results.

[0101] S10, evaluation index: Based on the test results obtained in S9, use the data formula and the image to calculate the peak noise ratio and structural similarity;

[0102] S11. Result discussion: Visually check the test results and evaluation index results.

[0103] According to the above technical solution, the model consists of two pairs of generation networks and discrimination networks. The task of the generation network is to continuously generate forged data to deceive the discrimination network, and the task of the discrimination network is to identify the forged data of the generation network.

[0104] The model is constrained using a loss function, which is as follows:

[0105] Loss(front)=||bG A (G B (b,x),x1)||;

[0106] Loss(after)=||aG B (G A (a,z),z1)||;

[0107] Among them, Loss(front) represents the first half of the generated network, Loss(back) represents the second half of the generated network, b represents the image input by the first image domain, and image b is transferred through the generator G through transfer learning. B Convert to x, and then pass x through the generator G A On the other hand, the second image domain input image a is transferred through the generator G through transfer learning A Converted to z, and then z passes through the generator G B Convert to z1.

[0108] The first half represents the analytical pathway, and the second half represents the synthetic pathway.

[0109] In step S10, the peak noise ratio includes a mean square error factor. The mean square error factor is given a clean image I and a noise image K of size m*n. The clean image I and the noise image K represent the real image and the image generated by the model respectively. The mean square error factor (MSE) is defined as:

[0110]

[0111] The peak-to-noise ratio is a definition related to the mean square error, with the unit being db. Some parameters of the formula are explained below: is the maximum possible pixel value of the picture, and the peak noise ratio PSNR is defined as follows:

[0112]

[0113] The structural similarity is an important indicator for evaluating images, which is composed of three measurement factors, namely brightness L, contrast C and structure S. The formulas of the three measurement factors are as follows:

[0114]

[0115]

[0116]

[0117] Among them, u x is the mean of x, and similarly, u y is the mean of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y;

[0118] The structural similarity SSIM is achieved by coupling three measurement factors. The definition of structural similarity SSIM is as follows:

[0119] SSIM(x,y)=[L(x,y) α C(x,y) β S(x,y) χ ];

[0120] If α, β, and χ are all set to 1, the original formula can be simplified. After substituting the three measurement factors, the simplified structural similarity SSIM formula can be defined as follows:

[0121]

[0122] When rotating, it rotates, such as Fig.12 As shown, from point A (x0, y0) to point B (x', y'), the length of the chord is R, the angle of rotation is a, and the angle between the chord at point A and the X-axis is b, we can get:

[0123] x0 = R*cosb;

[0124] y0=R*sinb;

[0125] x' = R*cos(ba);

[0126] y' = R*sin(ba);

[0127] Rotate the original image 45 degrees left and right to obtain a data-enhanced image;

[0128] like Fig.13 As shown, folding is to fold the original image A into B by folding up and down. In this embodiment, not only left-right folding but also up-down folding is adopted, that is, the original image A is folded downward once and folded left-right once.

[0129] By rotating and flipping, the content of the data set is expanded, making the training of the model more sufficient and making the images converted by the trained model closer to the real images.

[0130] In this embodiment, the following environment of Table 1 is used for implementation:

[0131] Table 1

[0132]

[0133] Among them, Dual3D&PatchGAN represents the name of this model;

[0134] The Dual3D&PatchGAN model in the present invention is improved based on Dual2DGAN. The Dual3D&PatchGAN model is no longer limited to a 2D model, that is, it is no longer limited to a planar image on an image, but is targeted at small 3D spaces. On the other hand, a PatchGAN module is added. When calculating Softmax, the entire image is no longer calculated, but the image is divided into several small blocks, each small block is calculated separately, and then the blocks are summarized and the average value is obtained to obtain the Softmax result, thereby ensuring the accuracy of the calculation result. In addition, the model also adds a training order disturbance module, a random cropping module, and a dropout part of the neural network module, and at the same time disrupts the training order of the model to ensure the randomness of the training. At the same time, during the initial network model training, a smaller fixed value (2E-4) is selected for the learning rate. However, by viewing the Tendorboard in the Tensorflow framework, it is found that the loss function of the model is difficult to reduce, and it is often oscillating in the final stage, which means that it is difficult to obtain a stable training model. Therefore, a solution is proposed to set the learning rate to 1E-5 and gradually reduce it with the training time.

[0135] It can be seen intuitively from the image that the image generated by DualGAN3D&PatchGAN has better results. Then from the two evaluation indicators of peak noise ratio PSNR and structural similarity SSIM (in order to facilitate the observation of the results, all results are retained to 4 decimal places), as shown in Table 2 and Table 3 below:

[0136] Table 2

[0137]

[0138] Table 3

[0139]

[0140] In summary, compared with traditional fuzzy clustering (FCM), CycleGAN, WGAN, and Dual2DGAN, the peak noise ratio PSNR and structural similarity SSIM of the DualGAN3D&PatchGAN model of the present invention are relatively high, which indirectly reflects that the model is more accurate in realizing the process of medical image generation and has a more perfect processing of details.

[0141] Different algorithms and the DualGAN3D & PatchGAN model of the present invention and the generated patient brain medical images are as follows Figure 6-Figure 10 As shown in Figure 2, the real image of the patient’s brain is as follows: Fig.11 shown.

[0142] In the embodiment, brain CT images and magnetic resonance (MR) images of 9 patients are mainly used as data sets. Before actual training, data cleaning is first required to exclude images with artifacts, blur, incompleteness and other adverse factors, and select several images with better quality. In principle, the number of images should be selected as much as possible. The selected images are placed according to the CT domain and the magnetic resonance domain (MR). Since the patient data in this embodiment is limited, data enhancement is performed on the selected CT domain and magnetic resonance domain (MR) images. Data enhancement is to make limited data produce the value equivalent to more data without substantially increasing the data. In this embodiment, flipping, rotation, cropping, deformation, scaling and other strategies are used to enhance the original data to ensure the learnability of the data as much as possible. After processing the data required for the model, model training can be performed.

[0143] The two data domains of this embodiment refer to the CT domain of 9 patients and the MR domain images of 9 patients, respectively. Since the Dual3D&PatchGAN model is based on the idea of ​​transfer learning, the data set does not need to be used in pairs, so when selecting the data set, only high-quality images in the two domains need to be selected. The model needs to be trained multiple times, and the data details during training can be viewed through Tensorboard and the output console, and the model parameters can be modified in time to improve the generalization ability of the model. After multiple trainings, a stable and high-performance network model is gradually established for subsequent steps. During each training, the images of 8 patients are used as a training set (including CT domain and MR domain), and the images of another patient are selected as a test set (CT domain and MR domain can be selected) for the generation of simulated images. During each test, the test results need to be stored and annotated to check the quality of the image, so as to adjust the network model parameters and gradually find the optimal parameter model. In practical applications, the patient has a medical image (such as an MR image), but the lesion requires one or more other medical images as an auxiliary means. In order to minimize the radiation exposure to the patient within a limited time, a computer can be used for simulation conversion as an auxiliary means of medical treatment.

[0144] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A medical image generation method based on deep 3D network and transfer learning, characterized by: The generation method comprises the following steps: S1. Image preprocessing: determine medical images in different domains, select several images with clear image details and consistent size for training set, and select several images as test set. The training set and test set constitute the data set; S2. Prepare a data set: filter the data set of S1, remove blurry and artifact unclear images, place images of different domains in different folders, mark domains of different folders for model training, and convert the original data set images into different types; the model is a three-dimensional model; S3, data enhancement: expand the data set produced by S2 by rotating and folding the original data set; S4, model training: Use the data set enhanced by S3 to train the model, and put images from different domains into the model as input of the model image; S5. Observation model: observe the training process of the model and determine the range of model parameters; S6, grid optimization: determine the range of optimal values ​​of model training parameters. The range of optimal values ​​has been determined in step S5, and the optimal values ​​of model parameters are gradually determined in this range. S7, retraining: retraining the model using the parameters within the range of the optimal values ​​obtained in S6; S8, recheck: observe the model training process again to determine whether the model has achieved the best fit. If not, readjust the model parameters; S9. Data testing: Use the test set to test the trained model and check the test results. S10, evaluation index: Based on the test results obtained in S9, use the data formula and the image to calculate the peak noise ratio and structural similarity; S11. Result discussion: Visually check the test results and evaluation index results; The model consists of two pairs of generating networks and discriminating networks. The task of the generating network is to continuously generate forged data to deceive the discriminating network, and the task of the discriminating network is to identify the forged data of the generating network. The model is constrained using a loss function, which is as follows: Loss(front) = ||b - G A (G B (b, x), x1)||; Loss(after) = ||a - G B (G A (a, z), z1)||; Among them, Loss(front) represents the first half of the generated network, Loss(back) represents the second half of the generated network, b represents the image input by the first image domain, and image b is transferred through the generator G through transfer learning. B Convert to x, and then pass x through the generator G A Converted to x1, on the other hand, the second image domain input image a is transferred through the generator G through transfer learning A Converted to z, and then z passes through the generator G B Convert to z1; The first half represents the analytical pathway, and the second half represents the synthetic pathway.

2. The medical image generation method based on deep 3D network and transfer learning according to claim 1, characterized in that: In step S10, the peak noise ratio includes a mean square error factor. The mean square error factor is given by a clean image I and a noise image K of size m*n. The clean image I and the noise image K represent the real image and the image generated by the model, respectively. The mean square error factor MSE is defined as: The peak-to-noise ratio is a definition related to the mean square error, with the unit of db. Some parameters of the formula are explained, where MAX I 2 is the maximum possible pixel value of the picture, and the peak noise ratio PSNR is defined as follows:

3. The medical image generation method based on deep 3D network and transfer learning according to claim 1, characterized in that: The structural similarity is an important indicator for evaluating images, which is composed of three measurement factors, namely brightness L, contrast C and structure S. The formulas of the three measurement factors are as follows: Among them, u x is the mean of x, and similarly, u y is the mean of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y; The structural similarity SSIM is achieved by coupling three measurement factors. The definition of structural similarity SSIM is as follows: SSIM(x,y)=[L(x,y) α ·C(x,y) β ·S(x,y) χ ]; If α, β, and χ are all set to 1, the original formula can be simplified. After substituting the three measurement factors, the simplified structural similarity SSIM formula can be defined as follows:

4. The method for generating medical images based on deep 3D network and transfer learning according to claim 3, characterized in that: When rotating from point A (x0, y0) to point B (x', y'), the length of the chord is R, the angle of rotation is a, and the angle between the chord at point A and the X-axis is b, we can get: x0 = R*cosb; y0=R*sinb; x' = R*cos(ba); y' = R*sin(ba); Rotate the original image 45 degrees left and right to obtain a data-enhanced image; The folding is achieved by folding up and down.

Images