Liver us-ct medical image conversion method based on cyclegan network

Abstract

The application discloses a liver US-CT medical image conversion method based on a CycleGAN network. The method comprises the following steps: inputting a liver US image x as a training sample into a generator G; the generator G generates a liver pseudo-CT image y' according to the liver US image x; inputting the liver pseudo-CT image y' as a training sample into a generator F; the generator F generates a loopback liver US image x'' according to the liver pseudo-CT image y'; training the generator G; training a discriminator Dy; inputting a liver CT image y as a training sample into the generator F; the generator G generates a liver pseudo-US image x' according to the liver CT image y; inputting the generated liver pseudo-US image x' as a training sample into the generator G; the generator G generates a loopback liver CT image y'' according to the liver pseudo-US image x'; training the generator F; and the discriminator Dx.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a CycleGAN network-based liver US-CT medical image conversion method. BACKGROUND

[0002] Ultrasound detection (US) and computerized tomography detection (CT) are two commonly used means for checking and observing the condition of internal organs in medicine. There are certain differences in imaging principles between the two. US uses the principle of ultrasonic imaging to display the image of ultrasonic waves in various tissues of the body, find abnormalities, and diagnose diseases. CT, on the other hand, uses X-rays to check for abnormalities in various organs through computerized layer-by-layer scanning and image processing, and to diagnose and differentiate diseases. Both of these methods are also very commonly used in the clinical treatment of the liver.

[0003] From a medical perspective, CT detection has certain advantages in terms of image presentation. The image obtained by ultrasound detection is referred to as a US image. The image obtained by computerized tomography detection is referred to as a CT image. Compared to a US image, a CT image has higher resolution, and unlike the fixed cross-sectional view of a US image, a CT image can view the liver from various angles.

[0004] However, compared to US detection, CT detection also has certain limitations, such as the negative health effects of X-rays on the human body, higher cost, inapplicability to special populations such as pregnant women, and the possibility of allergic reactions to contrast agents required for CT detection in some patients, among others. Overall, the effectiveness of US detection is generally lower than that of CT detection, but there are certain advantages in terms of safety and cost. We hope to achieve the conversion between US images and CT images of a patient's liver, so that the patient can obtain a CT-like image while obtaining a US image of their liver.

[0005] We consider using a CycleGAN (Cycle Generative Adversarial Network: https: / / arxiv.org / pdf / 1703.10593) method to achieve the above purpose. The CycleGAN network is a variant of the GAN (Generative Adversarial Network). The original GAN network uses a generator to generate images and a discriminator to judge whether the images belong to a specific data set, and the two optimize each other to generate fake images. The CycleGAN network adds a cycle structure to the GAN network. Two groups of images with different styles are input from both ends of the CycleGAN network, and under the condition of preserving the characteristics of either group of images, the CycleGAN network is trained and processed to generate fake images similar to the other group of images, thereby achieving the conversion between the two types of images.

[0006] In the prior art CycleGAN network, in order to train the CycleGAN network to realize the conversion from one picture to another, it is necessary to input a pair of two pictures (for example, a picture of a horse in oil painting style and a picture of a horse in watercolor style) into the CycleGAN network at the same time for training. The pair of two pictures, although different in style or belonging to different categories, represent the same object. Thus, the CycleGAN network not only realizes the conversion of the picture category, but also retains the features of the object in the image (for example, both are horses) in the conversion. However, in the medical field, taking the liver US image and CT image as an example, it is difficult to obtain the CT image and the US image of the same liver at the same time. The liver US image / CT image data set owned has no corresponding relationship between the US image and the CT image. In other words, for a certain liver US image in the liver US image data, it is difficult to obtain a liver CT image representing the same liver at the same time; conversely, for a certain liver CT image in the liver CT image data, it is difficult to obtain a liver US image representing the same liver at the same time. Thus, the prior art CycleGAN network cannot be directly used to convert the liver CT image and the liver US image. SUMMARY

[0007] In order to solve the above technical problems, the present application improves the CycleGAN network and its training. By training each part of the CycleGAN network one by one by using the training samples obtained from the liver US image data set and the liver CT image data set, the restriction on the paired relationship of the input data in the training samples of the CycleGAN network is eliminated, and the training process of the CycleGAN network is more suitable for medical images.

[0008] In order to achieve the above object, the present application provides the following technical scheme: a conversion method for liver US images and CT images based on a CycleGAN network, characterized in that it comprises the following steps: S1, liver US image and CT image processing, including: obtaining multiple liver US image data and liver CT image data, and segmenting the region including the liver image itself from the data as the liver US image and the liver CT image; S2, constructing a CycleGAN network, including constructing a generator G, a generator F, a discriminator Dx and a discriminator Dy; wherein the generator G is used to convert the input liver US image into a pseudo liver CT image, and is used to convert the input pseudo liver US image into a cycle liver CT image; the generator F is used to convert the input liver CT image into a pseudo liver US image, and is used to convert the input pseudo liver CT image into a cycle liver US image; the discriminator Dx is used to judge whether the input image is a liver US image; the discriminator Dy is used to judge whether the input image is a liver CT image; S3, inputting the liver US image x obtained in step S1 into the generator G as a training sample; the generator G generates a liver pseudo CT image y' according to the liver US image x; inputting the generated liver pseudo CT image y' into the generator F as a training sample; the generator F generates a cycle liver US image x" according to the liver pseudo CT image y'; S4, calculating a loss Loss1 = Loss1 GAN + LOSS1 cycle , wherein Loss1 GAN is calculated according to the result of the discriminator Dy discriminating the liver pseudo CT image y', and LOSS1 cycle is calculated according to the difference between the liver US image X and the cycle liver US image x"; updating the network parameters of the generator G with the loss Loss1, and keeping the network parameters of the generator F and the discriminator Dy unchanged; here, Loss1 GAN only depends on the discrimination of the discriminator Dy on the pseudo liver CT image y' generated by G, and does not include the discrimination of the discriminator Dy on the real liver CT image y; S5, after training the generator G in step S4, providing the liver pseudo CT image y' obtained in step S3 to the discriminator Dy as a training sample, and providing the liver CT image y obtained in step 1 to the discriminator Dy as a training sample; wherein the liver pseudo CT image y' and the liver CT image Y do not need to represent images of the same liver; calculating a loss Loss2 = Loss2 GAN , wherein Loss2 GAN is calculated according to the result of the discriminator Dy discriminating the liver pseudo CT image y' and the liver CT image y; updating the network parameters of the discriminator Dy with the loss Loss2, and keeping the network parameters of the generator G, the generator F and the discriminator Dx unchanged;

[0009] Step S6, after the generator G is trained in step S4 and the discriminator Dy is trained in step S5, the liver CT image y obtained in step S1 is input to the generator F as a training sample; the generator G generates a liver pseudo-US image x' according to the liver CT image y; the generated liver pseudo-US image x' is input to the generator G as a training sample; the generator G generates a liver CT image y'' according to the liver pseudo-US image x'; here, the liver US image x is not used;

[0010] Step S7, the loss Loss3 = Loss3 GAN + LOSS3 cycle is calculated, where Loss3 GAN is calculated according to the result of the discriminator Dx discriminating the liver pseudo-US image x', and Loss3 cycle is calculated using the difference between the liver CT image y and the liver CT image y''; the network parameters of the generator F are updated using the loss Loss3, and the network parameters of the generator G and the discriminator Dx remain unchanged; here, Loss3 GAN only depends on the discrimination of the pseudo-liver US image x^' generated by F by the discriminator Dx, and does not include the discrimination of the real liver US image by the discriminator Dx; step S8, the liver pseudo-US image x' obtained in step S6 is provided to the discriminator Dx as a training sample, and the liver US image x obtained in step 1 is provided to the discriminator Dx as a training sample; where the liver pseudo-US image x' and the liver US image x do not need to represent images of the same liver; the loss Loss4 = Loss4 GAN is calculated, where Loss4 GAN is calculated according to the result of the discriminator Dx discriminating the liver pseudo-US image x' and the liver US image x; the network parameters of the discriminator Dx are updated using the loss Loss4, and the network parameters of the generator G, the generator F and the discriminator Dy remain unchanged; here, Loss4 GAN includes both the discrimination of the pseudo-liver US image x^' generated by F by the discriminator Dx, and the discrimination of the real liver US image by the discriminator Dx; step S9, the liver US image to be converted is provided to the trained generator G, and the generator G outputs a liver CT image corresponding to the liver US image to be converted; and / or the liver CT image to be converted is provided to the trained generator F, and the generator F outputs a liver US image corresponding to the liver CT image to be converted.

[0011] Compared with the traditional image conversion method, the advantages of the method of the present application are:

[0012] 1) The liver US and CT images of the patient do not need to be matched one by one, which greatly reduces the difficulty of obtaining clinical data samples.

[0013] 2) The method includes verification of the similarity of the original and generated image content, which is suitable for image research with cross-modal problems.

[0014] 3) The conversion between two images is realized, and the conversion from a CT image to a US image is realized at the same time as the conversion from a US image to a CT image. Although the latter is currently considered to have limited significance in the medical field.

[0015] 4) The effect of the conversion between the liver US and CT images shows that the method has the opportunity to be applied to the conversion between US images and CT images of other organs. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only are the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0017] Figure 1A The liver CT image before segmentation (left) and after segmentation (right) is shown.

[0018] Figure 1B The liver US image before segmentation (left) and after segmentation (right) is shown.

[0019] Figure 2 The schematic diagram of the framework of the CycleGAN model is shown.

[0020] Figure 3 The network block diagram of the generator according to the embodiments of the present application is shown.

[0021] Figure 4 The network block diagram of the discriminator according to the embodiments of the present application is shown.

[0022] Figure 5 The schematic diagram of the training process of the CycleGAN model according to the embodiments of the present application is shown. DETAILED DESCRIPTION

[0023] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0024] The embodiments of the present application will be described more clearly below with reference to the drawings.

[0025] According to the embodiments of the present application, medical images are processed to obtain training samples for training a CycleGAN model. For example, liver US image data obtained through liver ultrasound detection and liver CT image data obtained through liver CT detection are processed. The processed liver US image data and the liver CT image data are referred to as liver US images and liver CT images, respectively.

[0026] Step 1: Processing of medical images

[0027] Step 1-1: Obtaining segmented liver US images and CT images

[0028] After obtaining medical clinical liver US image and CT image data, it is necessary to first perform segmentation processing. Referring to Figure 1A and Figure 1B , the medical clinical liver US image data and the liver CT image data include annotation text such as time, parameters, etc., which do not belong to the images of the liver itself, and are removed through image segmentation. Using the currently mature segmentation algorithm for medical images, the date and medical annotation information generated in the original film are ignored, and only the part containing the image content itself is segmented out, which will make the subsequent network training more accurate and efficient, as shown in Figure 1A . For the US image, we also segment out the part containing only the picture content, as shown in Figure 1B .

[0029] Step 1-2: Preprocessing of the segmented images

[0030] After step 1-1, it can be considered that a single-channel grayscale image (0-256 grayscale level color description grayscale is set between all black and all white) dataset is obtained and the image data is preprocessed in the Pytorch deep learning framework.

[0031] The preprocessing method is the same as the usual grayscale image preprocessing method, that is, first, the US image set and the CT image set are cropped to the same size through the built-in Transforms method of Pytorch, which is convenient for subsequent network generation and comparison, and then the grayscale pixels are normalized according to the mean and variance of the image grayscale data.

[0032] It should be noted that, unlike the [0-1] normalization in the usual preprocessing, the data preprocessing for the CycleGAN model and its related network according to the present application needs to be normalized to a normal distribution between [-1, 1]. This normalization method is more suitable for calculation and propagation in the CycleGAN network according to experience.

[0033] Step 2: Construction of CycleGAN network

[0034] The main purpose of the CycleGAN network is to achieve Domain Adaptation. In this application, taking the US images and CT images of the liver as an example, the set storing the liver US images is referred to as dataset X, and the dataset storing the liver CT images is referred to as dataset Y. Referring to Figure 2 , the CycleGAN network includes generators F and G, and discriminators Dx and Dy.

[0035] The goal is to train a generator G that inputs a liver US image and outputs a liver CT image, denoted as G(x) = y', x e X; at the same time, another generator F is also expected to be trained, which inputs a liver CT image and outputs a liver US image, denoted as F(y) = x', y e Y. The output y' of the generator G for the input liver US image x is referred to as a pseudo liver CT image. The output x' of the generator F for the input liver CT image y is referred to as a pseudo liver US image. The pseudo liver CT image y' is input to the generator F, and the output of the generator F is x", denoted as F(y') = F(G(x)) = x", which is referred to as a loopback liver US image. The pseudo liver US image x' is input to the generator G, and the output of the generator G is y", denoted as G(x') = G(F(y)) = y", which is referred to as a loopback liver CT image.

[0036] In order to achieve this purpose, two discriminators Dx and Dy also need to be trained to judge the quality of the images generated by the two generators: if the image y' generated by the generator G does not look like the image y in the dataset Y, then the discriminator Dy should give the image y' a low score (for example, the lowest score is 0), otherwise if the image y' looks like the image y in the dataset Y, then the discriminator Dy should give the image y' a high score (for example, the highest score is 1). In addition, the discriminator Dy should always give the image y in the dataset Y a high score.

[0037] The same is true for the discriminator Dx. If the image x' generated by the generator F does not look like the image x in the dataset X, then the discriminator Dx should give the image x' a low score (for example, the lowest score is 0), otherwise if the image x' looks like the image x in the dataset X, then the discriminator Dx should give the image x' a high score (for example, the highest score is 1). In addition, the discriminator Dx should always give the image x in the dataset X a high score.

[0038] In the training process of CycleGAN network, the discriminators (Dx and Dy) and the generators (G and F) are trained separately. For example, when the parameters of the generator are fixed and the discriminator is trained, the discriminator can learn better discrimination skills; when the parameters of the discriminator are fixed and the generator is trained, the generator is forced to generate better quality images in order to deceive the now more powerful discriminator. Both gradually evolve in this iterative learning process and eventually reach a dynamic balance.

[0039] Step 2-1: Constructing the generator

[0040] The purpose of constructing the generator is to extract the features of the input image and generate new images through mapping. Therefore, the generator (G and F) can be divided into three parts for understanding. The first part can be understood as feature extraction (encoding), which extracts the abstract features of the original image, the second part is conversion, which converts the features from image domain A to image domain B, and then becomes a picture in domain B through restoration (decoding). Here A and B can represent the liver US image dataset and the CT image dataset respectively, and vice versa. Taking the conversion of US image to CT image as an example (not the corresponding conversion in the example, only for example), the process can be represented as Figure 3 .

[0041] Referring to Figure 3 , as an example, the Conv convolutional layer used in the encoding part of the generator G, the residual module (Resnet Block) used in the feature conversion part, and the DeConv deconvolutional layer used in the decoding part can be directly defined and combined through the Pytorch framework. Figure 3 The pooling layer, activation layer and other parts are omitted. Generator F has an optional same structure as generator.

[0042] Step 2-2: Constructing the discriminator

[0043] After one side in the CycleGAN model cycle generates a fake image through the generator (for example, generating a liver CT image from a liver US image in the example of step 2-1), it completes the G(x) = y' part described in step 2. Then the generated image y' needs to be discriminated by the discriminator Dy. Continuing from step 2-1, taking the conversion from US image to CT image as an example, the discrimination process of the discriminator Dy can be realized by classifying the image into two categories, referring to Figure 4 . The Conv convolutional layer used in the discriminator can be directly defined and combined through the Pytorch framework. Figure 4 The pooling layer, activation layer and other parts are omitted. Discriminator Dx has an optional same structure as discriminator Dy.

[0044] Step 3: Defining the network loss function

[0045] After the construction of the CycleGAN neural network structure is completed, the network-related loss function needs to be defined for subsequent network training and optimization. The loss of the CylceGAN network is mainly composed of two parts: the GAN loss Loss GAN and the cycle loss Loss cycle .

[0046] Step 3-1: Define the GAN loss Loss GAN

[0047] There is the same GAN loss part in the CycleGAN network as in the traditional GAN network. Since it exists by X to Y and by Y to X two parts of the transformation, its GAN loss is also divided into two parts. Its expression form is:

[0048]

[0049] Loss GAN To ensure the evolution of the generator and the discriminator, and thus ensure that the generator can generate more realistic images. The above formula is the first half of the part For example. In this paper This part is used to express the GAN loss of the conversion process of the US image to the CT image. The first term represents the judgment of the discriminator Dy on the CT image in the data set Y. When Dy can accurately judge that the real CT image y in the data set Y is true, D Y (y) tends to 1, then log D Y (y) tends to 0, at which time it can be considered that the loss is very small. The second term [log(1-D Y (G(x))] represents the judgment of the discriminator Dy on the CT image y' generated by the generator G through the US image x. If the CT image y' generated by the generator G is enough to reach the step of deceiving, then D Y (G(x)) tends to 1, that is, 1-D Y (G(x)) tends to 0, and the second term [log(1-D Y (G(x))] tends to infinitesimal. This embodies the meaning of the first confrontation in the generative adversarial network.

[0050] The above is the overall introduction of the GAN loss. In the training process of the CycleGAN network according to the present application, the specific GAN loss used is different based on the object being trained (generator F / G, discriminator Dx / Dy), which will be introduced in detail later.

[0051] Specifically, for example, in the process of updating the discriminator Dx (Dx, G, F parameters are fixed), its Loss GANThe expression form is:

[0052]

[0053] That is, for the discriminator Dx, the optimization direction is to maximize the log D X (x) (that is, expect D X (x) to be 1) so that the loss is minimized. X (x) approaches 1, so that log D X (x) approaches 0, at which time the loss is maximized). And log(1-D X (F(y)) (that is, expect D X (F(y)) to be 0) so that the loss is minimized. X (F(y)) approaches 0, 1-D X (F(y)) approaches 1, [log(1-D X (F(y))] approaches 0, at which time the loss is maximized). By maximizing this loss, the discriminator Dx has the ability to better judge that the image x in the X set is true, and the pseudo image x' generated by the image y in the Y set through the generator F is false. The same is true for Dx.

[0054] For the generator, taking the generator G as an example, we want the generated image y' = G(x) to be judged as true by the discriminator Dy, and vice versa, we also want the generated image x' = F(y) to be judged as true by the discriminator Dx.

[0055] In the process of updating the generator G (the parameters of Dx and Dy are fixed), the expression form of its loss is:

[0056]

[0057]

[0058] We want to minimize the loss in the direction, that is, D Y (G(x)) and D X (F(y)) both approach 1, that is, the image generated by the generator obtains a very high confidence when judged by the discriminator of the other party, so that 1-D Y (G(x)) and 1-D X (F(y)) both approach 0, [log(1-D Y (G(x))] and [log(1-D X(F(y))] both tend to negative infinity, so as to minimize the loss in the formula. As can be seen, the generator and the discriminator have certain paradox and confrontation for the optimization direction of the network. In the process of constantly fixing one side to optimize the other side, the two sides constantly compete and confront each other, so as to achieve the purpose of common progress and optimization. This reflects the essence of the generative adversarial network.

[0059] Step 3-2: Defining the cycle loss Loss cycle

[0060] Through step 3-1, the loss of CycleGAN propagation in two directions is defined. However, only by doing so cannot guarantee to achieve our expected effect. In CycleGAN, not only do we want the picture generated by the generator to be the same as the picture style in the dataset, but also we want the picture generated by the generator to be the same as the content of his input picture (for example, inputting a liver section US image, we hope to generate a CT image corresponding to this liver section, rather than a random CT image, such as a CT image similar to other diseases or other sections, etc.), otherwise the meaning is not great. Therefore, while converting the image mode, the content of the image itself should also be limited to ensure that the input US image can be converted into a CT image expressing the same target object (for example, the same liver) content. Thus, the cycle loss Loss of the CycleGAN network is defined cycle , which is expressed as:

[0061]

[0062] Where F(G(x)) represents that the CT image y' generated from the US image x (referred to as a pseudo-liver CT image) is converted back through the F generator, and becomes a US image x" (referred to as a loop-back liver US image) again. Then the first norm of the difference value of the corresponding pixels of the two images x and x" is calculated, which is taken as the cycle loss of the network. When the network is optimized in the direction of reducing the loss, the similarity between the original image and the image mapped back by the generator of the other party is improved, thereby ensuring that the content contained after the image conversion does not change.

[0063] The final loss of the network is defined as the sum of the two losses:

[0064] Step 4: Training the CylceGAN network

[0065] After the above steps, after preparing the dataset, building the network structure and defining the loss function, the CycleGAN network built is trained next. According to the CylceGAN network structure built in step 2 and the segmented training mode (i.e. fixing the discriminator parameters to train the generator, and fixing the generator parameters to train the discriminator) proposed in step 3, the following network training process is proposed according to the embodiments of the present application, as shown inFigure 5 as shown.

[0066] In the whole CycleGAN network training process, the generator G, the generator F, the discriminator Dx, and the discriminator Dy are trained respectively. The order in the whole training depends on the category of the picture input at present. For example, if the training sample input at first is the liver US image dataset X, the training relationship order is to train the generator G first, and then train the discriminator Dy. Conversely, if the training sample input at first is the liver CT image dataset Y, the training relationship order is to train the generator F first, and then train the discriminator Dx. Moreover, the images of the liver US image dataset X and the liver CT image dataset Y used in the training do not need to have a corresponding relationship, in other words, when using the liver US image x for training, there is no need to obtain the liver CT image y representing the same liver at the same time.

[0067] Taking the training of the generator G as an example, in the process of training and updating the generator G to make it advance in the direction of generating better pictures, the liver US image x is provided to the generator G, and the generator G outputs the pseudo liver CT image y' in response, and the discriminator Dx is used to distinguish the pseudo liver CT image y' generated by the generator G as true or false. Next, the pseudo liver CT image y' is reflected through the generator F again to obtain the loopback liver US data x". Therefore, in the process of training the generator G, the discriminator Dx and the generator F are used. However, the parameters of the discriminator Dx are fixed at this time, and only the Dx needs to make a judgment, only the parameters of the generator G are updated, and the parameters of the Dx are unchanged in this process. Conversely, the same is true for the training of F.

[0068] In the training of the generator F / G, the loss used is Loss = Loss GAN + Loss cycle .

[0069] After one stage of training of the generator G, the discriminator Dy is trained next. In the process of training the Dy, only the pseudo liver CT image generated by the generator G and the image of the liver CT image dataset Y are used. Therefore, from the structure, the gradient propagation is only related to G, and F and Dx do not need to be used. Conversely, the same is true for the Dx process.

[0070] In the training of the Dx, the loss used is only the discrimination loss LOSSgan, and the cycle loss Loss cycle is not calculated.

[0071] In the process of calculating the discriminator loss Loss GAN , since it is a simple binary classification probability discrimination, the traditional cross-entropy loss function BCE Loss (Binary Cross Entropy Loss) can be used.

[0072] Further, for the problems in the field of medical images, more complex and suitable loss functions such as Dice Loss and Jaccard Loss are used to better evaluate the image conversion effect based on the previous experience in the industry.

[0073] Dice Loss is an index for measuring similarity, and its calculation formula is as follows:

[0074]

[0075] wherein X and Y represent the binary segmentation map predicted by the model and the real binary segmentation map respectively. |X| and |Y| represent the number of all pixels in the two binary segmentation maps. |X∩Y| represents the number of intersection pixels of the two binary segmentation maps.

[0076] Jaccard Loss is another commonly used loss function for evaluating segmentation tasks, and its calculation formula is as follows:

[0077]

[0078] wherein X and Y represent the binary segmentation map predicted by the model and the real binary segmentation map respectively. |X| and |Y| represent the number of all pixels in the two binary segmentation maps. |X∩Y| represents the number of intersection pixels of the two binary segmentation maps, and |X∪Y| represents the number of union pixels of the two binary segmentation maps.

[0079] In the present application, as an example, the pseudo liver US image x' output by the generator F is further mapped by the generator G to obtain the cycle liver CT image y". The real liver CT image y and the cycle liver CT image y" are compared (also see Figure 5 LOSScycle in the connecting line between the real y and the reflection y" in the figure). In the above comparison, the two images participating in the comparison (for example, the real liver CT image y and the cycle liver CT image y") are the two binary segmentation maps used in the calculation of Dice Loss and Jaccard Loss.

[0080] In the CycleGAN network, adding Dice Loss and Jaccard Loss as loss functions in the original BCELoss for training can improve the evaluation effect of the network, mainly in the following aspects:

[0081] ​Improve the accuracy of segmented pixel classification: The training objective of CycleGAN network is to convert input images into output images, while Dice Loss and Jaccard Loss measure the accuracy of binary segmented pixel classification, so adding these two loss functions to CycleGAN network can effectively improve the accuracy of generated segmentation maps.

[0082] Improve image quality: Dice Loss and Jaccard Loss consider both the classification accuracy of segmented pixels and the spatial shape information of images, so they can reduce noise and artifacts in generated segmentation images and improve image quality.

[0083] Reduce the risk of overfitting: The objective of CycleGAN network is to convert input images into output images, while Dice Loss and Jaccard Loss can effectively avoid the risk of overfitting, so adding these two loss functions to CycleGAN network can reduce the risk of overfitting.

[0084] For the loss of the part cycle The loss of the part is calculated by the first-order norm introduced above. The optimizer is usually selected as Adam optimization algorithm based on the conventional GAN-related network training experience.

[0085] Innovation and significance:

[0086] 1. The conversion of US images to CT images is realized for liver parts, which reduces the cost and improves the safety

[0087] 2. The non-matching training mode conforming to the actual situation of medicine is realized by CycleGAN network, which reduces the difficulty of data collection

[0088] 3. The loss function adapted to the field of medical images is adopted

[0089] 4. The method may have certain universality for other possible modal conversion

[0090] The above only describes the preferred examples of the present application and does not limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made to the present application shall be included in the protection scope of the present application.

Claims

1. A CycleGAN network-based conversion method for liver ultrasound detection (US) images and CT images, characterized in that, Comprising: Step S1, processing liver US images and CT images, comprising: obtaining multiple liver US image data and liver CT image data, and segmenting the region including the liver image itself from them as liver US images and liver CT images; Step S2, constructing a CycleGAN network, comprising constructing a generator G, a generator F, a discriminator Dx and a discriminator Dy; wherein the generator G is used to convert the input liver US image into a liver pseudo-CT image, and is used to convert the input liver pseudo-US image into a liver CT image; the generator F is used to convert the input liver CT image into a liver pseudo-US image, and is used to convert the input liver pseudo-CT image into a liver US image; the discriminator Dx is used to judge whether the image input thereto is a liver US image; the discriminator Dy is used to judge whether the image input thereto is a liver CT image; Step S3, inputting the liver US image x obtained in step S1 to the generator G as a training sample; the generator G generates a liver pseudo-CT image y' according to the liver US image x; inputting the generated liver pseudo-CT image y' to the generator F as a training sample; the generator F generates a liver US image x'' according to the liver pseudo-CT image y'; Step S4, calculating loss wherein Loss1 is calculated according to the result of discriminating the liver pseudo-CT image y' by the discriminator Dy GAN and LOSS1 is calculated using the difference between the liver US image X and the looped liver US image x" cycle ; updating the network parameters of the generator G with the loss Loss1, and keeping the network parameters of the generator F and the discriminator Dy unchanged; Step S5, providing the liver pseudo-CT image y' obtained in step S3 to the discriminator Dy as a training sample, and providing the liver CT image y obtained in step 1 to the discriminator Dy as a training sample; wherein the liver pseudo-CT image y' and the liver CT image y do not necessarily represent images of the same liver; calculating a loss Loss2 = Loss2 GAN , wherein Loss2 is calculated according to the result of the discriminator Dy discriminating the liver pseudo-CT image y' and the liver CT image y GAN ; updating the network parameters of the discriminator Dy with the loss Loss2, and keeping the network parameters of the generator G, the generator F and the discriminator Dx unchanged; Step S6, inputting the liver CT image y obtained in step S1 to the generator F as a training sample; the generator F generates a liver pseudo-US image x' according to the liver CT image y; inputting the generated liver pseudo-US image x' to the generator G as a training sample; the generator G generates a liver CT image y'' according to the liver pseudo-US image x'; Step S7, calculate loss Loss3 = Loss3 GAN + LOSS3 cycle , wherein Loss3 is calculated according to the result of discriminating the liver pseudo-US image x' by the discriminator Dx GAN , and LOSS3 is calculated by the difference between the liver CT image y and the recirculation liver CT image y" cycle ; update the network parameters of the generator F with the loss Loss3, and keep the network parameters of the generator G and the discriminator Dx unchanged; Step S8, providing the liver pseudo-US image x' obtained in step S6 to the discriminator Dx as a training sample, and providing the liver US image x obtained in step 1 to the discriminator Dx as a training sample; wherein the liver pseudo-US image x' and the liver US image x do not need to represent images of the same liver; calculating a loss Loss4=Loss4 GAN , wherein Loss4 is calculated according to the result of the discriminator Dx discriminating the liver pseudo-US image x' and the liver US image x GAN ; updating the network parameters of the discriminator Dx with the loss Loss4, and keeping the network parameters of the generator G, the generator F and the discriminator Dy unchanged; Step S9, providing a liver US image to be converted to the trained generator G, and the generator G outputs a liver CT image corresponding to the liver US image to be converted; and / or providing a liver CT image to be converted to the trained generator F, and the generator F outputs a liver US image corresponding to the liver CT image to be converted.

2. The method of claim 1, wherein The step 1 further comprises respectively cropping the liver US images and the liver CT images to a uniform size, and normalizing the gray scale pixels according to the image gray scale data, so that the gray scale pixels of the liver US images and the liver CT images are normalized to satisfy a normal distribution between [-1, 1]; wherein for each image of the liver US images, there is no need to exist a corresponding image representing the same liver in the liver CT images, and for each image of the liver CT images, there is no need to exist a corresponding image representing the same liver in the liver US images.

3. The method of claim 1 or 2, wherein The steps S2 to S8 are repeatedly executed to perform multiple rounds of training on the CycleGAN network.

4. The method of any one of claims 1-3, wherein In the step S4, wherein x represents an input liver US image x, G(x) represents a liver pseudo-CT image y' generated by the generator G according to the input liver US image x, D Y () represents the result of the discriminator D judging the liver pseudo-CT image, x ~ p data (x) represents that the liver US image x belongs to the training sample of the liver US image, represents the mathematical expectation; and wherein F(G(x)) represents that the generator F converts the input liver pseudo-CT image y' into a loopback liver US image x", and || ||1 represents a first order norm.

5. The method of claim 4, wherein or wherein |A| denotes the number of pixels of the binarized segmentation map A, |A∩B| denotes the number of pixels of the intersection of the two binarized segmentation maps A and B, and |A∪B| denotes the number of pixels of the union of the two binarized segmentation maps A and B.

6. The method of claim 5, wherein In the step S5, wherein x represents an input liver US image x, G(x) represents a liver pseudo-CT image y' generated by the generator G according to the input liver US image x, D Y (G(x)) represents a result of the discriminator Dydiscriminating the liver pseudo-CT image, D Y (y) represents a result of the discriminator Dydiscriminating the liver CT image y, x ~ p data (x) represents that the liver US image x belongs to the training sample of the liver US image, y ~ p data (y) represents that the liver CT image y belongs to the training sample of the liver CT image, represents a mathematical expectation.

7. The method of claim 6, wherein The step S7, wherein y represents the input liver CT image y, F(y) represents the liver pseudo US image x' generated by the generator F according to the input liver CT image y, D x () represents the result of the discriminator Dx discriminating the liver pseudo US image, y ~ p data (y) indicates that the liver US image y belongs to the training sample of the liver CT image, represents the mathematical expectation; and wherein G(F(y)) represents the input liver pseudo US image x' converted into the loopback liver CT image y" by the generator G, and || ||1 represents the first order norm.

8. The method of claim 7, wherein or wherein |A| denotes the number of pixels of the binarized segmentation map A, |A∩B| denotes the number of pixels of the intersection of the two binarized segmentation maps A and B, and |A∪B| denotes the number of pixels of the union of the two binarized segmentation maps A and B.

9. The method of claim 8, wherein The step S8, wherein x represents an input liver US image x, y represents an input liver CT image y, F(y) represents a liver pseudo US image x' generated by the generator Y according to the input liver CT image y, D x (F(y)) represents the result of the discriminator Dx discriminating the liver pseudo US image, D x (x) represents the result of the discriminator Dx discriminating the liver US image x, x ~ p data (x) indicates that the liver US image x belongs to the training sample of the liver US image, y ~ p data (y) indicates that the liver CT image y belongs to the training sample of the liver CT image, represents the mathematical expectation.

10. An information processing apparatus comprising a storage, a processor, and a program stored on the storage and capable of running on the processor, characterized by the processor implements the program when running to realize the method according to any one of claims 1-9.

Images