Method and System for Evaluating Post-Interventional Response of Liver Tumors Based on Adversarial Generative Network

Abstract

The present invention provides a method and system for evaluating the response after liver tumor intervention based on a generative adversarial network. A method for evaluating the response after liver tumor intervention based on a generative adversarial network includes the following steps: S1. Training a deep learning model to obtain an evaluation model capable of giving a prognosis evaluation result; S2. The evaluation model takes the plain CT scan after liver surgery and the postoperative serological test information as inputs and outputs a prognosis evaluation result. The present invention does not require liver puncture, which can reduce the trauma risk and medical resources for patient prognosis estimation; after the operation, the patient of the present invention does not need to take enhanced CT, and there will be no radiation risk and medical resource consumption caused by taking enhanced CT. At the same time, it can give a prognosis evaluation result based on enhanced CT, ensuring that the CT image used as the evaluation basis can clearly show the lesion, having the effect of enhanced CT, and being able to give a more accurate evaluation result.

Description

Technical Field

[0001] The present invention belongs to the technical field of prognosis assessment, and particularly relates to a method and system for evaluating the response after liver tumor intervention based on a generative adversarial network. Background Art

[0002] After liver tumor patients undergo interventional treatment, they are prone to multiple or even diffuse recurrence or metastasis, which seriously endangers the health of patients and affects the clinical diagnosis and treatment effect. Evaluating the response after liver tumor intervention can provide a reference for post-response treatment, help patients prevent diffuse recurrence or metastasis, and thus achieve better diagnosis and treatment effects. Currently, there is no method for visually discriminating and predicting the prognosis of recurrence, metastasis, and invasion in liver tumor patients who have received this type of treatment. Currently, most evaluations of the response after liver tumor intervention mainly have three methods:

[0003] 1. One is to perform a liver puncture, take pathological tissues for pathological analysis of pathological sections, and then obtain the prognosis assessment result;

[0004] 2. Another is to take a plain CT scan and perform a prognosis assessment manually based on the plain CT;

[0005] 3. There is also another method, which is to take a contrast-enhanced CT scan and perform a prognosis assessment manually based on the contrast-enhanced CT.

[0006] However, each of the above three methods has its own defects. For example, the first method requires a liver puncture, which has a certain risk of trauma, consumes a large amount of medical resources, and the accuracy of the prognosis assessment result is not ideal; the second method is to perform a prognosis assessment manually first, which is heavily affected by the doctor's experience and subjective cognition, and the reliability of the prognosis assessment result is not high. Moreover, the plain CT cannot clearly show the lesion. Most of the time, it only shows a low-density shadow in the liver, and it is impossible to determine what kind of lesion this low-density shadow is, so it will affect the doctor's judgment and lead to inaccurate assessment; the third method, the contrast-enhanced CT can show the lesion more clearly, and clarify the tendency of the lesion through its enhancement method. By the contrast-enhanced CT, the enhancement degree and enhancement method of this lesion in the arterial phase, venous phase, and delayed phase can be understood, so as to help the doctor make a more accurate prognosis assessment. However, the contrast-enhanced CT consumes more medical resources than the plain CT, and because the contrast-enhanced CT requires the injection of a contrast agent, there is a greater radiation risk. Summary of the Invention

[0007] The purpose of the present invention is to solve the above problems and provide a method and system for evaluating the response after liver tumor intervention based on a generative adversarial network.

[0008] To achieve the above purpose, the present invention adopts the following technical solutions:

[0009] A method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, comprising the following steps:

[0010] S1. Train a deep learning model to obtain an evaluation model capable of giving a prognosis evaluation result;

[0011] S2. The evaluation model takes the plain CT after liver surgery and the postoperative serological test information as inputs and outputs the prognosis evaluation result.

[0012] In the above method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, the deep learning model includes a first model and a second model. The first model takes the plain CT after liver surgery and the postoperative serological test information as inputs and outputs the enhanced CT after simulated liver surgery;

[0013] The second model takes the enhanced CT after simulated liver surgery as an input and outputs the prognosis evaluation result.

[0014] In the above method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, in step S1, different sample data are used to train the first model and the second model respectively, and then the trained first model and the second model are cascaded to obtain an evaluation model for giving a prognosis evaluation result.

[0015] In the above method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, in step S1, the first model is a generative adversarial network. The trained generative adversarial network is used to combine serological test information to generate enhanced CT from plain CT.

[0016] In the above method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, the sample data for training the first model includes the plain CT before liver surgery, the enhanced CT before liver surgery, and the preoperative serological test information.

[0017] In the above method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, the sample data for training the first model includes the plain CT after liver surgery, the enhanced CT after liver surgery, and the postoperative serological test information.

[0018] In the above method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, the generative adversarial network includes a generator, a discriminator, and a classifier. The generator is used to generate enhanced CT from plain CT according to serological test information. The discriminator is used to perform adversarial training with the generator to ensure the image generation ability of the generator. The classifier is used to ensure that the images generated by the generator conform to the serological test information to integrate the serological test information into the generator.

[0019] In the above method for evaluating the response after liver tumor intervention based on the adversarial generative network, the method for the adversarial generative network to generate enhanced CT from plain CT by combining serological test information includes:

[0020] S11. Convert the serological test information into a one-hot vector c;

[0021] S12. Input the vector c, the vector z used to determine the image style, and the plain CT after liver surgery into the generator;

[0022] S13. The generator generates a simulated enhanced CT after liver surgery based on the vector c, the vector z, and the plain CT after liver surgery, and outputs the simulated enhanced CT after liver surgery to the discriminator and the classifier;

[0023] S14. The discriminator judges the fidelity of the simulated enhanced CT after liver surgery, and the classifier extracts a new vector from the simulated enhanced CT after liver surgery and judges whether the vector is consistent with the vector c. When the extracted vector is consistent with the vector c and the fidelity of the simulated enhanced CT after liver surgery is higher than the fidelity threshold, it is determined that the simulated enhanced CT after liver surgery has been obtained.

[0024] In the above method for evaluating the response after liver tumor intervention based on the adversarial generative network, in step S1, the second model is a convolutional network, and the trained convolutional network is used to output a prognosis evaluation result according to the enhanced CT.

[0025] In the above method for evaluating the response after liver tumor intervention based on the adversarial generative network, the sample data of the second model includes enhanced CT after liver surgery and postoperative prognosis data.

[0026] A system for evaluating the response after liver tumor intervention based on the adversarial generative network is used to evaluate the response after liver tumor intervention using the above evaluation method.

[0027] The advantages of the present invention are as follows:

[0028] 1. There is no need to perform liver puncture, reducing the trauma risk and medical resources for patient prognosis estimation;

[0029] 2. After the operation, the patient does not need to take enhanced CT, avoiding the radiation risk and medical resource consumption caused by taking enhanced CT. At the same time, a prognosis evaluation result can be given based on the enhanced CT, ensuring that the CT image used as the evaluation basis can clearly show the lesion, having the effect of taking enhanced CT, and being able to give a more accurate evaluation result. Description of the Drawings

[0030] Figure 1 It is a flowchart of the training method of the deep learning model used in the method for evaluating the response after liver tumor intervention based on the adversarial generative network of the present invention;

[0031] Figure 2 This is the evaluation process diagram of the evaluation model in the liver tumor interventional response evaluation method based on the adversarial generative network of the present invention;

[0032] Figure 3 This is the network structure diagram of the adversarial generative network of the present invention;

[0033] Figure 4 This is the ROC curve diagram of the evaluation model for predicting the 5-year survival of patients of the present invention;

[0034] Figure 5a This is the tumor identification area of patient A in the plain scan period before TACE treatment of the present invention;

[0035] Figure 5b This is the tumor identification area of patient A in the arterial phase before TACE treatment of the present invention;

[0036] Figure 5c This is the tumor identification area of patient A in the venous phase before TACE treatment of the present invention;

[0037] Figure 6 This is the real image of the postoperative plain scan CT of patient A's liver after TACE treatment;

[0038] Figure 7a Based on the present invention Figure 6 The arterial phase image of the enhanced CT of the simulated postoperative liver generated from the postoperative plain scan CT of the liver and the postoperative serological test information of the present invention;

[0039] Figure 7b Corresponding to Figure 7a The real CT arterial phase image;

[0040] Figure 8a Based on the present invention Figure 6 The venous phase image of the enhanced CT of the simulated postoperative liver generated from the postoperative plain scan CT of the liver and the postoperative serological test information of the present invention;

[0041] Figure 8b Corresponding to Figure 8a The real CT venous phase image. Detailed implementation manner

[0042] The present invention will be further described in detail below with reference to the drawings and specific implementation manners.

[0043] This solution proposes an evaluation method for the response of liver tumors after interventional treatment based on a generative adversarial network and a system for implementing this method to address the problems existing in current various prognostic evaluation methods. It aims to visually evaluate the situation of liver tumors after the interventional response of liver tumors with the support of postoperative serological test information and plain abdominal CT data of the liver after surgery, and applies deep learning technology. The deep learning models used mainly include a first model and a second model. The first model is a generative adversarial network for generating enhanced CT from plain abdominal CT by combining serological test information, and the second model is a convolutional network for outputting a prognostic evaluation result based on the enhanced CT. As Figure 1 and Figure 2 shown, the specific implementation method includes the following steps:

[0044] S1. First, use the preoperative plain abdominal CT of the patient's liver, the preoperative enhanced CT of the liver, and the preoperative serological test information to train a generative adversarial network that can generate enhanced CT images from plain abdominal CT images by style transfer, combining serological test information;

[0045] S2. Then, use the postoperative enhanced CT of the liver and the postoperative prognostic data obtained from follow-up observations to train a convolutional network that can output a prognostic evaluation result based on the postoperative enhanced CT of the liver by labeling the corresponding enhanced CT based on the postoperative prognostic data, and perform a regression prediction on the patient's prognostic survival duration (5-year survival rate).

[0046] S3. Finally, cascade the trained generative adversarial network and the convolutional network model to obtain an evaluation model that can give a prognostic evaluation result;

[0047] S4. Then, the evaluation model can be put into use. When in use, only the postoperative plain abdominal CT of the liver and the postoperative serological test information need to be input. The evaluation model obtains the simulated postoperative enhanced CT of the liver based on the postoperative plain abdominal CT of the liver, and then outputs a prognostic evaluation result based on the simulated postoperative enhanced CT of the liver.

[0048] In step S1, the sample data of the generative adversarial network, "the preoperative plain abdominal CT of the liver, the preoperative enhanced CT of the liver, and the preoperative serological test information", can be changed to "the postoperative plain abdominal CT of the liver, the postoperative enhanced CT of the liver, and the postoperative serological test information", or both can be included.

[0049] The postoperative enhanced CT of the liver in step S2 can be the real postoperative enhanced CT of the liver, or the simulated postoperative enhanced CT of the liver generated by the trained generative adversarial network, or both can be included.

[0050] The prognostic data in step S2 includes the patient's prognostic survival results, such as the 5-year survival situation, the recurrence situation at three months, etc. The prognostic evaluation result in step S3 includes the prediction of the patient's future survival situation.

[0051] Specifically, the adversarial generative network introduces the Info vector of InfoGAN into the generator of StyleGAN to construct a new InfoStyleGAN with semantic constraints, so as to integrate serological test information into the generator. As Figure 3 shown, it specifically includes a generator Generator, a discriminator Discriminator, and a classifier Classifier. During training, first, the serological test information is converted into a one-hot vector, that is, the info vector c in the generator. The z vector input into the generator is obtained in advance by a z-code generator, which determines the style of the image. The generator generates a picture-enhanced CT based on the vector c, the vector z, and the plain CT. The generated enhanced CT is simultaneously transmitted to the discriminator and the classifier. The discriminator is used to perform adversarial training with the generator to ensure the picture generation ability of the generator, that is, to ensure the clarity and realism of the generated pictures. The method of adversarial training is the same as that of the prior art and will not be elaborated here specifically. The classifier is used to ensure that the parameters of the generated picture (the enhanced CT in this embodiment) conform to the control of the vector c. It extracts a new vector from the generated picture, and this vector should be as consistent as possible with c. Since the information contained in c is only serological test information, only when the generated picture also conforms to the serological test information, the extracted vector can be consistent with c. By ensuring that the extracted vector is consistent with c, it is ensured that the parameters of the generated picture conform to the control of the vector c, that is, the serological test information is integrated into the generator. Through the above steps, the generator, discriminator, and classifier are continuously trained, and the parameters of the generator, discriminator, and classifier are adjusted. Finally, a trained adversarial generative network is obtained according to the needs such as the training degree and training effect.

[0052] When this evaluation model is put into use, the method for generating enhanced CT from plain CT by the trained adversarial generative network in combination with serological test information includes:

[0053] Converting the serological test information into a one-hot vector c;

[0054] Inputting the vector c, the vector z for determining the image style, and the plain CT after liver surgery into the generator;

[0055] The generator generates an enhanced CT after simulated liver surgery based on the vector c, the vector z, and the plain CT after liver surgery, and outputs the enhanced CT after simulated liver surgery to the discriminator and the classifier;

[0056] The discriminator judges the fidelity of the enhanced CT after liver simulation surgery. The classifier extracts a new vector from the enhanced CT after liver simulation surgery and determines whether this vector can be consistent with vector c. When the extracted vector is consistent with vector c and the fidelity of the enhanced CT after liver simulation surgery is higher than the fidelity threshold, it is determined that the enhanced CT after liver simulation surgery has been obtained. This enhanced CT after liver simulation surgery can be continuously input into the convolutional network, and the convolutional network outputs the prognosis evaluation result according to the enhanced CT after liver simulation surgery, so as to effectively predict the 5-year survival rate of the patient after surgery.

[0057] This solution uses enhanced CT for prognosis evaluation, without the need for liver puncture of the patient, which can effectively reduce the trauma risk and medical resources of the patient's prognosis estimation. Moreover, compared with plain CT, it can better reflect the tumor manifestation. At the same time, without the patient actually taking enhanced CT, that is, it can be used for prognosis evaluation with enhanced CT without consuming a large amount of medical resources required for taking enhanced CT, and there is also no radiation risk caused by injecting contrast agents. Moreover, this solution uses a deep learning model, which can generate enhanced CT after liver simulation surgery, and at the same time can perform prognosis evaluation based on the enhanced CT after liver simulation surgery, and visually evaluate the tumor situation.

[0058] To verify the feasibility and prediction effect of this solution, a large number of experiments were carried out in this embodiment, and some experimental data are provided here:

[0059] Figure 4 The ROC curve graph of the evaluation model for predicting the 5-year survival situation of patients in the present invention. It can be seen that the area under the ROC curve is 0.792 (0.744 - 0.841), which is relatively close to 1, indicating that this method can give a relatively accurate prognosis evaluation result and has good predictive value.

[0060] In addition, Figure 5a - Figure 5c The tumor identification area before patient A was treated with TACE is given. Figure 6 This is the plain CT of the liver of patient A after being treated with TACE. It can be seen that from the plain CT, it is difficult to clearly see whether there are still lesions in the patient's liver, and it is necessary to rely on the enhanced venous phase and enhanced arterial phase to judge. And Figure 7a and 7b , 8a and 8b prove that the predicted position and size of the lesions in the simulated liver enhanced CT scan images generated by this method are both close to the real scan results, and it has strong simulation ability.

[0061] In addition, doctors can also combine the enhanced CT after liver simulation surgery and the prognosis evaluation to give the final prognosis evaluation result, which can ensure a higher accuracy rate compared with the traditional method of doctors' own evaluation.

[0062] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Those skilled in the art to which the present invention pertains may make various modifications or supplements to the described specific embodiments or use similar means for substitution, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network, characterized in that, it includes the following steps: S1. Train a deep learning model to obtain an evaluation model that can give a prognosis evaluation result; S2. The evaluation model takes the plain CT after liver surgery and the postoperative serological test information as inputs and outputs a prognosis evaluation result; The deep learning model includes a first model and a second model. The first model is used to take the plain CT after liver surgery and the postoperative serological test information as inputs and output the enhanced CT after simulated liver surgery; The second model is used to take the enhanced CT after simulated liver surgery as an input and output a prognosis evaluation result; Use different sample data to train the first model and the second model respectively, and then cascade the trained first model and the second model to obtain an evaluation model for giving a prognosis evaluation result; The first model is a generative adversarial network. The trained generative adversarial network is used to combine serological test information to generate enhanced CT from plain CT; The generative adversarial network includes a generator, a discriminator and a classifier. The generator is used to generate enhanced CT from plain CT according to serological test information. The discriminator is used to perform adversarial training with the generator to ensure the image generation ability of the generator. The classifier is used to ensure that the images generated by the generator conform to serological test information to integrate serological test information into the generator; The method for the generative adversarial network to generate enhanced CT from plain CT by combining serological test information includes: S11. Convert the serological test information into a one-hot vector c; S12. Input the vector c, the vector z used to determine the image style, and the plain CT after liver surgery into the generator; S13. The generator generates the enhanced CT after simulated liver surgery according to the vector c, the vector z and the plain CT after liver surgery, and outputs the enhanced CT after simulated liver surgery to the discriminator and the classifier; S14. The discriminator judges the fidelity of the enhanced CT after simulated liver surgery. The classifier extracts a new vector from the enhanced CT after simulated liver surgery and judges whether the vector is consistent with the vector c. When the extracted vector is consistent with the vector c and the fidelity of the enhanced CT after simulated liver surgery is higher than the fidelity threshold, it is determined that the enhanced CT after simulated liver surgery has been obtained.

2. The method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network according to claim 1, characterized in that, the sample data for training the first model includes the plain CT before liver surgery, the enhanced CT before liver surgery and the preoperative serological test information.

3. The method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network according to claim 2, characterized in that, the sample data for training the first model includes the plain CT after liver surgery, the enhanced CT after liver surgery and the postoperative serological test information.

4. The method for evaluating the response after interventional treatment of liver tumors based on a generative adversarial network according to claim 3, characterized in that, In step S1, the second model is a convolutional network, and the trained convolutional network is used to output a prognosis evaluation result according to the enhanced CT; the sample data of the second model includes enhanced CT after liver surgery and postoperative prognosis data.

5. A post-interventional response evaluation system for liver tumors based on a generative adversarial network, characterized in that it is used to evaluate the post-interventional response of liver tumors using the evaluation method described in any one of claims 1-4.

Images