An Advantage Actor-Critic Algorithm 18 F-FDG PET / CT method for estimating dynamic parameters of liver cancer

Abstract

This invention relates to an algorithm based on the Advantage Actor-Critic algorithm. 18 This invention relates to a method for estimating hepatocellular carcinoma dynamic parameters using F-FDG PET / CT, belonging to the fields of reinforcement learning and hepatocellular carcinoma diagnosis. The invention employs a reinforcement learning algorithm to calculate perfusion parameters, including those in blood and liver tissue, on a dual-input three-compartment model. 18 The rate constant and blood clearance rate of F-FDG 18 F-FDG is further phosphorylated to 18 The perfusion parameters were obtained by estimating the rate constant of F-FDG-6-phosphatase, the dephosphorylation rate of phosphatase, the aortic blood supply fraction, and the proportion of the measured blood volume. Utilizing the characteristics of the Advantage Actor-Critic algorithm, a neural network was introduced to guide the agent to obtain relatively accurate perfusion parameters with statistically significant differences. This invention effectively uses the Advantage Actor-Critic algorithm on a reversible dual-input three-compartment model to estimate perfusion parameters corresponding to hepatocellular carcinoma tissue and normal liver tissue, and performs statistical analysis showing statistically significant differences.

Description

Technical Field

[0001] This invention provides a method based on the Advantage Actor-Critic algorithm. 18 The F-FDG PET / CT method for estimating dynamic parameters of liver cancer belongs to the fields of reinforcement learning and liver cancer diagnosis technology. Background Technology

[0002] Liver cancer refers to malignant tumors originating from hepatocytes and bile duct cells. It is divided into two main categories: primary and secondary, and is a common malignant tumor in my country. Hepatocellular carcinoma (HCC) is the most common primary malignant tumor of the liver. Early-stage liver cancer often presents with nonspecific symptoms, while mid-to-late-stage liver cancer has more symptoms. Common clinical manifestations include pain in the liver area, abdominal distension, loss of appetite, fatigue, weight loss, progressive hepatomegaly, or an upper abdominal mass. Some patients may experience low-grade fever, jaundice, diarrhea, or upper gastrointestinal bleeding. Liver cancer is predicted to be the sixth most common cancer.

[0003] Recent research has shown that reinforcement learning has potential for parameter estimation in room-compartment models. Existing papers have used temporal difference learning (TD-learning) and state-action-reward-state-action (SARSA) algorithms for parameter estimation in dynamical models, i.e., room-compartment models. However, these reinforcement learning algorithms require maintaining a Q-table to store the value of each state-action pair. When the state space or action space is large, the size of the Q-table becomes difficult to store and update, leading to data sparsity to some extent. Therefore, these algorithms have certain limitations in the agent's exploratory capabilities.

[0004] Technically, the Advantage Actor-Critic algorithm is a classic algorithm in reinforcement learning. It was proposed early on and has been validated in other parameter estimation fields, such as industrial parameter estimation and optimization. Introducing a neural network into the Advantage Actor-Critic algorithm can effectively alleviate the data sparsity problem of Q-table-based reinforcement learning algorithms. However, no one has yet applied the Advantage Actor-Critic algorithm to… 18 In the study of F-FDG PET / CT hepatocellular carcinoma dynamic parameter estimation, this invention employs the Advantage Actor-Critic algorithm with neural networks for this purpose. 18 Parameter estimation of F-FDG PET / CT hepatocellular carcinoma dynamics, among which, 18 F-FDG means 18F-fluorodeoxyglucose, PET / CT is positron emission tomography / computed tomography. Summary of the Invention

[0005] In view of the aforementioned defects or deficiencies in the existing technology, and in response to the research gaps regarding liver cancer, this invention proposes a method based on the Advantage Actor-Critic algorithm. 18 F-FDG PET / CT hepatocellular carcinoma dynamic parameter estimation method, implementing the Advantage Actor-Critic algorithm for... 18 Estimation of perfusion parameters between F-FDG PET / CT hepatocellular carcinoma tissue and normal liver tissue.

[0006] The technical solution of this invention is: a method based on the Advantage Actor-Critic algorithm. 18 Methods for estimating dynamic parameters of liver cancer using F-FDG PET / CT include the following:

[0007] Step 1: Construct a reinforcement learning interactive environment based on parameter estimation of a dual-input three-compartment model, including defining dynamic parameters, state, action distribution and action space, designing reward function and resetting the environment logic;

[0008] Step 2: Build the overall framework of the Advantage Actor-Critic algorithm, including the construction of the policy network (Actornetwork) and the value function network (Critic network);

[0009] Step 3: Use the Advantage Actor-Critic algorithm to estimate parameters and compare the perfusion parameters of hepatocellular carcinoma tissue and normal liver tissue.

[0010] As a further aspect of the present invention, the specific steps of Step 1 are as follows:

[0011] Step 1.1. Construct a reversible two-input three-compartment model. The parameters of the reversible two-input three-compartment model include those in blood and liver tissue. 18 F-FDG rate constant, blood clearance rate, 18 F-FDG is further phosphorylated to 18 The rate constant of F-FDG-6-phosphatase, the dephosphorylation rate of phosphatase, the fractional blood supply to the aorta, and the proportion of the measured blood volume.

[0012] Step 1.2. Build a reinforcement learning interactive environment based on parameter estimation of a two-input three-compartment model; define the six parameters of the two-input three-compartment model as the state vector in the reinforcement learning environment and set initial values; the increase or decrease of these six parameters is used as the action space vector; define the method for calculating the fitting error as root-mean-square error (RMSE);

[0013] Step 1.3. Design the reward function into two parts: a piecewise reward signal that guides the increase of reward as RMSE decreases, and a reward signal that is positive or negative based on the difference between the error size obtained by adjusting the current parameters and the error size obtained by adjusting the parameters in the previous round.

[0014] Step 1.4. When the parameter value exceeds the upper and lower bounds of the preset clinical prior knowledge parameters, reset the interactive environment and the dynamic parameters of the dual-input three-compartment model.

[0015] As a further aspect of the present invention, Step 2 includes: constructing the structure of a policy network and a value function network. The dynamic parameters of the dual-input three-compartment model will be used as the common input of the two networks. The policy network outputs a 12-dimensional action adjustment probability distribution after convolution and normalization, and the value function network outputs the value of the action that the network considers most suitable after convolution.

[0016] As a further aspect of the present invention, Step 3 includes: testing the fitting results using T-test, area under the curve (AUC), and the magnitude of the fitting RMSE error, and determining whether there is a statistically significant difference in perfusion parameters between hepatocellular carcinoma tissue and normal liver tissue based on the P-value.

[0017] The beneficial effects of this invention are:

[0018] 1. Implement the Advantage Actor-Critic algorithm for... 18 Estimation of perfusion parameters between F-FDG PET / CT hepatocellular carcinoma tissue and normal liver tissue;

[0019] 2. The estimated perfusion parameters of hepatocellular carcinoma tissue and normal liver tissue showed statistically significant differences. Attached Figure Description

[0020] Figure 1 This is a flowchart from the present invention;

[0021] Figure 2 This invention describes the delineation of ROI maps on CT images and PET / CT images.

[0022] Figure 3This is a structural diagram of the Advantage Actor-Critic algorithm that combines a reversible dual-input three-compartment model according to the present invention. Detailed Implementation

[0023] The abbreviations in this invention embodiment mean: K1 (in blood and liver tissue) 18 F-FDG rate constant), k2 (blood clearance rate), k3 ( 18 F-FDG is further phosphorylated to 18 F-FDG-6-phosphatase rate constant), k4 (phosphatase dephosphorylation rate), fa (aortic blood supply fraction), vb (proportion of measured blood volume), A2C (Advantage Actor-Critic algorithm).

[0024] Example 1: As Figures 1-3 As shown, an Advantage Actor-Critic algorithm... 18 Methods for estimating dynamic parameters of liver cancer using F-FDG PET / CT include the following:

[0025] Experimental materials for this embodiment: 14 patients underwent PET / CT scans and metabolic PET / CT scans; patient data were provided by the Southern Pet Center of Southern Medical University Southern Hospital, and written informed consent was obtained from all patients; this study was approved by the Yunnan Provincial People's Hospital University Review Committee, with the approval number KHLL2022-KY189. This embodiment included 14 patients (13 males and 1 female), aged 31–78 years; 17 cases were diagnosed with liver cancer, with the longest axis diameter of the tumor ranging from 6.5 ± 3.6 cm (1.9–15.0 cm).

[0026] Initial PET / CT scan: After the patient fasted for more than 6 hours and their blood glucose stabilized, a liver CT scan (120 kV, 100 mAs) was performed; 5.5 MBq / kg was administered intravenously. 18Following F-FDG, a 5-minute dynamic PET scan of the liver region was performed, followed by a 1-minute static PET scan (60 minutes total). The dynamic PET data was segmented into 17 frames: 12 frames at 5 seconds, 4 frames at 60 seconds, and 1 additional frame at 60 minutes, and reconstructed using the standard ordered subset expectation-maximization algorithm. Regions of interest (ROIs) in the CT images were manually defined for each patient, including hepatocellular carcinoma, normal liver tissue, aorta, and portal vein (especially the extrahepatic portion). After image fusion, the ROIs were copied onto the PET / CT images to generate time activity curves (TACs), and the maximum standard uptake values ​​(SUVmax) were extracted from each frame. Figure 2 The image shows the ROIs (Regions of Interest) delineated at corresponding locations in a CT image and a PET / CT image. The black area represents the outline of hepatocellular carcinoma, the yellow area represents the aorta, the red area represents the portal vein, and the green area represents normal liver tissue.

[0027] This embodiment presents an Advantage Actor-Critic algorithm. 18 The specific steps for estimating the dynamic parameters of liver cancer using F-FDG PET / CT are as follows:

[0028] Step 1: Construct a reinforcement learning interactive environment based on parameter estimation of a dual-input three-compartment model, including defining dynamic parameters, state, action distribution and action space, designing reward function and resetting the environment logic;

[0029] Step 2: Build the overall framework of the Advantage Actor-Critic algorithm, including the construction of the policy network (Actornetwork) and the value function network (Critic network);

[0030] Step 3: Use the Advantage Actor-Critic algorithm to estimate parameters and compare the perfusion parameters of hepatocellular carcinoma tissue and normal liver tissue.

[0031] As a further aspect of the present invention, the specific steps of Step 1 are as follows:

[0032] Step 1.1. Construct a reversible two-input three-compartment model. The parameters of the reversible two-input three-compartment model include K1 (in blood and liver tissue). 18 F-FDG rate constant), k2 (blood clearance rate), k3 ( 18 F-FDG is further phosphorylated to 18The rate constant of F-FDG-6-phosphatase, k4 (dephosphorylation rate of phosphatase), fa (fraction of aortic blood supply), and vb (proportion of the measured blood volume).

[0033] Step 1.2. Build a reinforcement learning interactive environment based on parameter estimation of a two-input three-compartment model; define the six parameters of the two-input three-compartment model as the state vector in the reinforcement learning environment and set initial values; the increase or decrease of these six parameters is used as the action space vector; define the method for calculating the fitting error as root-mean-square error (RMSE).

[0034] Step 1.3. The reward function is designed in two parts: a piecewise reward signal that guides the increase in reward as RMSE decreases, and a reward signal that indicates the sign of the difference between the error magnitude obtained from the current parameter adjustment and the error magnitude obtained from the previous round of parameter adjustment. The reward function is defined as follows:

[0035]

[0036]

[0037] reward=λ1·r1+λ2·r2 (3)

[0038] Where err represents the fitting error, and this invention uses RMSE. △err represents the difference between the current error and the error of the previous parameter adjustment, and λ1 and λ2 represent the weights of the two reward signals.

[0039] Step 1.4. When the parameter value exceeds the upper and lower bounds of the preset clinical prior knowledge parameters, reset the interactive environment and the dynamic parameters of the dual-input three-compartment model.

[0040] As a further aspect of the present invention, Step 2 includes: constructing the structures of a policy network and a value function network. The dynamic parameters of the dual-input three-compartment model are used as the common inputs of the two networks. The policy network outputs a 12-dimensional action adjustment probability distribution after convolution and normalization, while the value function network outputs the value of the action it deems most suitable after convolution. 18 The structure diagram of the Advantage Actor-Critic algorithm, which combines a reversible dual-input three-compartment model for estimating the dynamic parameters of F-FDG PET / CT liver cancer, is shown below. Figure 3 As shown.

[0041] As a further aspect of the present invention, Step 3 includes: testing the fitting results using a T-test, area under the curve (AUC), and RMSE error, and determining whether there is a statistically significant difference in perfusion parameters between hepatocellular carcinoma tissue and normal liver tissue based on the P-value. If P < 0.05, the difference is considered statistically significant. The closer the AUC value is to 1, the better the model performance; the closer it is to 0.5, the worse the model performance.

[0042] T-test: A statistical method used to compare whether there is a significant difference between the means of two groups of data.

[0043] Area Under the Curve (AUC): The area under the ROC curve, used to evaluate the performance of a binary classification model.

[0044] RMSE error: Calculate the average of the squared differences between the predicted and observed values, and then take the square root.

[0045] Following the experimental procedure, the test was conducted on data from 12 HCC patients. The specific parameter estimation results and experimental evaluation indicators are shown in Table 1.

[0046] Table 1 Experimental results of the present invention

[0047] <![CDATA[K 1( (ml / min / ml)]]> 0.8896±0.0283 0.8213±0.0255 <0.001 0.934 <![CDATA[k2(1 / min)]]> 0.8531±0.0175 0.8304±0.0124 <0.001 0.837 <![CDATA[k3(1 / min)]]> 0.0367±0.0180 0.0184±0.0090 <0.001 0.900 <![CDATA[k4(1 / min)]]> 0.0187±0.0029 0.0207±0.0016 0.029 0.720 <![CDATA[f a ]]> 0.6606±0.1146 0.3394±0.0891 <0.001 0.941 <![CDATA[v b ]]> 0.0549±0.0022 0.0587±0.0013 <0.001 0.917 RMSE 2.5057±1.0353 1.0584±0.5354 / /

[0048] Pharmacokinetics includes blood perfusion and intracellular metabolism. Blood perfusion-related parameters are fa and vb, while intracellular metabolism-related parameters are K1–K4. As shown in Table 1, fa was significantly elevated in HCC (P<0.001), a finding consistent with the assumption that dual infusion is the actual blood supply pattern for most HCC tumors, and also indicating that hepatic artery blood flow accounts for the majority, but not all, of the total blood flow in HCC. vb was also significantly elevated (P<0.001), indicating that the blood supply intensity in hepatocellular carcinoma areas differs from that in normal liver tissue.

[0049] Regarding intracellular metabolic parameters, compared with normal liver tissue, HCC showed significantly increased K1, K2, and K3 (P<0.001), and significantly decreased K4 (P<0.05). The significant increases in K1 and K3 indicate increased glucose transporter and hexokinase activity in hepatocellular carcinoma, while K2 activity reflects the difference between HCC and normal liver tissue. 18 The clearance rate of F-FDG transported back into the blood varies. The significantly lower K4 level in HCC compared to normal liver tissue is due to increased G6P activity.

[0050] In summary, the Advantage Actor-Critic algorithm provides relatively accurate parameter estimation for hepatocellular carcinoma and normal liver tissue, and the estimated parameters show significant differences, which will help doctors diagnose early-stage hepatocellular carcinoma.

[0051] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method based on the Advantage Actor-Critic algorithm 18 The method for estimating dynamic parameters of hepatocellular carcinoma using F-FDG PET / CT is characterized by, Specifically, the following steps are included: Step 1: Construct a reinforcement learning interactive environment for parameter estimation of a reversible two-input three-compartment model, including defining dynamic parameter states, action distribution and action space, designing reward function, and resetting the environment logic; Step 2: Build the overall framework of the Advantage Actor-Critic algorithm, including the construction of the policy network (Actornetwork) and the value function network (Critic network); Step 3: Use the Advantage Actor-Critic algorithm to estimate parameters and compare the perfusion parameters of hepatocellular carcinoma tissue and normal liver tissue; The specific steps of Step 1 are as follows: Step 1.

1. Construct a reversible two-input three-compartment model. The parameters of the reversible two-input three-compartment model include those in blood and liver tissue. 18 F-FDG rate constant, blood clearance rate, 18 F-FDG is further phosphorylated to 18 The rate constant of F-FDG-6-phosphatase, the dephosphorylation rate of phosphatase, the fractional blood supply to the aorta, and the proportion of the measured blood volume. Step 1.

2. Construct a reinforcement learning interactive environment for parameter estimation of a reversible two-input three-compartment model; define the six parameters of the reversible two-input three-compartment model as the state vector in the reinforcement learning environment and set initial values; the increase or decrease of these six parameters is used as the action space vector; define the method for calculating the fitting error as root mean square error; Step 1.

3. Design the reward function into two parts: a piecewise reward signal that guides the increase of reward as RMSE decreases, and a reward signal that is positive or negative based on the difference between the error size obtained by adjusting the current parameters and the error size obtained by adjusting the parameters in the previous round. Step 1.

4. When the parameter value exceeds the upper and lower bounds of the preset clinical prior knowledge parameters, reset the interactive environment and the dynamic parameters of the reversible dual-input three-compartment model; Step 2 includes: constructing the structure of the policy network and the value function network. The dynamic parameters of the reversible dual-input three-compartment model will be used as the common input of the two networks. The policy network outputs a 12-dimensional action adjustment probability distribution after convolution and normalization, and the value function network outputs the value of the action that the network considers most suitable after convolution.

2. The algorithm based on Advantage Actor-Critic as described in claim 1 18 The method for estimating dynamic parameters of hepatocellular carcinoma using F-FDG PET / CT is characterized by: Step 3 includes: testing the fitting results using T-test, area under the curve (AUC), and the magnitude of the fitting RMSE error, and determining whether there is a statistically significant difference in perfusion parameters between hepatocellular carcinoma tissue and normal liver tissue based on the P-value.

Images