Result simulation method, device and equipment based on generative adversarial network

Abstract

Embodiments of the present application provide a result simulation method and device based on a generative adversarial network, and an equipment, and relate to the field of image processing. The method comprises: inputting a CT image and a proton beam flow parameter into a trained generative adversarial network; after the CT image is encoded and decoded by a primary generator, a basic tissue deformation field is obtained; after the basic tissue deformation field and a Bragg peak distribution diagram obtained based on the proton beam flow parameter are subjected to feature fusion and attention weight mapping by a fine-tuning generator, an enhanced image and an output feature vector are obtained; after the output feature vector is subjected to spatial feature discrimination in a double-channel discriminator, an image authenticity result is obtained; after the output feature vector is subjected to dosimetric feature discrimination in the double-channel discriminator, a dosimetric evaluation result is obtained; and after the enhanced image is discriminated according to the image authenticity result and the dosimetric evaluation result, an output image of the trained generative adversarial network is obtained. The embodiments provided in the present application improve the prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a result simulation method based on a generative adversarial network, a result simulation device based on a generative adversarial network and an electronic device. BACKGROUND

[0002] Current proton therapy plans are mostly designed and implemented based on pre-treatment static CT images. However, in actual treatment, the position, geometry and biological characteristics of tumors and surrounding normal tissue organs all show dynamic changes. For example, physiological activities such as patient breathing, swallowing, heartbeat, and tumor regression or growth during treatment will all cause real-time changes in anatomical structures. If the initial static treatment plan is still followed for irradiation, it will inevitably lead to off-target (i.e. the proton beam flow fails to accurately irradiate the tumor) and increase the irradiation dose of the surrounding normal tissue organs, thereby reducing the efficacy of tumor treatment and increasing the risk of radiotherapy side effects.

[0003] In proton radiotherapy, the dose conformality and limited range characteristics of the proton beam flow make it extremely sensitive to changes in anatomical structures. Any minor changes, such as a reduction in tumor volume or a shift in normal tissue, can cause significant changes in dose distribution. For example, a reduction in tumor volume can result in insufficient penetration depth of the originally planned proton beam flow, thereby affecting tumor control rate; at the same time, a shift in normal tissue can result in unexpected high-dose irradiation, increasing the risk of side effects. Therefore, adaptive radiotherapy for proton therapy has become a necessary requirement in clinical practice to adjust the treatment plan in real time to adapt to the dynamic changes in patient anatomy.

[0004] The accurate position of the proton Bragg peak is the key to achieving precise treatment, but its accurate positioning in the patient's body is affected by the uncertainty of the proton range. The uncertainty of the proton range is mainly due to the influence of CT imaging artifacts on tissue density. Artifacts in CT images (such as metal artifacts, motion artifacts, etc.) can disrupt the accurate measurement of Hounsfield Unit (HU) tissue density, thereby affecting the conversion and calibration of the proton Stopping Power Ratio (SPR), leading to deviations in the Bragg peak position. This problem is particularly prominent in proton therapy, because the dose deposition of the proton beam flow is highly concentrated, and small errors in the range can cause significant changes in the dose distribution. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a result simulation method, device and equipment based on a generative adversarial network, to solve the problem of insufficient accuracy of side effect prediction and long-term efficacy tracking in proton therapy, especially the technical challenges in dynamic simulation, long-term tracking and multi-center data standardization, to at least solve part of the problems in the background art.

[0006] To achieve the above-mentioned purpose, in the present application, a result simulation method based on a generative adversarial network is provided, the method comprising: inputting a CT image and a proton beam flow parameter into a trained generative adversarial network, the generative adversarial network comprising a primary generator, a fine-tuning generator and a double-channel discriminator; after the CT image is encoded and decoded by the primary generator, a basic tissue deformation field is obtained; after the basic tissue deformation field and a Bragg peak distribution map obtained based on the proton beam flow parameter are feature fused and attention weight mapped by the fine-tuning generator, an enhanced image and an output feature vector are obtained; after the output feature vector is processed by a spatial feature discrimination branch in the double-channel discriminator, an image authenticity result is obtained, and after the output feature vector is processed by a dosimetric feature discrimination branch in the double-channel discriminator, a dosimetric evaluation result is obtained; after the enhanced image is discriminated according to the image authenticity result and the dosimetric evaluation result, an output image of the trained generative adversarial network is obtained.

[0007] Optionally, after the output image of the generative adversarial network is obtained, the method further comprises: calculating the difference between the output image and the input CT image; and obtaining the influence of the proton beam flow based on the proton beam flow parameter on different types of regions in the input CT image according to the difference.

[0008] Optionally, the different types of regions include tumor regions; calculating the difference between the output image and the input CT image comprises: segmenting a tumor region from the output image based on an adaptive segmentation algorithm and calculating a first volume of the tumor region; segmenting a tumor region from the input CT image based on the adaptive segmentation algorithm and calculating a second volume of the tumor region; and obtaining a tumor volume difference according to the first volume and the second volume.

[0009] Optionally, the different types of regions include normal regions; calculating the difference between the output image and the input CT image comprises: calculating a Monte Carlo dose according to a nuclear physics engine based on the output image and the proton beam flow parameter; and obtaining a dose distribution difference between a dose distribution map of a normal region in the input CT image and a dose distribution map of a normal region in the output image.

[0010] Optionally, before inputting the CT image and the proton beam flow parameter into the trained generative adversarial network, the method further comprises: using a domain adaptation generative adversarial network to establish an image standardization channel to perform grayscale normalization and spatial registration on multi-center CT images while retaining high-frequency texture features.

[0011] Optionally, the method further comprises: constructing a multi-modal incremental learning framework to integrate follow-up CBCT images, biomarker data and quality of life survey scores, and consolidating parameters of the trained generative adversarial network through dynamic updating of the algorithm by using elastic weights.

[0012] Optionally, the method further comprises: deploying a real-time dose tracking system for visualizing and displaying risks corresponding to doses; the real-time dose tracking system comprises: a deformation registration module for aligning images using a non-rigid registration algorithm; an online reconstruction engine for decomposing a proton beam into a plurality of pencil beams, respectively calculating dose deposition of the pencil beams, and then superimposing contributions of all the pencil beams to generate a three-dimensional dose distribution and output a dose matrix; and a risk visualization interface for generating a color-coded risk map with dose volume constraints based on the dose matrix.

[0013] Optionally, the method further comprises: designing a special prediction subnetwork for each type of impact result, and obtaining a risk probability of occurrence of the impact result according to an output of the special prediction subnetwork; the special prediction subnetwork comprises: an input layer, a feature fusion layer and an output layer; the input layer comprises: a dosimetric feature branch, a biomarker branch and an image feature branch; the feature fusion layer is configured to perform feature fusion by using a cross-attention mechanism to obtain a fusion result; and the output layer is configured to output, through activation, a risk probability of occurrence of the impact result.

[0014] The application also provides a result simulation device based on a generative adversarial network, comprising: a data input module configured to input a CT image and a proton beam flow parameter into a trained generative adversarial network, the generative adversarial network comprising a primary generator, a fine-tuning generator and a double-channel discriminator; a first processing module configured to obtain a basic tissue deformation field by encoding and decoding the CT image through the primary generator; a second processing module configured to obtain an enhanced image and an output feature vector by performing feature fusion and attention weight mapping on the basic tissue deformation field and a Bragg peak distribution map obtained based on the proton beam flow parameter through the fine-tuning generator; a third processing module configured to obtain an image authenticity result by inputting the output feature vector through a spatial feature discrimination branch in the double-channel discriminator, and obtain a dosimetric evaluation result by inputting the output feature vector through a dosimetric feature discrimination branch in the double-channel discriminator; and a data output module configured to obtain an output image of the trained generative adversarial network by discriminating the enhanced image according to the image authenticity result and the dosimetric evaluation result.

[0015] An electronic device is also provided in the present application, comprising: at least one processor; a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the foregoing result simulation method based on a generative adversarial network by executing the instructions stored in the memory.

[0016] A machine readable storage medium is also provided in the present application, and the machine readable storage medium stores instructions, which, when executed by a processor, cause the processor to be configured to implement the foregoing result simulation method based on a generative adversarial network.

[0017] A computer program product is also provided in the present application, comprising a computer program, which, when executed by a processor, implements the foregoing result simulation method based on a generative adversarial network.

[0018] The foregoing technical solutions have the following beneficial effects:

[0019] By deeply fusing a generative adversarial network, real-time monitoring technology and an individualized prediction model, the accuracy of automatically generating an image of a CT image affected by a proton beam is improved. Accurate data basis is provided for the precision and efficiency of subsequent proton therapy side effect prediction and efficacy tracking. The method provides an efficient auxiliary tool for clinicians, promotes the development of proton therapy in a more accurate and more personalized direction, and has significant clinical application value.

[0020] Other features and advantages of the embodiments of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the embodiments of the present application together with the following specific implementation, but do not constitute a limitation to the embodiments of the present application. In the drawings:

[0022] Figure 1 A schematic diagram of steps of the result simulation method based on a generative adversarial network in the embodiments of the present application is schematically shown;

[0023] Figure 2 A schematic diagram of a data processing process of the generative adversarial network in the embodiments of the present application is schematically shown;

[0024] Figure 3 A schematic diagram of an output image of the generative adversarial network in the embodiments of the present application is schematically shown;

[0025] Figure 4A schematic diagram of a prediction sub-network structure according to an embodiment of the present application is shown for a specific impact;

[0026] Figure 5 An implementation schematic diagram of a result simulation method based on a generative adversarial network according to an embodiment of the present application is shown;

[0027] Figure 6 A structural schematic diagram of a result simulation device based on a generative adversarial network according to an embodiment of the present application is shown;

[0028] Figure 7 An internal structure diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present application, and are not intended to limit the present application.

[0030] Figure 1 A step schematic diagram of a result simulation method based on a generative adversarial network according to an embodiment of the present application is shown. As shown in Figure 1 A result simulation method based on a generative adversarial network, the method comprising:

[0031] S01, inputting a CT image and a proton beam flow parameter into a trained generative adversarial network, the generative adversarial network comprising a primary generator, a fine-tuning generator and a double-channel discriminator; wherein the training of the generative adversarial network can use the following data samples, 200 cases of high-resolution CT images of head and neck tumor patients before treatment (layer thickness 1 mm, matrix 512x512) are obtained, and proton treatment parameters are synchronously collected, including beam energy (70-230 MeV), Bragg peak broadening (SOBP width 3-10 mm) and target area dose prescription (54-70 Gy). Perform isotropic resampling of the CT image to 1x1x1 mm3, and realize gray scale standardization by histogram matching (reference window width: 1500 HU, window level: -500 HU).

[0032] S02, the CT image is encoded and decoded by the primary generator to obtain a basic tissue deformation field; the primary generator adopts a 3D U-Net architecture, and the input is a CT image of 512x512x256 voxels, and the gray value may need to be normalized to [0, 1]. The encoder used for encoding includes 5 layers of 3D convolution, which can be selected with a step of 2, a convolution kernel of 3x3x3, padding of 1, and a ReLU activation function and batch normalization (Batch Normalization) after each layer of convolution; the decoder used for decoding restores the spatial resolution through bilinear upsampling and skip connection, and outputs the basic tissue deformation field, which can be selected with a size of 512x512x256, a channel number of 3, and is used to simulate the continuity change of anatomical structure before and after treatment.

[0033] S03, the basic tissue deformation field and the Bragg peak distribution diagram obtained based on the proton beam parameter are fused and attention weight mapped by the fine adjustment generator to obtain an enhanced image and an output feature vector. On the basis of the primary generator, the fine adjustment generator adds an attention gate module (Attention Gate), which inputs the fused proton Bragg peak distribution diagram, which can be selected with a size of 512x512x256 and a channel number of 1, dynamically adjusts the feature weight through the attention mechanism, and focuses on the key area of dose deposition. The specific steps of the attention gate module providing the attention mechanism include: inputting the fused proton Bragg peak distribution diagram, denoted as X.

[0034] 1) The output feature map F of the primary generator, with a size of 512x512x256 and a channel number C, is spliced with the Bragg peak distribution diagram X, with a size of 512x512x256 and a channel number of 1, in the channel dimension to obtain a fused feature map;

[0035] 2) Attention weight calculation:

[0036] wherein, and are 1x1x1 convolution layers that map the fused feature map to the attention weight, is an activation function, and the output attention weight map A has a size of 512x512x256 and a channel number of 1;

[0037] 3) Feature weighting

[0038] wherein, represents element-wise multiplication to obtain the feature weighted .

[0039] 4) Output feature , as the input of the next layer.

[0040] S04, the output feature vector is input into the spatial feature discrimination branch in the dual-channel discriminator to obtain an image authenticity result, and the output feature vector is input into the dosimetric feature discrimination branch in the dual-channel discriminator to obtain a dosimetric evaluation result; wherein the spatial feature discrimination branch is composed of a 7-layer 3D convolutional network, the convolution kernel sizes are 4x4x4, 3x3x3, …, 1x1x1 in sequence, the step length is 2, and an image authenticity score is output, the score interval being (0, 1). The dosimetric feature discrimination branch adopts a graph neural network (GNN), inputs a dose distribution map, the dose distribution map can be represented by DVH features, evaluates the physical rationality of dose deposition, and outputs a dosimetric score interval consistent with the image authenticity score interval, which can be selected as (0, 1). The above dual-channel design ensures the dual accuracy of the generated image in anatomical structure and dosimetric characteristics.

[0041] S05, the output image of the trained generative adversarial network is obtained after the enhanced image is discriminated according to the image authenticity result and the dosimetric evaluation result. The discrimination is performed by a discriminator in the generative adversarial network, and the implementation mechanism is based on the operation mechanism of the generative adversarial network.

[0042] Through the above implementation, the accuracy of the automatically generated CT image after the influence of the proton beam is improved by deep fusion of the generative adversarial network, real-time monitoring technology and individualized prediction model.

[0043] Figure 2 A schematic diagram of a data processing process of the generative adversarial network in the embodiments of the present application is schematically shown. As shown in Figure 2 , the input data, output data and data processing flow of the primary generator, fine-tuning generator and dual-channel discriminator are schematically shown.

[0044] In some optional embodiments, the loss function adopts a weighted combination: the loss function L total adopts a weighted combination of the adversarial loss L adv and the deformation consistency loss L consist , and the specific form is , wherein α=0.7, β=0.3.

[0045] In some embodiments of the present application, after obtaining the output image of the generative adversarial network, the method further comprises: calculating the difference between the output image and the input CT image; and obtaining the influence of the proton beam flow based on the proton beam flow parameters on different types of regions in the input CT image according to the difference.

[0046] In some embodiments of the present application, the different types of regions include tumor regions; the difference between the output image and the input CT image is calculated, including: segmenting the tumor region from the output image based on an adaptive segmentation algorithm, and calculating a first volume of the tumor region; segmenting the tumor region from the input CT image based on the adaptive segmentation algorithm, and calculating a second volume of the tumor region; obtaining a tumor volume difference according to the first volume and the second volume. Wherein, the adaptive segmentation algorithm includes two stages:

[0047] The first stage: based on an improved V-Net model, input CT image and initial dose distribution map. By introducing residual connection (Residual Block), the problem of gradient disappearance is reduced, and deep supervision mechanism (Deep Supervision) is added in the decoder, forcing the network to pay attention to multi-scale features. (Dice coefficient > 0.85);

[0048] The second stage: applying conditional random field (CRF) to post-process the initial segmentation result, and combining the dose gradient map to generate a risk-sensitive segmentation mask. CRF optimizes the boundary smoothness and dose sensitivity through the energy function, ensuring that the segmentation result not only accurately reflects the tumor morphology, but also highlights the high-dose risk area, providing accurate basis for subsequent risk assessment. The specific steps of conditional random field are:

[0049] 1) Calculate the conditional probability distribution of CRF:

[0050]

[0051] Wherein, X represents the observation sequence, Y represents the label sequence, and each node represents a label associated with the observation . represents the relationship between the observation and the label , represents the relationship between adjacent labels and , Z(X) represents the normalization factor, which is used to ensure that the sum of the probability distribution is 1;

[0052] 2) Maximize the log-likelihood function: ;

[0053] 3) Inference stage, find the most likely label sequence .

[0054] Through the present embodiment, the change characteristics of the tumor volume can be obtained.

[0055] In some embodiments of the present application, the different types of regions include normal regions; the difference between the output image and the input CT image is calculated, including: based on the output image and the proton beam parameters, a Monte Carlo dose is calculated according to the nuclear physics engine; and according to the Monte Carlo dose, a dose distribution difference between a dose distribution map of the normal region in the input CT image and a dose distribution map of the normal region in the output image is obtained. The embodiment includes: obtaining the output image of step S05; performing Monte Carlo dose calculation according to a Monte Carlo dose calculation model; obtaining a biological effect according to the Monte Carlo dose and a mapping of dose to biological effect; and quantifying the risk of the biological effect.

[0056] The input parameters of the embodiment are the output image of step S05 and the proton beam parameters, and the output parameter is a damage risk heat map of healthy tissue.

[0057] The Monte Carlo dose calculation formula is:

[0058] ;

[0059] Wherein, r is the tissue position, E is the energy, x represents the proton, is the energy deposited per unit mass, is the tissue density from CT, is the proton collision stopping power, is the proton flux density.

[0060] The mapping of dose to biological effect includes:

[0061]

[0062] Wherein, e is a natural constant, D is the energy deposited per unit mass, IL-6 is interleukin-6, a, b, and c are fitting parameters, and P is a biological effect.

[0063] Figure 3 The schematic diagram of the output image generated by the generative adversarial network in the embodiment of the present application is schematically shown. As shown in FIG. 6, it shows the identification and division of the tumor segmentation and the risk region in the output image. Figure 3

[0064] ​In some embodiments of the present application, before the CT image and the proton beam flow parameter are input into the trained generative adversarial network, the method further comprises: using a domain adaptation generative adversarial network to establish an image standardization channel, performing gray scale normalization and spatial registration on multi-center CT images while preserving high-frequency texture features. Specifically, using a domain adaptation generative adversarial network to establish an image standardization channel, performing gray scale normalization (μ = 1200 HU, σ = 50 HU) and spatial registration (resolution 1x1x1 mm³) on multi-center CT images while preserving more than 5% of high-frequency texture features, i.e., edge information such as organ edges, lesion areas, and texture details of microstructures inside tissues, to ensure data consistency and cross-institutional applicability of the model.

[0065] In some embodiments of the present application, the method further comprises: constructing a multi-modal incremental learning framework, integrating follow-up CBCT images, biomarker data, and quality of life survey scores, and consolidating the parameters of the trained generative adversarial network through an elastic weight to dynamically update the algorithm. The present embodiment provides incremental learning and model optimization, which mainly includes the following steps: Step 1, multi-modal data integration. Construct an incremental learning data set containing the following elements: image data: follow-up CBCT images (slice thickness 2 mm, registration error <1 mm); biomarkers: serum IL-6, TGF-β1 concentrations (detection frequency: once a week); quality of life survey scores: CTC-AE 4.0 toxicity grading, EORTC QLQ-C30 quality of life scores. Step 2, apply elastic weight consolidation algorithm (Elastic Weight Consolidation, EWC). Importance weight calculation: calculate Fisher information matrix every 200 new data, and select key parameters (threshold: Top 20%); parameter update rules include:

[0066] for param in model.parameters():

[0067] update = new_grad - λ * fisher_matrix[param] * (param - old_param)

[0068] param.data -= learning_rate * update.

[0069] Wherein, the elastic coefficient is found by grid search on the validation set λ = 0.75, the learning rate adopts the cosine annealing strategy (initial value 0.001, adjustment period 50 epochs), dynamically optimizes the convergence process, and avoids catastrophic forgetting; Memory management: retain the gradient information of the first 50 key cases, and adopt a ring buffer strategy to prevent data forgetting. Preferably, the multi-modal fusion adopts a cross-attention mechanism:

[0070]

[0071] Wherein, the query vector Q comes from the image feature, and the key-value pair K, V comes from the clinical data, realizes feature alignment, dk is the dimension of the key, and is set to 64.

[0072] In some embodiments of the application, the method further comprises: deploying a real-time dose tracking system for visualizing the risk corresponding to the dose; the real-time dose tracking system is mainly used for: based on daily CBCT images (registration error <1 mm), using a dynamic deformation registration algorithm (Demons algorithm, iteration 50 times, step 0.5 mm) to update the organ segmentation contour, combined with online dose reconstruction (GPU accelerated proton pencil beam algorithm, calculation speed ≥10 fields / second) to generate a risk heat map. The system comprises:

[0073] A deformation registration module is used to align images using a non-rigid registration algorithm. Specifically, a Demons algorithm (iteration 50 times, step 0.5 mm) is used to realize non-rigid registration of daily CBCT and planning CT, with an error of <1 mm.

[0074] Wherein, the Demons non-rigid registration algorithm is used to align images by optimizing the energy function, including:

[0075]

[0076] Wherein, represents the CBCT image to be registered, represents the CT image to be registered, x represents the gray value of the image at the coordinate, and u represents the displacement field, represents the control of the smoothing term weight. Then the organ contour is updated, including: based on the fast marching method (Fast Marching) to segment the tumor boundary, set the growth speed factor: F(x) = 1 / (1 + |∇I(x)|), wherein I(x) is the image gray value, and ∇I(x) is the gray gradient.

[0077] An online reconstruction engine is used to decompose the proton beam into multiple pencil beams, calculate the dose deposition of each pencil beam respectively, then superimpose the contribution of all pencil beams to generate a three-dimensional dose distribution, and output a dose matrix; a GPU-accelerated proton pencil beam algorithm is used to calculate at a speed of ≥10 fields per second, preferably at a speed of up to 15 fields per second.

[0078] In addition, a risk visualization interface is used to generate a color-coded risk map with dose volume constraints based on the dose matrix. For example, a risk heat map is generated as follows: red warning area: D95%≥50Gy (such as brainstem, optic nerve); yellow observation area: 30Gy<D95%<50Gy (such as parotid gland, temporal lobe); green safe area: D95%≤30Gy.

[0079] In some embodiments, the method further comprises generating a corresponding alarm according to the color-coded risk map or the risk heat map. For example, when the following conditions occur: the cumulative dose around the spinal cord is ≥45 Gy and the tumor volume reduction rate is <5% for 3 consecutive days, a level 3 alarm is triggered, indicating the risk of drug resistance.

[0080] In some embodiments of the present application, the method further comprises: designing a special prediction subnetwork for each impact result, and obtaining the risk probability of the occurrence of the impact result according to the output of the special prediction subnetwork; the special prediction subnetwork comprises: an input layer, a feature fusion layer and an output layer; the input layer comprises: a dosimetric feature branch, a biomarker branch and an image feature branch; the feature fusion layer is used to perform feature fusion by using a cross-attention mechanism to obtain a fusion result; and the output layer is used to output the risk probability of the occurrence of the impact result after activation of the fusion result. In order to specifically predict a certain impact, the present embodiment provides a special prediction subnetwork. The core goal of the special prediction subnetwork is to improve the prediction accuracy through multi-modal data fusion to solve the problem that traditional single subnetwork is difficult to efficiently process. Among them, the special prediction subnetwork architecture comprises: an input layer: a dosimetric feature branch (DYH parameter), a biomarker branch (IL-6, TGF-β1), and an image feature branch (GAN generated high-risk area). The feature fusion layer adopts a cross-attention mechanism. The output layer adopts a Sigmoid function to output the side effect risk probability. Figure 4 An illustrative diagram of the prediction subnetwork structure for a specific impact according to an embodiment of the present application is shown. Figure 4 As shown, it illustrates the prediction subnetwork structure for the impact result of neurotoxicity and the output of the prediction result.

[0081] Figure 5 An illustrative diagram of the implementation of the result simulation method based on the generative adversarial network according to an embodiment of the present application is shown. Figure 5As shown, it shows the front and back relationship of steps such as GAN-based image output, data integration and model updating, and special subnetwork prediction.

[0082] Through the above embodiments, the accuracy of the influence prediction of the proton beam and the CT image is significantly improved by deep fusion of the generative adversarial network, real-time monitoring technology and personalized prediction model.

[0083] Based on the same inventive concept, the application also provides a result simulation device based on a generative adversarial network, Figure 6 The structure of the result simulation device based on the generative adversarial network in the embodiments of the application is schematically shown. As shown in the figure, Figure 6 The device comprises a data input module for inputting a CT image and a proton beam parameter into a trained generative adversarial network, the generative adversarial network comprising a primary generator, a fine-tuning generator and a double-channel discriminator; a first processing module for obtaining a basic tissue deformation field after encoding and decoding of the CT image by the primary generator; a second processing module for obtaining an enhanced image and an output feature vector after feature fusion and attention weight mapping of the basic tissue deformation field and a Bragg peak distribution map obtained based on the proton beam parameter by the fine-tuning generator; a third processing module for obtaining an image authenticity result after the output feature vector is processed by a spatial feature discrimination branch in the double-channel discriminator, and obtaining a dosimetric evaluation result after the output feature vector is processed by a dosimetric feature discrimination branch in the double-channel discriminator; and a data output module for obtaining an output image of the trained generative adversarial network after discriminating the enhanced image according to the image authenticity result and the dosimetric evaluation result.

[0084] In some optional embodiments, after obtaining the output image of the generative adversarial network, the device further comprises: calculating the difference between the output image and the input CT image; and obtaining the influence of the proton beam based on the proton beam parameter on different types of regions in the input CT image according to the difference.

[0085] In some optional embodiments, the different types of regions include tumor regions; calculating the difference between the output image and the input CT image comprises: segmenting a tumor region from the output image based on an adaptive segmentation algorithm and calculating a first volume of the tumor region; segmenting a tumor region from the input CT image based on the adaptive segmentation algorithm and calculating a second volume of the tumor region; and obtaining a tumor volume difference according to the first volume and the second volume.

[0086] In some optional embodiments, the different types of regions include normal regions; and the calculating the difference between the output image and the input CT image comprises: calculating a Monte Carlo dose according to the nuclear physics engine based on the output image and the proton beam parameters; and obtaining a dose distribution difference between a dose distribution map of the normal regions in the input CT image and a dose distribution map of the normal regions in the output image according to the Monte Carlo dose.

[0087] In some optional embodiments, before inputting the CT image and the proton beam parameters into the trained generative adversarial network, the apparatus further comprises: using a domain adaptation generative adversarial network to establish an image standardization channel, performing gray scale normalization and spatial registration on multi-center CT images while preserving high-frequency texture features.

[0088] In some optional embodiments, the apparatus further comprises: constructing a multi-modal incremental learning framework, integrating follow-up CBCT images, biomarker data and quality of life survey scores, and consolidating the parameters of the trained generative adversarial network through dynamic updating of the algorithm by elastic weights.

[0089] In some optional embodiments, the apparatus further comprises: deploying a real-time dose tracking system for visualizing and displaying dose corresponding risks; the real-time dose tracking system comprises: a deformation registration module for aligning images using a non-rigid registration algorithm; an online reconstruction engine for decomposing the proton beam into a plurality of pencil beams, respectively calculating the dose deposition thereof, and then superimposing the contributions of all pencil beams to generate a three-dimensional dose distribution, and outputting a dose matrix; and a risk visualization interface for generating a color-coded risk map with dose volume constraints based on the dose matrix.

[0090] In some optional embodiments, the apparatus further comprises: designing a special prediction subnetwork for each type of impact result, and obtaining a risk probability of the impact result occurring according to the output of the special prediction subnetwork; the special prediction subnetwork comprises: an input layer, a feature fusion layer and an output layer; the input layer comprises: a dosimetric feature branch, a biomarker branch and an image feature branch; the feature fusion layer is used to perform feature fusion to obtain a fusion result using a cross-attention mechanism; and the output layer is used to output the risk probability of the impact result occurring by activating the fusion result.

[0091] The specific definitions of each functional module in the result simulation device based on the generative adversarial network described above can refer to the definitions of the result simulation method based on the generative adversarial network described above, which will not be repeated here. Each module in the above system can be realized by software, hardware and their combination. Each module described above can be embedded in the processor in the computer device in hardware form or independent of the processor in the computer device, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each module. It also uses a deep fusion generative adversarial network to achieve the effect of improving image prediction accuracy.

[0092] In some embodiments of the present application, an electronic device is also provided, comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the result simulation method based on the generative adversarial network described above. Its internal structure diagram can be as shown in Figure 7 Figure 7 The internal structure diagram of the electronic device according to the embodiments of the present application is schematically shown. The electronic device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected by a system bus. Among them, the processor A01 of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operating system B01 and the computer program B02 in the non-volatile storage medium A04 to run. The network interface A02 of the electronic device is used to communicate with the external terminal through the network connection. The computer program B02 is executed by the processor A01 to implement a result simulation method based on the generative adversarial network.

[0093] Those skilled in the art can understand that Figure 7 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0094] In an embodiment provided by the present application, a machine readable storage medium is provided, and the machine readable storage medium stores instructions which, when executed by a processor, cause the processor to be configured to execute the result simulation method based on the generative adversarial network described above.

[0095] ​In an embodiment provided by the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the foregoing result simulation method based on a generative adversarial network.

[0096] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0097] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block or blocks.

[0098] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block or blocks.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block or blocks.

[0100] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0101] Memory can include non-persistent memory, Random Access Memory (RAM), and / or non-volatile memory, such as Read Only Memory (ROM) or flash memory, in a computer readable medium. Memory is an example of computer readable media.

[0102] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "comprising", "containing", or any other variant thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0104] The above merely provides an example of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A result simulation method based on a generative adversarial network, characterized by, The method comprises: inputting the CT image and the proton beam flow parameter into a trained generative adversarial network, the generative adversarial network comprising a primary generator, a fine-tuning generator and a double-channel discriminator; after the CT image is encoded and decoded by the primary generator, a basic tissue deformation field is obtained; after the basic tissue deformation field and a Bragg peak distribution diagram obtained based on the proton beam flow parameter are subjected to feature fusion and attention weight mapping by the fine-tuning generator, an enhanced image and an output feature vector are obtained; after the output feature vector is subjected to a spatial feature discrimination branch in the double-channel discriminator, an image authenticity result is obtained, and after the output feature vector is subjected to a dosimetric feature discrimination branch in the double-channel discriminator, a dosimetric evaluation result is obtained; after the enhanced image is discriminated according to the image authenticity result and the dosimetric evaluation result, an output image of the trained generative adversarial network is obtained; after the output image of the trained generative adversarial network is obtained, the method further comprises: calculating the difference between the output image and the input CT image; obtaining the influence of the proton beam flow based on the proton beam flow parameter on different types of regions in the input CT image according to the difference; the different types of regions include normal regions; calculating the difference between the output image and the input CT image comprises: based on the output image and the proton beam flow parameter, calculating a Monte Carlo dose according to a nuclear physics engine; obtaining the dose distribution difference between the dose distribution diagram of the normal region in the input CT image and the dose distribution diagram of the normal region in the output image according to the Monte Carlo dose; the method further comprises: deploying a real-time dose tracking system for visualizing and displaying the risk corresponding to the dose; the real-time dose tracking system comprises: a deformation registration module for aligning images using a non-rigid registration algorithm; an online reconstruction engine for decomposing the proton beam into a plurality of pencil beams, calculating the dose deposition of each pencil beam respectively, then superimposing the contributions of all pencil beams to generate a three-dimensional dose distribution, and outputting a dose matrix; and a risk visualization interface for generating a color-coded risk map with dose volume constraints based on the dose matrix.

2. The method of claim 1, wherein, the different types of regions include tumor regions; calculating the difference between the output image and the input CT image comprises: segmenting the tumor region from the output image based on an adaptive segmentation algorithm and calculating a first volume of the tumor region; segmenting the tumor region from the input CT image based on the adaptive segmentation algorithm and calculating a second volume of the tumor region; obtaining a tumor volume difference according to the first volume and the second volume.

3. The method of claim 1, wherein, Before inputting the CT image and the proton beam flow parameter into the trained generative adversarial network, the method further comprises: using a domain adaptation generative adversarial network to establish an image standardization channel to perform gray scale normalization and spatial registration on multi-center CT images while retaining high-frequency texture features.

4. The method of claim 1, wherein, The method further comprises: constructing a multi-modal incremental learning framework to integrate follow-up CBCT images, biomarker data and quality of life survey scores, and consolidating the parameters of the trained generative adversarial network through dynamic updating of the algorithm by using an elastic weight.

5. The method of claim 1, wherein, The method further comprises: designing a special prediction subnetwork for each influence result, and obtaining a risk probability of occurrence of the influence result according to an output of the special prediction subnetwork; The special prediction subnetwork comprises: an input layer, a feature fusion layer, and an output layer; The input layer comprises: a dosimetric feature branch, a biomarker branch, and an image feature branch; The feature fusion layer is configured to perform feature fusion by using a cross-attention mechanism to obtain a fusion result; The output layer is configured to output, through activation, a risk probability of occurrence of an influence result.

6. A result simulation apparatus based on a generative adversarial network, configured to perform the result simulation method based on a generative adversarial network according to any one of claims 1-5, characterized in that, The device comprises: a data input module configured to input a CT image and a proton beam flow parameter into a trained generative adversarial network, the generative adversarial network comprising a primary generator, a fine-tuning generator, and a double-channel discriminator; a first processing module configured to obtain a basic tissue deformation field by encoding and decoding the CT image by the primary generator; a second processing module configured to obtain an enhanced image and an output feature vector by performing feature fusion and attention weight mapping on the basic tissue deformation field and a Bragg peak distribution map obtained based on the proton beam flow parameter by the fine-tuning generator; a third processing module configured to obtain an image authenticity result by inputting the output feature vector into a spatial feature discrimination branch of the double-channel discriminator, and obtain a dosimetric evaluation result by inputting the output feature vector into a dosimetric feature discrimination branch of the double-channel discriminator; a data output module configured to obtain an output image of the trained generative adversarial network by discriminating the enhanced image according to the image authenticity result and the dosimetric evaluation result.

7. An electronic device, comprising: comprises: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the result simulation method based on the generative adversarial network according to any one of claims 1 to 5 by executing the instructions stored in the memory.

Images