Tumor radiotherapy dose prediction method based on generative adversarial network

Abstract

The invention discloses a tumor radiotherapy dose prediction method based on a generative adversarial network. The tumor radiotherapy dose prediction method comprises the following steps: (1) establishing a training data set containing a patient CT image, a planned target region, an organ at risk and real dose distribution; (2) constructing a generative adversarial network based on adaptive weight loss, establishing a generator by adopting a CNN-Transform hybrid structure, introducing an adaptive weight adjustment strategy into a discriminator, and dynamically adjusting loss weights of real data and generated data; (4) training the generative adversarial network by using the training data set; and (5) testing data to be tested by using the trained network, wherein the output of the generator is a dose prediction result. According to the method, the problem of balance of adversarial training of the generator and the discriminator can be effectively solved, the performance of the GAN model is improved, the stability is reduced, and the precision of tumor radiotherapy dose prediction is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and particularly to a method for predicting tumor radiotherapy dose based on a generative adversarial network. Background Art

[0002] Common tumor treatment methods include surgical operation, chemotherapy, and radiotherapy. Surgical treatment is the most important means for patients to achieve long-term survival. However, due to the large surgical trauma and the influence of the patient's physical condition, not all tumor patients are suitable. Especially for some advanced patients, the opportunity for surgery has been lost. Chemotherapy is a treatment method that causes great harm to the body and is not applicable to elderly, pregnant, and patients with poor physical fitness. Radiotherapy uses the method of irradiating rays to kill cancer cells to shrink or disappear tumors, which belongs to non-invasive treatment and has become one of the main means for treating liver cancer.

[0003] In clinical radiotherapy planning, different organs or tissues have specific dose distribution requirements. This includes allocating a sufficiently high and uniform dose distribution on the planning target volume (PTV), while reducing the dose deposition on the organs at risk (OAR). There are often many organs at risk involved in radiotherapy. Accurately planning the irradiation dose for each organ is crucial for improving the treatment effect and quality of life of cancer patients. In clinical practice, in order to achieve an ideal dose prescription, close cooperation between physicists and oncologists is required, and the radiotherapy plan needs to be repeatedly adjusted through a trial-and-error method, which is very cumbersome and time-consuming.

[0004] In recent years, with the rapid development of artificial intelligence and deep learning technologies, some scholars have proposed automatic tumor radiotherapy dose prediction methods based on CNN and Transformer. This type of method usually compensates for the deficiency that CNN cannot capture global information by embedding a Transformer module in CNN. However, a simple embedding strategy, such as directly embedding a Transformer block at certain parts of the skip connection, bottleneck structure, etc. in the CNN network to construct a CNN-Transformer hybrid network, can only capture some specific local and global features of the anatomical features and dose distribution in the image, and it is difficult to achieve deep interaction and fusion of local and global features, thereby resulting in low dose prediction accuracy.

[0005] In addition, some scholars regard the dose prediction task as an image synthesis task and have proposed some dose prediction models based on the Generative Adversarial Network (GAN). However, there are still some limitations that cannot be ignored in the existing GAN-based methods. First, most GAN-based methods use built-in adversarial losses and overall image losses to reduce the difference between the generated dose distribution map and the reference dose distribution map. This may ignore the refinement of the local specific dose distribution and cannot accurately reflect the subtle dosimetric features of the local area. Second, the above GAN-based methods only focus on the design of the generator and do not fully consider the balance problem of the adversarial training between the generator and the discriminator, which may lead to a decrease in the performance and stability of the GAN model. Summary of the Invention

[0006] Aiming at the disadvantages and deficiencies of the existing technology, the present invention improves the generator and discriminator of the generative adversarial network and develops an automatic and accurate tumor radiotherapy dose prediction method. The method designs a cross-sensing fusion module based on CNN and Transformer technologies, and uses this module to construct the generator in GAN, improving the model's ability to extract image features from both global and local perspectives. At the same time, an adaptive weight loss is introduced to refine the dose prediction accuracy in different regions and improve the stability of GAN network training.

[0007] The present invention is achieved through the following solutions:

[0008] (1) Establish a training dataset B including patient CT images, planning target volumes, organs at risk, and true dose distribution maps;

[0009] (2) Construct a generative adversarial network based on adaptive weight loss, denoted as AdpLossGAN, which specifically includes:

[0010] (2-a) Construct a generator network composed of a four-stage encoder-decoder structure, denoted as CPFTransGenerator. The specific structure is as follows: The encoder consists of four stages, and each stage is sequentially connected by an embedding module and a cross-sensing fusion module. The embedding module converts the size of the input feature to a specified size through a learnable embedding layer. That is, for an input feature of size H×W×C, where H, W, and C represent the height, width, and number of channels of the input feature respectively, after passing through the embedding module, its size is converted to The cross-sensing fusion module serves as an encoding block to extract features from the feature map output by the embedding module. The decoder also consists of four stages, and each stage uses a cross-sensing fusion module as a decoding block for decoding. The specific structure includes: In the fourth stage of the decoder, the decoding block uses the output of the encoding block in the same stage as the input for decoding, and the decoded feature The upsampling convolution module with a step size of 2 is used to obtain feature F4, and at the same time

[0011] 1×1 convolution and upsampling with a step size of 8 are performed in sequence to obtain the fourth stage decoding feature D4; in the third stage of the decoder, the gated attention module uses the feature F4 and the output of the encoding block in the same stage as input for feature fusion, and the fused feature is added to F4. The added feature is input to the cross-sensing fusion module for decoding, and the decoded feature is The upsampling convolution module with a step size of 2 is input to obtain feature F3, and at the same time 1×1 convolution and upsampling with a step size of 4 are performed in sequence to obtain the third-stage decoding feature D3; in the second stage of the decoder, the gated attention module uses the feature F3 and the output of the encoding block in the same stage as input for feature fusion, and the fused feature is added to F3. The added feature is input to the cross-sensing fusion module for decoding, and the decoded feature is The upsampling convolution module with a step size of 2 is used to obtain the feature F2, and 1×1 convolution and upsampling with a step size of 2 are performed in sequence to obtain the second-stage decoding feature D2; in the first stage of the decoder, the gated attention module uses the feature F2 and the output of the encoding block in the same stage as input for feature fusion, and the fused feature is added to F2. The added feature is input to the cross-sensing fusion module for decoding, and the decoded feature is Perform 1×1 convolution to obtain the first-stage decoding feature D1; then, add the decoding features D1, D2, D3, and D4 of each stage and perform Sigmoid activation to obtain the dose prediction result;

[0012] (2-a-i) The cross-sensing fusion module described in step (2-a) is composed of N cross-sensing hybrid

[0013] The combined CNN-Transformer structure is denoted as CPF Transformer, which is a cascade structure.

[0014] N is a natural number greater than 0, preferably a natural number between 3 and 8;

[0015] (2-a-ii) The hybrid CNN-Transformer structure based on cross perception described in step (2-a-i)

[0016] The structure is denoted as CPF Transformer. The specific structure includes: constructing two branches, where the first branch is composed of a self-attention residual module and two interaction enhancement modules, and the second branch is composed of a multi-scale perception residual module and two interaction enhancement modules; introducing a cross connection between the two branches. The specific structure is: for the input feature

[0017] Among them, H1, W1, and C1 respectively represent the height, width, and number of channels of the input feature map. First, perform channel separation to obtain two sub-features

[0018] and sub-feature X 0 and X 1 After passing through the self-attention residual module and the multi-scale perception residual module respectively, feature A 0 and feature A 1 are obtained; input feature A 0 and X 1 into the first interaction enhancement module of the first branch to obtain feature F 0 ; input feature A 1 and X 0 into the first interaction enhancement module of the second branch to obtain feature F 1 ; input feature F 0 and A 1 into the second interaction enhancement module of the first branch to obtain feature T 0 ; input feature F 1 and A 0 into the second interaction enhancement module of the second branch to obtain feature T 1 ;

[0019] Finally, splice feature T 0 and feature T 1 to obtain the output of the CPF Transformer; (2-a-ⅲ) The self-attention residual module described in step (2-a-ⅱ) specifically includes: for

[0020] input feature X 0 , successively pass through a normalization layer and a multi-head self-attention structure to obtain feature R 0 , then use residual connection to add feature X 0 and R 0 to obtain feature A 0 ;

[0021] The multi-head self-attention structure preferably uses the CSWin Transformer structure;

[0022] (2-a-ⅳ) The multi-scale perception residual module described in step (2-a-ⅱ) specifically includes: for

[0023] input feature X 1 , successively pass through a normalization layer and a multi-scale perception fusion structure to obtain feature R 1, and then using the residual connection, the feature X 1 and R 1 are added together to obtain the feature

[0024] A 1 ;

[0025] (2-a-v) The multi-scale perception fusion structure described in step (2-a-iv) is denoted as MSPF, and the specific structure

[0026] includes: The input feature X in is respectively input into three linear layers to obtain the features h, w, and p; the feature h is input into the multi-scale hybrid convolution layer to obtain the feature m1, and the feature m1 is multiplied by the feature w to obtain the feature f1; the feature f1 is input into the multi-scale hybrid convolution layer to obtain the feature m2, and the feature m2 is multiplied by the feature w to obtain the feature f2; the feature f2 is input into the multi-scale hybrid convolution to obtain the feature m3, and the feature m3 is multiplied by the feature w to obtain the feature f3; the feature f3 is input into the linear layer to obtain the feature k, and the feature k is multiplied by the feature p to obtain the output

[0027] R 1 of the MSPF module;

[0028] (2-a-vi) The multi-scale hybrid convolution layer described in step (2-a-v), the specific structure includes: The input

[0029] feature is separated by channels, where H2, W2, and C2 respectively represent the height, width, and number of channels of the input feature q, that is, it is evenly divided into four groups according to the number of channels of the input feature to obtain 4 sub-features and The 4 sub-features are respectively input into the depthwise separable convolutions with convolution kernels of 3×3, 5×5, 7×7,

[0030] 9×9, and then the obtained features are concatenated to obtain the output feature q' of the multi-scale hybrid convolution layer;

[0031] (2-a-vii) The interaction enhancement module described in step (2-a-ii) is denoted as IE, and its feature is that the specific structure includes: The IE module accepts two inputs, that is, the input feature from the current branch and the input feature from the other branch; First, the input feature C from the current branch and the input feature O from the other branch are concatenated, and then the concatenated feature is successively input into the normalization layer and the fully connected feedforward network to obtain the feature H, and then through the residual connection,

[0032] the feature H and C are added together to obtain the output feature;

[0033] (2-a-ⅷ) The gated attention module described in step (2-a), denoted as AG, specifically includes the following structure:

[0034] The AG module receives two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, that is, the output of the encoding block in the same stage; v1 and v2 are respectively input into two parallel branches to obtain features q1 and q2, and each branch includes a 1×1 convolution, a normalization layer, and a ReLU activation function connected in sequence; the features q1 and q2 are added together, and the result of the addition is sequentially subjected to a 1×1 convolution, a normalization layer, and a Sigmoid activation to obtain a feature z; z is multiplied by the input feature s1 to obtain the output of the gated attention module;

[0035] (2-b) Construct a discriminator network, which is sequentially composed of an initial convolutional layer, three intermediate convolutional layers, and an output convolutional layer. The specific structure includes: for the input image

[0036] where H3, W3, and C3 respectively represent the height, width, and number of channels of the input feature I. Input into an initial convolutional layer containing a 4x4 convolution with a stride of 2 and a LeakyReLU activation function to obtain a feature The feature t1 is sequentially input into three intermediate convolutional layers to obtain features respectively.

[0037] Each intermediate convolutional layer includes a 4x4 convolution, a normalization layer, and a LeakyReLU activation function. The stride of the 4x4 convolution in the first two intermediate convolutional layers is 2, and the stride of the 4x4 convolution in the third intermediate convolutional layer is 1; input the features

[0038] into an output convolutional layer containing a 4x4 convolution with a stride of 1 and a Sigmoid activation function to obtain a two-dimensional discrimination matrix related to the size of the input image Matrix

[0039] Each element in the matrix J represents the probability that an 8×8 local area in the input image is real data; (2-c) Input the generated dose image G output by the CPFTransGenerator generator network and the real dose

[0040] image R into the discriminator described in step (2-b) respectively to obtain two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the real dose image R; construct a discriminator loss function, and by dynamically adjusting the loss weights of real data and generated data, solve the problem of gradient direction conflict caused by traditional equal-weight summation.

[0041] Sudden problem; specifically, the discriminator loss function is defined as:

[0042]

[0043] Where L1 and L2 represent the generated data loss and the real data loss respectively, N1 and N2 represent the total number of pixels of the discriminant matrices J1 and J2 respectively, J2(p) represents the value of the p-th pixel in the discriminant matrix J2, and J1(p) represents the value of the p-th pixel in the discriminant matrix J1; w1 and w2 are adaptively dynamically adjusted weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the discriminator's judgment on the authenticity of real and generated dose maps. Specifically, first, calculate the included angle between the gradient directions of the generated result loss L1 and the real dose loss L2. If the included angle exceeds 90°, it indicates that the two gradients

[0044] conflict in direction. At this time, the weights are dynamically allocated according to the gradient, specifically:

[0045]

[0046] Where ε is a constant with a relatively small value, used to prevent the denominator from being zero or the weight from being too small. The value range of ε is (0.001, 0.1). This way of weight allocation can make the total gradient direction orthogonal to the conflict term or to avoid reverse optimization interference; if the gradient directions of L1 and L2 are the same, then the weights w1 and w2 are adjusted in combination with the discriminator's judgment on real and generated samples. When the discriminator over-trusts real samples, that is, when the probability that the discriminator determines real samples as true approaches 1, then increase the weight w1 of generated samples to strengthen the learning of generated samples; when the discriminator cannot distinguish generated samples, that is, when the probability that generated samples are determined as true approaches 1, then increase the weight w2 of real samples to strengthen the learning of real samples and improve the discriminative ability of the discriminator; the main reason for this approach is that when the discriminator cannot distinguish generated samples, that is, when the probability that generated samples are determined as true approaches 1, it indicates that the generated samples are highly realistic and the discriminator has lost its effective adversarial ability. If the learning of generated samples continues to be strengthened, the risk of gradient disappearance and mode collapse will be exacerbated. Therefore, at this time, it is necessary to increase the weight of real samples to further improve the discriminative ability of the discriminator;

[0047] (3) According to the discriminator loss, construct the loss function L of the AdpLossGAN network:

[0048]

[0049] Where G(i) and R(i) represent the values of the i-th pixel of the generated dose map G and the real dose map R respectively, N3 is the total number of pixels of the generated dose map G, LG represents the loss obtained by calculating the mean absolute error between the measured dose map G and the true dose map R. represents the adaptive weighted loss calculated by the discriminator;

[0050] (4) Use the training dataset B to train the AdpLossGAN network until the loss function L converges. Among them, the CT images, planning target volumes, and organs at risk in the training dataset B are spliced and used as the network input, and the true dose distribution map is used for loss calculation;

[0051] (5) Use the trained network model to predict the data to be tested. The test data includes patient CT images, planning target volumes, and organs at risk maps, and the output of the CPFTransGenerator is the dose prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is the framework diagram of the generative adversarial network based on adaptive weight loss

[0053] Figure 2 is the structural diagram of the generator network

[0054] Figure 3 is the schematic diagram of the cross-sensing fusion module

[0055] Figure 4 is the schematic diagram of the cross-sensing-based hybrid CNN-Transformer structure

[0056] Figure 5 is the schematic diagram of the multi-scale sensing fusion structure

[0057] Figure 6 is the schematic diagram of the multi-scale hybrid convolutional layer

[0058] Figure 7 is the schematic diagram of the interaction enhancement module

[0059] Figure 8 is the schematic diagram of the gated attention module

[0060] Figure 9 is the structural diagram of the discriminator network

[0061] Figure 10 is an example of the dose prediction result of the embodiment of the present invention. Figure 10 (a)~ Figure 10 (b) are the dose prediction result diagrams obtained by using the method in Embodiment 1. Figure 10 (c)~ Figure 10 (d) are respectively Figure 10 (a)~ Figure 10 (b) corresponding true dose distribution maps DETAILED DESCRIPTION OF THE INVENTION

[0062] Example 1

[0063] A tumor radiotherapy dose prediction method based on a generative adversarial network, and the specific implementation steps are as follows:

[0064] (1) Establish a training dataset B containing patient CT images, planning target volumes, organs at risk, and true dose distribution maps;

[0065] (2) Construct a generative adversarial network based on adaptive weight loss, denoted as AdpLossGAN, as Figure 1 shown, specifically including:

[0066] (2-a) Construct a generator network composed of a four-stage encoder-decoder structure, denoted as CPFTransGenerator, as Figure 2 shown, and the specific structure is: The encoder consists of four stages, and each stage is composed of an embedding module and a cross-sensing fusion module connected in sequence. The embedding module converts the size of the input feature to a specified size through a learnable embedding layer, that is, for an input feature of size H×W×C, where H, W, and C represent the height, width, and number of channels of the input feature respectively. After passing through the embedding module, its size is converted to The cross-sensing fusion module serves as an encoding block to extract features from the feature map output by the embedding module; The decoder also consists of four stages, and each stage uses a cross-sensing fusion module as a decoding block for decoding. The specific structure includes: In the fourth stage of the decoder, the decoding block uses the output of the same-stage encoding block as the input for decoding, and inputs the decoded feature into the upsampling convolutional module with a stride of 2 to obtain feature F4. At the same time, perform 1×1 convolution and upsampling with a stride of 8 in sequence to obtain the fourth-stage decoded feature D4; In the third stage of the decoder, the gated attention module takes feature F4 and the output of the same-stage encoding block as inputs for feature fusion. The fused feature is added to F4, and the added feature is input into the cross-sensing fusion module for decoding. The decoded feature is input into the upsampling convolutional module with a stride of 2 to obtain feature F3. At the same time, perform 1×1 convolution and upsampling with a stride of 4 in sequence to obtain the third-stage decoded feature D3; In the second stage of the decoder, the gated attention module takes feature F3 and the output of the same-stage encoding block as inputs for feature fusion. The fused feature is added to F3, and the added feature is input into the cross-sensing fusion module for decoding. The decoded feature is input into the upsampling convolutional module with a stride of 2 to obtain feature F2. At the same time, Perform a 1×1 convolution and an upsampling with a stride of 2 in sequence to obtain the decoding feature D2 of the second stage; in the first stage of the decoder, the gated attention module takes the feature F2 and the output of the encoding block in the same stage as inputs for feature fusion. The fused feature is added to F2, and the added feature is input into the cross-perception fusion module for decoding. The decoded feature is subjected to a 1×1 convolution to obtain the decoding feature D1 of the first stage; then, the decoding features D1, D2, D3, and D4 of each stage are added and activated by Sigmoid to obtain the dose prediction result;

[0067] (2-a-ⅰ) The cross-perception fusion module described in step (2-a) is composed of N cross-perception-based mixed

[0068] CNN-Transformer structures, denoted as CPF Transformer, cascaded, as Figure 3 shown, where N is a natural number greater than 0, preferably a natural number between 3 and 8. In this embodiment

[0069] preferably N = 4;

[0070] (2-a-ⅱ) The cross-perception-based mixed CNN-Transformer structure described in step (2-a-ⅰ), denoted as CPF Transformer, as

[0071] shown, specifically includes: constructing two branches, where the first branch is composed of a self-attention residual module and two interactive enhancement modules connected, and the second branch is composed of a multi-scale perception residual module and two interactive enhancement modules connected; introducing a cross-connection between the two branches. The specific structure is that for the input feature Figure 4 where H1, W1, and C1 respectively represent the height, width, and number of channels of the input feature map. First, perform channel separation on to obtain two sub-features

[0072] and The sub-feature X 0 and X 1 respectively pass through the self-attention residual module and the multi-scale perception residual module to obtain the feature A 0 and the feature A 1 ; input the feature A 0 and X 1 into the first interactive enhancement module of the first branch to obtain the feature F 0 ; input the feature A 1 and X 0 into the first interactive enhancement module of the second branch to obtain the feature F 1 ; input the feature F0 and A 1 Input the second interactive enhancement module of the first branch to obtain feature T 0 ; Input feature F 1 and A 0 Input the second interactive enhancement module of the second branch to obtain feature T 1 ;

[0073] Finally, concatenate feature T 0 and feature T 1 to obtain the output of the CPF Transformer; (2-a-iii) The self-attention residual module described in step (2-a-ii) specifically includes: For

[0074] Input feature X 0 , sequentially pass through a normalization layer and a multi-head self-attention structure to obtain feature R 0 , then use residual connection to add feature X 0 and R 0 to obtain feature A 0 ;

[0075] The multi-head self-attention structure in this embodiment preferably uses the CSWin Transformer structure;

[0076] (2-a-iv) The multi-scale perception residual module described in step (2-a-ii) specifically includes: For

[0077] Input feature X 1 , sequentially pass through a normalization layer and a multi-scale perception fusion structure to obtain feature R 1 , then use residual connection to add feature X 1 and R 1 to obtain feature

[0078] A 1 ;

[0079] (2-a-v) The multi-scale perception fusion structure described in step (2-a-iv), denoted as MSPF, as Figure 5

[0080] shown, specifically includes: Input feature X inThree linear layers are input separately to obtain features h, w, and p; the feature h is input into the multi-scale hybrid convolution layer to obtain feature m1, and feature m1 is multiplied by feature w to obtain feature f1; feature f1 is input into the multi-scale hybrid convolution layer to obtain feature m2, and feature m2 is multiplied by feature w to obtain feature f2; feature f2 is input into the multi-scale hybrid convolution to obtain feature m3, and feature m3 is multiplied by feature w to obtain feature f3; feature f3 is input into the linear layer to obtain feature k, and feature k is multiplied by feature p to obtain

[0081] the output R of the MSPF module 1 ;

[0082] (2-a-vi) The multi-scale hybrid convolution layer described in step (2-a-v) is as Figure 6 shown, and the specific structure

[0083] includes: separating the input feature where H2, W2, and C2 respectively represent the height, width, and number of channels of the input feature q, that is, the input feature is evenly divided into four groups according to the number of channels of the input feature to obtain 4 sub-features

[0084] and inputting the 4 sub-features into depthwise separable convolutions with convolution kernels of 3×3, 5×5, 7×7, and 9×9 respectively, and then splicing the obtained features to obtain

[0085] the output feature q' of the multi-scale hybrid convolution layer;

[0086] (2-a-vii) The interaction enhancement module described in step (2-a-ii), denoted as IE, is as Figure 7 shown, and its feature is that the specific structure includes: the IE module accepts two inputs, that is, the input feature from the current branch, such as Figure 4 and 7 the input feature shown by the solid line in, and the input feature from another branch, such as Figure 4 and 7 the input feature shown by the dashed line in; first, the input feature C from the current branch and the input feature O from another branch are spliced, and then the spliced feature is sequentially input into the normalization layer and the fully connected feedforward network to obtain feature H, and then through the residual connection, feature H and C are added to obtain the output feature;

[0087] (2-a-viii) The gated attention module described in step (2-a), denoted as AG, is as Figure 8 shown, and the specific

[0088] The body structure includes: The AG module receives two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, that is, the output of the encoding block in the same stage; v1 and v2 are respectively input into two parallel branches to obtain features q1 and q2, and each branch contains a 1×1 convolution, a normalization layer, and a ReLU activation function connected in sequence; the features q1 and q2 are added together, and the result of the addition is successively subjected to a 1×1 convolution, a normalization layer, and a Sigmoid activation to obtain the feature z; the feature z is input into the feature s1 for

[0089] dot multiplication to obtain the output of the gated attention module;

[0090] (2-b) Construct a discriminator network, as Figure 9 shown, this network is composed of an initial convolutional layer, three intermediate convolutional layers, and an output convolutional layer connected in sequence. The specific structure includes: For the input image

[0091] where H3, W3, and C3 respectively represent the height, width, and number of channels of the input feature I. Input

[0092] into an initial convolutional layer containing a 4x4 convolution with a stride of 2 and a LeakyReLU activation function to obtain the feature Input the feature t1 into three intermediate convolutional layers in sequence to obtain the features Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer, and a LeakyReLU activation function. The stride of the 4x4 convolution in the first two intermediate convolutional layers is 2, and the stride of the 4x4 convolution in the third intermediate convolutional layer is 1; input the feature into an output convolutional layer containing a 4x4 convolution with a stride of 1 and a Sigmoid activation function to obtain a two-dimensional discrimination matrix related to the size of the input image Each element in the matrix J represents the probability that an 8×8 local area in the input image is real

[0093] data;

[0094]

[0095] (2-c) Input the generated dose image G output by the CPFTransGenerator generator network and the real dose image

[0096] R into the discriminator described in step (2-b) respectively to obtain two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the real dose image R; construct a discriminator loss function, and by dynamically adjusting the loss weights of the real data and the generated data, solve the problem of gradient direction conflict caused by traditional equal-weight summation ​

[0097] Sudden problem; specifically, the discriminator loss function is defined as:

[0098]

[0099] where L1 and L2 respectively represent the generated data loss and the real data loss, N1 and N2 respectively represent the total number of pixels of the discriminant matrices J1 and J2, J2(p) represents the value of the p-th pixel in the discriminant matrix J2, and J1(p) represents the value of the p-th pixel in the discriminant matrix J1; w1 and w2 are adaptively and dynamically adjusted weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the discriminator's judgment on the authenticity of real and generated dose maps. Specifically, first, calculate the included angle between the gradient directions of the generated result loss L1 and the real dose loss L2. If the included angle exceeds 90°, it indicates that the two gradients

[0100] are in conflict. At this time, the weights are dynamically allocated according to the gradients, specifically:

[0101]

[0102] where ε is a constant with a relatively small value, used to prevent the denominator from being zero or the weight from being too small. The value range of ε is (0.001, 0.1). In this embodiment, ε = 0.01 is preferably used. This way of weight allocation can make the total gradient direction orthogonal to the conflict term or to avoid reverse optimization interference; if the gradient directions of L1 and L2 are the same, then the weights w1 and w2 are adjusted in combination with the discriminator's judgment on real and generated samples. When the discriminator over-trusts real samples, that is, when the probability that the discriminator determines real samples as true approaches 1, then increase the weight w1 of the generated samples to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, when the probability that the generated samples are determined as true approaches 1, then increase the weight w2 of the real samples to strengthen the learning of the real samples and improve the discriminative ability of the discriminator; the main reason for this approach is that when the discriminator cannot distinguish the generated samples, that is, when the probability that the generated samples are determined as true approaches 1, it indicates that the generated samples are highly realistic and the discriminator has lost its effective adversarial ability. If the learning of the generated samples continues to be strengthened, the risk of gradient disappearance and mode collapse will be exacerbated. Therefore, at this time, it is necessary to increase the weight of the real samples to further improve the discriminative ability of the discriminator;

[0103] (3) According to the discriminator loss, construct the loss function L of the AdpLossGAN network:

[0104]

[0105] Among them, G(i) and R(i) respectively represent the values of the i-th pixel of the generated dose map G and the true dose map R, N3 is the total number of pixels of the generated dose map G, and L G represents the loss obtained by calculating the mean absolute error between the measured dose map G and the true dose map R, and represents the adaptive weighted loss calculated by the discriminator;

[0106] (4) Use the training dataset B to train the AdpLossGAN network until the loss function L converges. Among them, the CT images, planning target volumes, and organs at risk in the training dataset B are spliced and used as the network input, and the true dose distribution map is used for loss calculation;

[0107] (5) Use the trained network model to predict the data to be tested. Among them, the test data includes patient CT images, planning target volumes, and organs at risk maps, and the output of the CPFTransGenerator is the dose prediction result.

[0108] Example 2

[0109] The method in Example 1 was used to conduct experiments on a clinical liver cancer dataset from a certain tertiary hospital. This dataset contains data of 60 liver cancer patients. Each patient's data includes patient CT images, planning target volume maps, organs at risk maps, and clinical dose distribution maps. The size of the CT images is 512×512×Q, where 112 ≤ Q ≤ 238. In the experiment, the data was divided into a test dataset and a training dataset according to a ratio of 1:4.

[0110] Figure 10 Shows some examples of dose prediction results obtained on the test set. Figure 10 (a)~ Figure 10 (b) are the dose prediction result diagrams obtained by using the method in Example 1, Figure 10 (c)~ Figure 10 (d) are respectively Figure 10 (a)~ Figure 10 (b) corresponding true dose distribution maps. It can be seen that the dose prediction results obtained by the method of the present invention can better approximate the true dose distribution.

[0111] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A tumor radiotherapy dose prediction method based on a generative adversarial network, characterized in that Including the following steps: (1) Establish a training dataset B including patient CT images, planning target volumes, organs at risk, and true dose distribution maps; (2) Construct a generative adversarial network based on adaptive weighted loss, denoted as AdpLossGAN, specifically including: (2-a) Construct a generator network consisting of a four-stage encoder-decoder structure, denoted as CPFTransGenerator. The specific structure is as follows: The encoder consists of four stages, and each stage is sequentially composed of an embedding module and a cross-sensing fusion module. The embedding module converts the size of the input features to a specified size through a learnable embedding layer. That is, for input features of size H×W×C, where H, W, and C represent the height, width, and number of channels of the input features respectively, after passing through the embedding module, its size is converted to The cross-sensing fusion module, as an encoding block, extracts features from the feature map output by the embedding module; The decoder also consists of four stages, and each stage uses a cross-sensing fusion module as a decoding block for decoding. The specific structure includes: In the fourth stage of the decoder, the decoding block uses the output of the encoding block in the same stage as the input for decoding, and the decoded features are input into the upsampling convolutional module with a stride of 2 to obtain feature F4. At the same time, perform 1×1 convolution and upsampling with a stride of 8 in sequence to obtain the decoding feature D4 of the fourth stage; In the third stage of the decoder, the gated attention module takes feature F4 and the output of the encoding block in the same stage as the input for feature fusion. The fused feature is added to F4, and the added feature is input into the cross-sensing fusion module for decoding. The decoded features are input into the upsampling convolutional module with a stride of 2 to obtain feature F3. At the same time, perform 1×1 convolution and upsampling with a stride of 4 in sequence to obtain the decoding feature D3 of the third stage; In the second stage of the decoder, the gated attention module takes feature F3 and the output of the encoding block in the same stage as the input for feature fusion. The fused feature is added to F3, and the added feature is input into the cross-sensing fusion module for decoding. The decoded features are input into the upsampling convolutional module with a stride of 2 to obtain feature F2. At the same time, perform 1×1 convolution and upsampling with a stride of 2 in sequence to obtain the decoding feature D2 of the second stage; In the first stage of the decoder, the gated attention module takes feature F2 and the output of the encoding block in the same stage as the input for feature fusion. The fused feature is added to F2, and the added feature is input into the cross-sensing fusion module for decoding. The decoded features perform 1×1 convolution to obtain the decoding feature D1 of the first stage; Then, add and perform Sigmoid activation on the decoding features D1, D2, D3, and D4 of each stage to obtain the dose prediction result; (2-a-ⅰ) The cross-perception fusion module described in step (2-a) is composed of N cross-perception-based hybrid CNN-Transformer structures, denoted as CPF Transformer, cascaded, where N is a natural number greater than 0, preferably a natural number between 3 and 8; (2-a-ⅱ) The cross-perception-based hybrid CNN-Transformer structure described in step (2-a-ⅰ) is denoted as CPFTransformer. The specific structure includes: constructing two branches. The first branch is composed of a self-attention residual module and two interaction enhancement modules connected in sequence. The second branch is composed of a multi-scale perception residual module and two interaction enhancement modules connected in sequence. A cross-connection is introduced between the two branches. The specific structure is as follows: for the input feature where H1, W1, and C1 represent the height, width, and number of channels of the input feature map respectively. First, is subjected to channel separation to obtain two sub-features and The sub-feature X 0 and X 1 after passing through the self-attention residual module and the multi-scale perception residual module respectively, obtain the feature A 0 and the feature A 1 ; The feature A 0 and X 1 are input into the first interaction enhancement module of the first branch to obtain the feature F 0 ; The feature A 1 and X 0 are input into the first interaction enhancement module of the second branch to obtain the feature F 1 ; The feature F 0 and A 1 are input into the second interaction enhancement module of the first branch to obtain the feature T 0 ; The feature F 1 and A 0 are input into the second interaction enhancement module of the second branch to obtain the feature T 1 ; Finally, the feature T 0 and the feature T 1 are concatenated to obtain the output of CPFTransformer; (2-a-ⅲ) The self-attention residual module described in step (2-a-ⅱ) specifically includes: for the input feature X 0 , it sequentially passes through a normalization layer and a multi-head self-attention structure to obtain the feature R 0 . Then, using residual connection, the feature X 0 and R 0 are added together to obtain the feature A 0 ; (2-a-iv) The multi-scale perception residual module described in step (2-a-ii) specifically includes: for the input feature X 1 , it sequentially passes through a normalization layer and a multi-scale perception fusion structure to obtain the feature R 1 . Then, using the residual connection, the feature X 1 and R 1 are added together to obtain the feature A 1 ; (2-a-v) The multi-scale perception fusion structure described in step (2-a-iv) is denoted as MSPF, and its specific structure includes: inputting the input feature X in into three linear layers respectively to obtain features h, w, and p; inputting the feature h into the multi-scale hybrid convolution layer to obtain the feature m1, and multiplying the feature m1 by the feature w to obtain the feature f1; inputting the feature f1 into the multi-scale hybrid convolution layer to obtain the feature m2, and multiplying the feature m2 by the feature w to obtain the feature f2; inputting the feature f2 into the multi-scale hybrid convolution to obtain the feature m3, and multiplying the feature m3 by the feature w to obtain the feature f3; inputting the feature f3 into the linear layer to obtain the feature k, and multiplying the feature k by the feature p to obtain the output R of the MSPF module 1 ; (2-a-vi) The multi-scale hybrid convolution layer described in step (2-a-v) specifically includes: separating the input features channel-wise, where H2, W2, and C2 represent the height, width, and number of channels of the input feature q respectively, that is, the input feature is evenly divided into four groups according to the number of channels of the input feature to obtain 4 sub-features and The 4 sub-features are respectively input into depthwise separable convolutions with convolution kernels of 3×3, 5×5, 7×7, and 9×9, and then the obtained features are concatenated to obtain the output feature q' of the multi-scale hybrid convolution layer; (2-a-ⅶ) The interaction enhancement module described in step (2-a-ⅱ), denoted as IE, is characterized in that the specific structure includes: The IE module accepts two inputs, namely the input feature from the current branch and the input feature from the other branch; First, the input feature C from the current branch and the input feature O from the other branch are concatenated, and then the concatenated feature is sequentially input into the normalization layer and the fully connected feed-forward network to obtain the feature H, and then through the residual connection, the feature H and C are added to obtain the output feature; (2-b) Construct a discriminator network, which is composed of an initial convolutional layer, three intermediate convolutional layers and an output convolutional layer connected in sequence. The specific structure includes: for the input image where H3, W3, and C3 respectively represent the height, width, and number of channels of the input feature I, and input it into an initial convolutional layer containing a 4x4 convolution with a stride of 2 and a LeakyReLU activation function to obtain a feature Input the feature t1 into three intermediate convolutional layers in sequence to obtain features Each intermediate convolutional layer includes a 4x4 convolution, a normalization layer, and a LeakyReLU activation function. The stride of the 4x4 convolution in the first two intermediate convolutional layers is 2, and the stride of the 4x4 convolution in the third intermediate convolutional layer is 1; input the feature into an output convolutional layer containing a 4x4 convolution with a stride of 1 and a Sigmoid activation function to obtain a two-dimensional discrimination matrix related to the size of the input image (2-c) Input the generated dose image G output by the CPFTransGenerator generator network and the true dose image R into the discriminator described in step (2-b) respectively to obtain the two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the true dose image R; Construct a discriminator loss function to solve the gradient direction conflict problem caused by traditional equal-weight summation by dynamically adjusting the loss weights of real data and generated data; Specifically, the discriminator loss function is defined as: where L1 and L2 respectively represent the generated data loss and the real data loss, N1 and N2 respectively represent the total number of pixels in the discrimination matrices J1 and J2, J2(p) represents the value of the p-th pixel in the discrimination matrix J2, and J1(p) represents the value of the p-th pixel in the discrimination matrix J1; w1 and w2 are adaptively and dynamically adjusted weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgments of the discriminator on the real and generated dose maps, specifically including: First, calculate the included angle between the gradient directions of the generated result loss L1 and the true dose loss L2. If the included angle exceeds 90°, it indicates that the two gradient directions conflict. At this time, the weights are dynamically allocated according to the gradient, specifically: Among them, ε is a constant with a relatively small value, which is used to prevent the denominator from being zero or the weight from being too small. The value range of ε is (0.001, 0.1). This way of weight allocation can make the total gradient direction orthogonal to the conflict terms ▽L1 or ▽L2 to avoid reverse optimization interference; if the gradient directions of L1 and L2 are the same, then the discriminator is combined to adjust the weights w1 and w2 according to the judgment of the real and generated samples. When the discriminator over-trusts the real samples, that is, when the probability that the discriminator determines the real samples as true approaches 1, the weight w1 of the generated samples is increased to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, when the probability that the generated samples are determined as true approaches 1, the weight w2 of the real samples is increased to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; (3) Construct the loss function L of the AdpLossGAN network according to the discriminator loss; wherein, G(i) and R(i) respectively represent the values of the i-th pixel of the generated dose map G and the true dose map R, N3 is the total number of pixels of the generated dose map G, and L G represents the loss obtained by calculating the mean absolute error between the measured dose map G and the true dose map R, represents the adaptive weighted loss calculated by the discriminator; (4) Use the training dataset B to train the AdpLossGAN network until the loss function L converges, where the CT images, planning target volumes, and organs at risk in the training dataset B are concatenated as the network input, and the true dose distribution map is used for loss calculation; (5) Use the trained network model to predict the data to be tested, where the test data includes patient CT images, planning target volumes, and organs at risk maps, and the output of the CPFTransGenerator generator is the dose prediction result.

2. The tumor radiotherapy dose prediction method based on a generative adversarial network according to claim 1, characterized in that The gated attention module described in step (2-a) is denoted as AG, and its specific structure includes: The AG module receives two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, that is, the output of the encoding block in the same stage; v1 and v2 are respectively input into two parallel branches to obtain features q1 and q2, and each branch contains a 1×1 convolution, a normalization layer, and a ReLU activation function connected in sequence; the features q1 and q2 are added together, and the result of the addition is sequentially subjected to a 1×1 convolution, a normalization layer, and a Sigmoid activation to obtain a feature z; z is multiplied by the input feature s1 to obtain the output of the gated attention module.

3. The tumor radiotherapy dose prediction method based on a generative adversarial network according to claim 1, wherein, The multi-head self-attention structure described in step (2-a-ⅲ) preferably adopts the CSWin Transformer structure.

Images