A tumor radiotherapy dose prediction method based on a generative adversarial network

Abstract

The application discloses a tumor radiotherapy dose prediction method based on a generative adversarial network, comprising the following steps: (1) establishing a training data set comprising a patient CT image, a planning target volume, an organ at risk and a real dose distribution; (2) constructing a generative adversarial network based on an adaptive weight loss, wherein a CNN-Transformer hybrid structure is adopted to establish a generator, and an adaptive weight adjustment strategy is introduced into a discriminator to dynamically adjust the loss weights of real data and generated data; (4) training the generative adversarial network by using the training data set; (5) testing the data to be tested by using the trained network, and the output of the generator is the dose prediction result. The application can effectively solve the balance problem of the generator and the discriminator in the adversarial training, improve the performance and stability of the GAN model, and improve the precision of the tumor radiotherapy dose prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, and in particular to a tumor radiotherapy dose prediction method based on a generative adversarial network. BACKGROUND

[0002] Common tumor treatment methods include surgery, chemotherapy and radiotherapy. Surgery is the most important means for patients to achieve long-term survival, but due to the large surgical trauma, it is not suitable for all patients. Especially for some advanced patients, there is no surgical opportunity. Chemotherapy is a treatment method that causes great harm to the body, and is not suitable for elderly, pregnant, and poor patients. Radiotherapy uses radiation to kill cancer cells to reduce or eliminate tumors, and is a non-traumatic treatment, which has become one of the main means for treating liver cancer.

[0003] In clinical radiotherapy planning, there are specific dose distribution requirements for different organs or tissues. This includes allocating a sufficiently high and uniform dose distribution on the planning target volume (PTV) while reducing dose deposition on organs at risk (OAR). There are often many organs at risk involved in radiotherapy, and accurate planning of the irradiation dose for each organ is crucial to improving the treatment effect and quality of life of cancer patients. In clinical practice, in order to achieve an ideal dose prescription, close cooperation between physicists and oncologists is required, and the radiotherapy plan is repeatedly adjusted through trial and error, which is a very tedious and time-consuming process.

[0004] In recent years, with the rapid development of artificial intelligence and deep learning technology, some scholars have proposed automatic tumor radiotherapy dose prediction methods based on CNN and Transformer. This type of method usually embeds a Transformer module in a CNN to compensate for the inability of CNN to capture global information. However, simple embedding strategies, such as directly embedding a Transformer block at certain positions such as the skip connection, bottleneck structure, etc. of the CNN network to construct a CNN-Transformer hybrid network, can only capture certain specific local and global features of anatomical features and dose distribution in the image, and it is difficult to achieve deep interaction and fusion of local and global features, which in turn leads to low dose prediction accuracy.

[0005] In addition, some scholars also regard the dose prediction task as an image synthesis task and propose some dose prediction models based on generative adversarial network (GAN). However, the existing GAN-based methods still have some limitations that cannot be ignored. First, most of the GAN-based methods use built-in adversarial loss and image overall loss to reduce the difference between the generated dose distribution map and the reference dose distribution map, which may ignore the refinement of the local specific dose distribution and cannot accurately reflect the subtle dosimetric characteristics of the local region. Second, the above GAN-based methods only focus on the design of the generator and do not fully consider the balance problem of the generator and the discriminator in the adversarial training, which may reduce the performance and stability of the GAN model. SUMMARY

[0006] In view of the shortcomings and deficiencies of the prior art, the present application develops an automatic and accurate tumor radiotherapy dose prediction method by improving the generator and discriminator of the generative adversarial network. The method uses CNN and Transformer technology to design a cross perception fusion module based on the module, and uses the module to construct the generator in the GAN, improves the ability of the model to extract image features from global and local perspectives, and introduces adaptive weight loss to refine the dose prediction accuracy in different regions, and at the same time improves the stability of the GAN network training.

[0007] The present application is realized by the following scheme:

[0008] (1) establishing a training data set B containing patient CT images, planning target volume, organs at risk and real dose distribution map;

[0009] (2) constructing a generative adversarial network based on adaptive weight loss, denoted as AdpLossGAN, which specifically includes:

[0010] (2-a) constructing a generator network composed of four-stage encoding-decoding structure, denoted as CPFTransGenerator, the specific structure is: the encoder includes four stages, each stage is composed of an embedding module and a cross perception fusion module connected in turn, the embedding module converts the size of the input feature to a specified size through a learnable embedding layer, that is, for the input feature with a size of HxWxC, wherein H, W and C represent the height, width and channel number of the input feature respectively, after the embedding module, the size is converted to The cross perception fusion module is used as an encoding block to extract features from the feature map output by the embedding module; the decoder also includes four stages, each stage uses a cross perception fusion module as a decoding block for decoding, and the specific structure includes: in the fourth stage of the decoder, the decoding block uses the output of the encoding block in the same stage as the input for decoding, and the decoded feature The input upsampling convolutional module with a stride of 2 is used to obtain feature F4, while simultaneously...

[0011] A 1×1 convolution and an upsampling step of 8 are performed sequentially to obtain the fourth-stage decoded feature D4. In the third stage of the decoder, the gated attention module uses feature F4 and the output of the same-stage coding block as input for feature fusion. The fused feature is added to F4, and the added feature is input to the cross-perception fusion module for decoding. The decoded feature is then processed. The input upsampling convolutional module with a stride of 2 is used to obtain feature F3, while simultaneously... A 1×1 convolution and upsampling with a stride of 4 are performed sequentially to obtain the third-stage decoded feature D3. In the second stage of the decoder, the gated attention module uses feature F3 and the output of the same-stage coding block as input for feature fusion. The fused feature is added to F3, and the added feature is input to the cross-perception fusion module for decoding. The decoded feature is then processed. The input upsampling convolutional module with a stride of 2 is used to obtain feature F2, while simultaneously... A 1×1 convolution and upsampling with a stride of 2 are performed sequentially to obtain the second-stage decoded feature D2. In the first stage of the decoder, the gated attention module uses feature F2 and the output of the same-stage coding block as input for feature fusion. The fused feature is added to F2, and the added feature is input to the cross-perception fusion module for decoding. The decoded feature is then processed. A 1×1 convolution is performed to obtain the first-stage decoding feature D1; then, the decoding features D1, D2, D3, and D4 of each stage are added together and activated by Sigmoid to obtain the dose prediction result.

[0012] (2-a-i) The cross-sensing fusion module described in step (2-a) consists of N cross-sensing-based fusion modules.

[0013] Combined with the CNN-Transformer structure, denoted as CPF Transformer, it is constructed by cascading, where

[0014] N is a natural number greater than 0, preferably a natural number between 3 and 8;

[0015] (2-a-ii) The hybrid CNN-Transformer structure based on cross-sensing described in step (2-a-i)

[0016] The structure, denoted as CPF Transformer, includes: constructing two branches, where the first branch consists of a self-attention residual module and two interaction enhancement modules connected together, and the second branch consists of a multi-scale perception residual module and two interaction enhancement modules connected together; introducing cross-connections between the two branches, specifically, for input features...

[0017] Where H1, W1, and C1 represent the height, width, and number of channels of the input feature map, respectively. First, for... Channel separation is performed to obtain two sub-features.

[0018] and Sub-feature X 0 and X 1 After passing through the self-attention residual module and the multi-scale perception residual module respectively, feature A is obtained. 0 and feature A 1 Feature A 0 and X 1 Input the first interaction enhancement module of the first branch to obtain feature F 0 Feature A 1 and X 0 Input the first interaction enhancement module of the second branch to obtain feature F 1 ; feature F 0 and A 1 Input the second interaction enhancement module of the first branch to obtain feature T. 0 ; feature F 1 and A 0 Input the second interaction enhancement module of the second branch to obtain feature T. 1 ;

[0019] Finally, feature T 0 and feature T 1 The output of the CPF Transformer is obtained by concatenation; (2-a-iii) The self-attention residual module described in step (2-a-ii) has the following specific structure: For

[0020] Input feature X 0 The feature R is obtained by passing through a normalization layer and a multi-head self-attention structure in sequence. 0 Then, using residual connections, the feature X is... 0 and R 0 Add them together to obtain feature A. 0 ;

[0021] The preferred multi-head self-attention structure is the CSWin Transformer structure.

[0022] (2-a-ⅳ) The multi-scale sensing residual module described in step (2-a-ⅱ) has the following specific structure: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.]

[0023] Input feature X 1 The feature R is obtained by passing through a normalization layer and a multi-scale perceptual fusion structure in sequence. 1Then, using residual connections, the feature X is... 1 and R 1 Add them together to obtain the features.

[0024] A 1 ;

[0025] (2-a-v) The multi-scale perceptual fusion structure described in step (2-a-v) is denoted as MSPF, and its specific structure is as follows:

[0026] The structure includes: input features X in The inputs are fed into three linear layers to obtain features h, w, and p. Feature h is then fed into a multi-scale hybrid convolutional layer to obtain feature m1, and feature m1 is multiplied by feature w to obtain feature f1. Feature f1 is fed into a multi-scale hybrid convolutional layer to obtain feature m2, and feature m2 is multiplied by feature w to obtain feature f2. Feature f2 is fed into a multi-scale hybrid convolutional layer to obtain feature m3, and feature m3 is multiplied by feature w to obtain feature f3. Feature f3 is fed into a linear layer to obtain feature k, and feature k is multiplied by feature p to obtain the output of the MSPF module.

[0027] Out of R 1 ;

[0028] (2-a-v) The multi-scale hybrid convolutional layer described in step (2-a-v) specifically includes the following structure: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.]

[0029] Input characteristics Channel separation is performed, where H2, W2, and C2 represent the height, width, and number of channels of the input feature q, respectively. That is, the input feature is divided into four groups based on the number of channels, resulting in four sub-features. and The four sub-features are input into convolutional kernels of 3×3, 5×5, and 7×7, respectively.

[0030] A 9×9 depthwise separable convolution is performed, and the resulting features are then concatenated to obtain the output feature q′ of the multi-scale hybrid convolutional layer.

[0031] (2-a-vii) The interaction enhancement module described in step (2-a-ii), denoted as IE, is characterized by the following structure: the IE module accepts two inputs, namely, input features from the current branch and input features from another branch; firstly, the input features C from the current branch and the input features O from the other branch are concatenated, and then the concatenated features are sequentially input into a normalization layer and a fully connected feedforward network to obtain features H, which are then processed through residual connections.

[0032] The features H and C are added together to obtain the output features;

[0033] (2-a-ⅷ) The gating attention module described in step (2-a) is denoted as AG, and the specific structure includes:

[0034] The AG module accepts two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, i.e. the output of the same stage encoding block; v1 and v2 are input into two parallel branches respectively to obtain features q1 and q2, each branch contains a 1x1 convolution, a normalization layer and a ReLU activation function connected in turn; the features q1 and q2 are added, and the added result is sequentially executed 1x1 convolution, normalization layer and Sigmoid activation to obtain feature z; z is input into feature s1 for point multiplication to obtain the output of the gating attention module;

[0035] (2-b) Construct a discriminator network, which is composed of an initial convolutional layer, three intermediate convolutional layers and an output convolutional layer connected in turn, and the specific structure includes:

[0036] The input image is input into an initial convolutional layer containing a 4x4 convolution with a step of 2 and a LeakyReLU activation function to obtain feature t1. Where H3, W3, C3 represent the height, width and channel number of the input feature I, respectively. An initial convolutional layer containing a 4x4 convolution with a step of 2 and a LeakyReLU activation function is input to obtain feature t1. Feature t1 is sequentially input into three intermediate convolutional layers to obtain features t2, t3 and t4 respectively.

[0037] Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer and a LeakyReLU activation function, where the first two intermediate convolutional layers have a 4x4 convolution with a step of 2, and the third intermediate convolutional layer has a 4x4 convolution with a step of 1; features t2, t3 and t4 are sequentially input into an output convolutional layer containing a 4x4 convolution with a step of 1 and a Sigmoid activation function to obtain a two-dimensional discrimination matrix J related to the size of the input image.

[0038] An output convolutional layer containing a 4x4 convolution with a step of 1 and a Sigmoid activation function is input to obtain a two-dimensional discrimination matrix J related to the size of the input image. Each element in matrix J represents the probability that an 8x8 local region in the input image is real data.

[0039] (2-c) The generated dose image G output by the CPFTransGenerator generator network and the real dose image R are input into the discriminator described in step (2-b) to obtain the two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the real dose image R; a discriminator loss function is constructed, and by dynamically adjusting the loss weights of real data and generated data, the gradient direction conflict caused by traditional equal weight summation is solved.

[0040] (2-c) The generated dose image G output by the CPFTransGenerator generator network and the real dose image R are input into the discriminator described in step (2-b) to obtain the two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the real dose image R; a discriminator loss function is constructed, and by dynamically adjusting the loss weights of real data and generated data, the gradient direction conflict caused by traditional equal weight summation is solved.

[0041] The problem is that the discriminator loss function is defined as:

[0042]

[0043] wherein L1 and L2 represent the generated data loss and the real data loss, respectively, N1 and N2 represent the total number of pixels in the discrimination matrices J1 and J2, respectively, J2(p) represents the value of the pth pixel in the discrimination matrix J2, J1(p) represents the value of the pth pixel in the discrimination matrix J1; w1 and w2 are adaptive dynamic adjustment weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgments of the discriminator on the real and generated dose maps, specifically including: first, calculating the included angle between the gradient directions of the generated result loss L1 and the real dose loss L2, if the included angle exceeds 90°, it indicates that the two gradients

[0044] conflict, at this time, the weight is dynamically allocated according to the gradient, specifically:

[0045]

[0046] wherein ε is a small constant value, used to prevent the denominator from being zero or the weight being too small, and the value range of ε is (0.001, 0.1), this weight allocation method can make the total gradient direction orthogonal to the conflicting item or avoiding reverse optimization interference; if the gradient directions of L1 and L2 are consistent, the weights w1 and w2 are adjusted according to the judgment of the discriminator on the real and generated two types of samples, when the discriminator excessively trusts the real samples, that is, the probability of the discriminator judging the real samples as true is close to 1, then the weight of the generated samples w1 is increased to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, then the weight of the real samples w2 is increased to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; the main reason for this approach is that when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, it indicates that the generated samples are highly realistic, and the discriminator has lost its effective countermeasures, if the learning of the generated samples is continued to be strengthened, the risk of gradient disappearance and mode collapse will be aggravated, therefore, at this time, the weight of the real samples needs to be increased to further improve the discrimination ability of the discriminator;

[0047] (3) According to the discriminator loss, the loss function L of the AdpLossGAN network is constructed:

[0048]

[0049] wherein G(i) and R(i) represent the values of the i th pixel of the generated dose map G and the real dose map R, respectively, N3 is the total number of pixels of the generated dose map G, and LG represents the loss obtained by calculating the mean absolute error between the calculated dose map G and the real dose map R, represents the adaptive weighted loss calculated by the discriminator;

[0050] (4) Train the AdpLossGAN network with the training data set B until the loss function L converges, wherein the CT image, the planned target volume and the organ at risk in the training data set B are spliced as the network input, and the real dose distribution map is used for loss calculation;

[0051] (5) Use the trained network model to predict the test data, wherein the test data contains patient CT image, planned target volume and organ at risk map, and the output of the CPFTransGenerator generator is the dose prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Framework diagram of the generative adversarial network based on adaptive weight loss

[0053] Figure 2 Structure diagram of the generator network

[0054] Figure 3 Schematic diagram of the cross-perception fusion module

[0055] Figure 4 Schematic diagram of the hybrid CNN-Transformer structure based on cross-perception

[0056] Figure 5 Schematic diagram of the multi-scale perception fusion structure

[0057] Figure 6 Schematic diagram of the multi-scale hybrid convolution layer

[0058] Figure 7 Schematic diagram of the interactive enhancement module

[0059] Figure 8 Schematic diagram of the gating attention module

[0060] Figure 9 Structure diagram of the discriminator network

[0061] Figure 10 Dose prediction result example of the embodiment of the present application, Figure 10 (a)~(b) Figure 10 (b) is the dose prediction result map obtained by using the method in embodiment 1, Figure 10 (c)~(d) Figure 10 (d) respectively, Figure 10 (a)~(b) Figure 10 (b) corresponds to the real dose distribution map DETAILED DESCRIPTION

[0062] Embodiment 1

[0063] A tumor radiotherapy dose prediction method based on a generative adversarial network, the specific implementation steps are as follows:

[0064] (1) Establish a training data set B containing patient CT images, planning target volume, organs at risk and real dose distribution map;

[0065] (2) Construct a generative adversarial network based on adaptive weight loss, denoted as AdpLossGAN, as shown in Figure 1 , specifically including:

[0066] (2-a) Construct a generator network composed of four-stage encoding-decoding structure, denoted as CPFTransGenerator, as shown in Figure 2 , the specific structure is: the encoder contains four stages, each stage is composed of an embedding module and a cross perception fusion module connected in turn, the embedding module converts the size of the input feature to the specified size through the learnable embedding layer, that is, for the input feature with the size of HxWxC, where H, W and C represent the height, width and channel number of the input feature, after the embedding module, the size is converted to The cross perception fusion module is used as the encoding block to extract features from the feature map output by the embedding module; the decoder also contains four stages, each stage uses a cross perception fusion module as a decoding block for decoding, the specific structure includes: in the fourth stage of the decoder, the decoding block uses the output of the same stage encoding block as the input for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F4, and is sequentially subjected to 1x1 convolution and up-sampling with a step size of 8 to obtain the fourth stage decoding feature D4; in the third stage of the decoder, the gating attention module takes the feature F4 and the output of the same stage encoding block as input to perform feature fusion, and the fused feature is added to F4, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F3, and is sequentially subjected to 1x1 convolution and up-sampling with a step size of 4 to obtain the third stage decoding feature D3; in the second stage of the decoder, the gating attention module takes the feature F3 and the output of the same stage encoding block as input to perform feature fusion, and the fused feature is added to F3, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F2, and A 1×1 convolution and upsampling with a stride of 2 are performed sequentially to obtain the second-stage decoded feature D2. In the first stage of the decoder, the gated attention module uses feature F2 and the output of the same-stage coding block as input for feature fusion. The fused feature is added to F2, and the added feature is input to the cross-perception fusion module for decoding. The decoded feature is then processed. A 1×1 convolution is performed to obtain the first-stage decoding feature D1; then, the decoding features D1, D2, D3, and D4 of each stage are added together and activated by Sigmoid to obtain the dose prediction result.

[0067] (2-a-i) The cross-sensing fusion module described in step (2-a) consists of N cross-sensing-based fusion modules.

[0068] Combined with the CNN-Transformer structure, denoted as CPF Transformer, they are cascaded together, as follows: Figure 3 As shown, N is a natural number greater than 0, preferably a natural number between 3 and 8. This embodiment...

[0069] For example, N=4 is preferred;

[0070] (2-a-ii) The hybrid CNN-Transformer structure based on cross-sensing described in step (2-a-i)

[0071] Structure, denoted as CPF Transformer, such as Figure 4 As shown, the specific structure includes: constructing two branches, where the first branch consists of a self-attention residual module and two interaction enhancement modules connected together, and the second branch consists of a multi-scale perception residual module and two interaction enhancement modules connected together; introducing cross-connections between the two branches, specifically, for the input features... Where H1, W1, and C1 represent the height, width, and number of channels of the input feature map, respectively. First, for... Channel separation is performed to obtain two sub-features.

[0072] and Sub-feature X 0 and X 1 After passing through the self-attention residual module and the multi-scale perception residual module respectively, feature A is obtained. 0 and feature A 1 Feature A 0 and X 1 Input the first interaction enhancement module of the first branch to obtain feature F 0 Feature A 1 and X 0 Input the first interaction enhancement module of the second branch to obtain feature F 1 ; feature F0 and A 1 The second interactive enhancement module of the first branch is inputted to obtain feature T 0 ; feature F 1 and A 0 The second interactive enhancement module of the second branch is inputted to obtain feature T 1 ;

[0073] Finally, feature T 0 and feature T 1 are spliced to obtain the output of the CPF Transformer; the self-attention residual module of step (2-a-ii) described in (2-a-iii), the specific structure comprising: for

[0074] Input feature X 0 , sequentially pass through a normalization layer and a multi-head self-attention structure to obtain feature R 0 , then utilize a residual connection to add feature X 0 and R 0 to obtain feature A 0 ;

[0075] The multi-head self-attention structure in the embodiment is preferably a CSWin Transformer structure;

[0076] The multi-scale perception residual module of step (2-a-ii) described in (2-a-iii), the specific structure comprising: for

[0077] Input feature X 1 , sequentially pass through a normalization layer and a multi-scale perception fusion structure to obtain feature R 1 , then utilize a residual connection to add feature X 1 and R 1 to obtain feature

[0078] A 1 ;

[0079] The multi-scale perception fusion structure of step (2-a-iii) described in (2-a-iv), denoted as MSPF, as shown in Figure 5

[0080] The specific structure comprises: input feature X inThe three linear layers are respectively input to obtain features h, w and p; the feature h is input to a multi-scale mixed convolution layer to obtain a feature m1, and the feature m1 is multiplied by the feature w to obtain a feature f1; the feature f1 is input to a multi-scale mixed convolution layer to obtain a feature m2, and the feature m2 is multiplied by the feature w to obtain a feature f2; the feature f2 is input to a multi-scale mixed convolution layer to obtain a feature m3, and the feature m3 is multiplied by the feature w to obtain a feature f3; the feature f3 is input to a linear layer to obtain a feature k, and the feature k is multiplied by the feature p to obtain

[0081] The output R of the MSPF module 1 ;

[0082] (2-a-v) the multi-scale mixed convolution layer of step (2-a-iv), as shown in Figure 6 , the specific structure of which comprises:

[0083] The input feature q is channel-separated, where H2, W2 and C2 respectively represent the height, width and channel number of the input feature q, that is, the input feature q is evenly divided into four groups according to the number of channels, to obtain four sub-features

[0084] and The four sub-features are respectively input to depth separable convolutions with convolution kernels of 3x3, 5x5, 7x7 and 9x9, and then the obtained features are spliced to obtain

[0085] The output feature q' of the multi-scale mixed convolution layer;

[0086] (2-a-vi) the interactive enhancement module of step (2-a-ii), denoted as IE, as shown in Figure 7 , the specific structure of which comprises: Figure 4 The IE module accepts two inputs, namely the input feature from the current branch, as shown by the solid line in 7 , and the input feature from another branch, as shown by the dashed line in Figure 4 and 7 ; first, the input feature C from the current branch and the input feature O from another branch are spliced, and then the spliced feature is sequentially input to a normalization layer and a fully connected feedforward network to obtain a feature H, and then the feature H and C are added through a residual connection to obtain an output feature;

[0087] (2-a-vii) the gating attention module of step (2-a), denoted as AG, as shown in Figure 8 , the specific structure of which comprises:

[0088] ​The structure includes: an AG module accepts two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, i.e., the output of the encoding block at the same stage; v1 and v2 are respectively input into two parallel branches to obtain features q1 and q2, each branch containing a sequentially connected 1×1 convolution, a normalization layer, and a ReLU activation function; features q1 and q2 are added together, and the result of the addition is sequentially processed by a 1×1 convolution, a normalization layer, and a Sigmoid activation to obtain feature z; and z is input into feature s1.

[0089] Perform a dot product to obtain the output of the gated attention module;

[0090] (2-b) Construct a discriminator network, such as Figure 9 As shown, the network consists of an initial convolutional layer and three intermediate convolutional layers.

[0091] The input image is composed of a convolutional layer and an output convolutional layer connected sequentially. The specific structure includes: for the input image...

[0092] Where H3, W3, and C3 represent the height, width, and number of channels of the input feature I, respectively. Input an initial convolutional layer containing a 4x4 convolution with a stride of 2 and a Leaky ReLU activation function to obtain features. The feature t1 is sequentially input into three intermediate convolutional layers to obtain the features. Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer, and a Leaky ReLU activation function. The stride of the first two intermediate convolutional layers' 4x4 convolutions is 2, and the stride of the third intermediate convolutional layer's 4x4 convolution is 1. The features... Inputting a 4x4 convolutional layer with a stride of 1 and an output convolutional layer with a sigmoid activation function yields a two-dimensional discriminant matrix that is related to the size of the input image.

[0093] Each element in matrix J represents an 8×8 local region in the input image that is the true local region.

[0094] The probability of the data;

[0095] (2-c) Compare the generated dose image G and the true dose image output by the CPFTransGenerator network.

[0096] Inputting R into the discriminator described in step (2-b) yields two-dimensional discriminant matrices J1 and J2 corresponding to the generated dose image G and the real dose image R, respectively. A discriminator loss function is constructed, and by dynamically adjusting the loss weights of the real and generated data, the gradient direction impulse caused by traditional equal-weight summation is resolved.

[0097] The problem is that the discriminator loss function is defined as:

[0098]

[0099] where L1 and L2 represent the generated data loss and the real data loss, respectively, N1 and N2 represent the total number of pixels in the discrimination matrices J1 and J2, respectively, J2(p) represents the value of the pth pixel in the discrimination matrix J2, J1(p) represents the value of the pth pixel in the discrimination matrix J1; w1 and w2 are adaptive dynamic adjustment weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgments of the discriminator on real and generated dose maps, specifically including: first, calculate the included angle of the gradient directions of the generated result loss L1 and the real dose loss L2, if the included angle exceeds 90°, it indicates that the two gradients

[0100] conflict, at this time, the weight is dynamically allocated according to the gradient, specifically:

[0101]

[0102] where ε is a small constant value, used to prevent the denominator from being zero or the weight being too small, ε is in the range of (0.001, 0.1), and the preferred value of ε is 0.01. This weight allocation method can make the total gradient direction orthogonal to the conflicting item or If the gradient directions of L1 and L2 are consistent, the weights w1 and w2 are adjusted according to the judgment of the discriminator on real and generated samples, when the discriminator excessively trusts the real samples, that is, the probability of the discriminator judging the real samples as true is close to 1, then the weight of the generated samples w1 is increased to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, then the weight of the real samples w2 is increased to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; the main reason for this approach is that when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, it indicates that the generated samples are highly realistic, and the discriminator has lost its effective countermeasures, if the learning of the generated samples is continued, it will exacerbate the risk of gradient disappearance and mode collapse, therefore, at this time, the weight of the real samples needs to be increased to further improve the discrimination ability of the discriminator;

[0103] (3) According to the discriminator loss, the loss function L of the AdpLossGAN network is constructed:

[0104]

[0105] wherein G(i) and R(i) represent the value of the i-th pixel of the generated dose map G and the real dose map R respectively, N3 is the total number of pixels of the generated dose map G, L G represents the loss obtained by calculating the mean absolute error between the measured dose map G and the real dose map R, represents the adaptive weighted loss calculated by the discriminator;

[0106] (4) training the AdpLossGAN network with the training data set B until the loss function L converges, wherein the CT image, the planned target volume and the organ at risk in the training data set B are spliced as the network input, and the real dose distribution map is used for loss calculation;

[0107] (5) using the trained network model to predict the test data, wherein the test data contains the patient CT image, the planned target volume and the organ at risk map, and the output of the CPFTransGenerator generator is the dose prediction result.

[0108] Embodiment 2

[0109] The method in embodiment 1 is used to conduct experiments on a clinical liver cancer data set from a certain third-class hospital. The data set contains 60 liver cancer patient data, each patient data contains a patient CT image, a planned target volume map, an organ at risk map and a clinical dose distribution map, the CT image size is 512*512*Q, 112≤Q≤238. In the experiment, the data is divided into a test data set and a training data set according to a ratio of 1:4.

[0110] Figure 10 Some dose prediction result examples obtained on the test set are shown. Figure 10 (a)~ Figure 10 (b) is a dose prediction result map obtained by using the method in embodiment 1, Figure 10 (c)~ Figure 10 (d) respectively, Figure 10 (a)~ Figure 10 (b) is the corresponding real dose distribution map, it can be seen that the dose prediction result obtained by the method can better approximate the real dose distribution.

[0111] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1.A method for predicting a tumor radiotherapy dose based on a generative adversarial network, characterized in that, The method comprises the following steps: (1) establishing a training data set B comprising a patient CT image, a planned target volume, an organ at risk and a true dose distribution map; (2) constructing an adaptive weight loss-based generative adversarial network, denoted as AdpLossGAN, which specifically comprises: (2-a) constructing a generator network composed of four-stage encoding-decoding structure, denoted as CPFTransGenerator, the specific structure is: the encoder contains four stages, each stage is composed of an embedding module and a cross perception fusion module connected in turn, the embedding module converts the size of the input feature to the specified size through the learnable embedding layer, that is, for the input feature with the size of HxWxC, where H, W and C represent the height, width and channel number of the input feature respectively, after the embedding module, the size is converted to The cross perception fusion module is used as an encoding block to extract features from the feature map output by the embedding module; the decoder also contains four stages, each stage uses a cross perception fusion module as a decoding block for decoding, the specific structure includes: in the fourth stage of the decoder, the decoding block uses the output of the encoding block in the same stage as the input for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F4, and at the same time is sequentially subjected to 1x1 convolution and up-sampling with a step size of 8 to obtain the fourth-stage decoding feature D4; in the third stage of the decoder, the gating attention module takes the feature F4 and the output of the encoding block in the same stage as the input for feature fusion, and the fused feature is added to F4, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F3, and at the same time is sequentially subjected to 1x1 convolution and up-sampling with a step size of 4 to obtain the third-stage decoding feature D3; in the second stage of the decoder, the gating attention module takes the feature F3 and the output of the encoding block in the same stage as the input for feature fusion, and the fused feature is added to F3, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F2, and at the same time is sequentially subjected to 1x1 convolution and up-sampling with a step size of 2 to obtain the second-stage decoding feature D2; in the first stage of the decoder, the gating attention module takes the feature F2 and the output of the encoding block in the same stage as the input for feature fusion, and the fused feature is added to F2, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the 1x1 convolution to obtain the first-stage decoding feature D1; then, the decoding features D1, D2, D3 and D4 of each stage are added and subjected to Sigmoid activation to obtain the dose prediction result; (2-a-i) the cross-perception fusion module in step (2-a) is composed of N cross-perception-based hybrid CNN-Transformer structures, denoted as CPF Transformer, in cascade, wherein N is a natural number greater than 0; (2-a-ii) The cross perception based hybrid CNN-Transformer structure of step (2-a-i) is recorded as CPFTransformer, and the specific structure comprises: two branches are constructed, wherein the first branch is composed of one self-attention residual module and two interactive enhancement modules, and the second branch is composed of one multi-scale perception residual module and two interactive enhancement modules; cross connection is introduced between the two branches, and the specific structure is that, for the input feature wherein H1, W1 and C1 represent the height, width and channel number of the input feature map respectively, firstly, channel separation is performed on to obtain two sub-features and After the sub-features X 0 and X 1 pass through the self-attention residual module and the multi-scale perception residual module respectively, the features A 0 and A 1 are obtained; the features A 0 and X 1 are input into the first interactive enhancement module of the first branch to obtain the feature F 0 ; the features A 1 and X 0 are input into the first interactive enhancement module of the second branch to obtain the feature F 1 ; the features F 0 and A 1 are input into the second interactive enhancement module of the first branch to obtain the feature T 0 ; the features F 1 and A 0 are input into the second interactive enhancement module of the second branch to obtain the feature T 1 ; finally, the features T 0 and T 1 are spliced to obtain the output of the CPFTransformer. (2-a-iii) the self-attention residual module of step (2-a-ii), and the specific structure comprises: for input feature X 0 , sequentially passing through a normalization layer and a multi-head self-attention structure to obtain feature R 0 , then using a residual connection to add features X 0 and R 0 to obtain feature A 0 ; (2-a-iv) the multi-scale perception residual module of step (2-a-ii), and the specific structure comprises: for input feature X 1 , sequentially passing through a normalization layer and a multi-scale perception fusion structure to obtain feature R 1 , then using a residual connection to add features X 1 and R 1 to obtain feature A 1 ; (2-a-Ⅴ) The multi-scale perception fusion structure described in step (2-a-Ⅳ) is denoted as MSPF, and the specific structure comprises: inputting the input feature X in into three linear layers respectively to obtain features h, w and p; inputting the feature h into a multi-scale mixed convolution layer to obtain a feature m1, and multiplying the feature m1 by the feature w to obtain a feature f1; inputting the feature f1 into a multi-scale mixed convolution layer to obtain a feature m2, and multiplying the feature m2 by the feature w to obtain a feature f2; inputting the feature f2 into a multi-scale mixed convolution layer to obtain a feature m3, and multiplying the feature m3 by the feature w to obtain a feature f3; inputting the feature f3 into a linear layer to obtain a feature k, and multiplying the feature k by the feature p to obtain an output R of the MSPF module 1 ; (2-a-Ⅴ) The multi-scale mixed convolutional layer described in step (2-a-Ⅳ) has the following specific structure: the input feature Channel separation is performed, where H2, W2, and C2 represent the height, width, and channel number of the input feature q, that is, the input feature is evenly divided into four groups according to the number of channels, and four sub-features are obtained and The four sub-features are respectively input into depth separable convolutions with convolution kernels of 3×3, 5×5, 7×7, and 9×9, and then the obtained features are spliced to obtain the output feature q' of the multi-scale mixed convolutional layer. (2-a-ⅵ) the interactive enhancement module in step (2-a-ⅱ), denoted as IE, is characterized in that the specific structure comprises: the IE module accepts two inputs, namely the input feature from the current branch and the input feature from the other branch; first, the input feature C from the current branch and the input feature O from the other branch are spliced, and then the spliced feature is sequentially input into a normalization layer and a fully connected feedforward network to obtain a feature H, and then the feature H and the feature C are added through a residual connection to obtain an output feature; (2-b) constructing a discriminator network, which is sequentially connected by an initial convolutional layer, three intermediate convolutional layers and an output convolutional layer, and the specific structure includes: for the input image wherein H3, W3, C3 represent the height, width and channel number of the input feature I respectively, and inputting an initial convolutional layer containing a 4x4 convolution with a step of 2 and a LeakyReLU activation function to obtain a feature inputting the feature t1 into three intermediate convolutional layers in sequence to obtain features Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer and a LeakyReLU activation function, wherein the step of the 4x4 convolution of the first two intermediate convolutional layers is 2, and the step of the 4x4 convolution of the third intermediate convolutional layer is 1; inputting the features inputting an output convolutional layer containing a 4x4 convolution with a step of 1 and a Sigmoid activation function to obtain a two-dimensional judgment matrix related to the size of the input image (2-c) inputting the generated dose image G and the real dose image R output by the CPFTransGenerator generator network into the discriminator described in step (2-b) respectively to obtain the two-dimensional judgment matrix J1 and J2 corresponding to the generated dose image G and the real dose image R; constructing a discriminator loss function to dynamically adjust the loss weight of the real data and the generated data; specifically, the discriminator loss function is defined as: wherein L1 and L2 represent the generated data loss and the true data loss respectively, N1 and N2 represent the total number of pixels of the discrimination matrices J1 and J2 respectively, J2(p) represents the value of the pth pixel in the discrimination matrix J2, J1(p) represents the value of the pth pixel in the discrimination matrix J1; w1 and w2 are adaptive dynamic adjustment weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgment of the discriminator on the true and generated dose maps, specifically comprising: first, the included angle between the gradient directions of the generated result loss L1 and the true dose loss L2 is calculated, if the included angle exceeds 90°, it indicates that the two gradient directions conflict, at this time, the weight is dynamically allocated according to the gradient, specifically: wherein ε is a small constant value, used to prevent the denominator from being zero or the weight being too small, and the value range of ε is (0.001, 0.1), and this weight distribution method makes the total gradient direction orthogonal to the conflict item or , avoiding interference of reverse optimization; if the gradient directions of L1 and L2 are consistent, the weights w1 and w2 are adjusted in combination with the judgment of the discriminator on the real and generated samples, when the discriminator excessively trusts the real samples, that is, the probability of the discriminator judging the real samples as real is close to 1, the weight w1 of the generated samples is increased to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as real is close to 1, the weight w2 of the real samples is increased to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; (3) constructing a loss function L of the AdpLossGAN network according to the discriminator loss: wherein, wherein, G(i) and R(i) represent the value of the i-th pixel of the generated dose map G and the real dose map R, respectively, N3 is the total number of pixels of the generated dose map G, L G represents the loss obtained by calculating the mean absolute error between the measured dose map G and the real dose map R, represents the adaptive weighted loss calculated by the discriminator; (4) training the AdpLossGAN network using the training data set B until the loss function L converges, wherein the CT image, the planned target volume and the organ at risk in the training data set B are spliced as the network input, and the true dose distribution map is used for loss calculation; (5) using the trained network model to predict the test data, wherein the test data comprises a patient CT image, a planned target volume and an organ at risk map, and the output of the CPFTransGenerator generator is the dose prediction result. 2.The tumor radiotherapy dose prediction method based on a generative adversarial network according to claim 1, wherein, The gating attention module described in step (2-a) is denoted as AG, and the specific structure includes: the AG module accepts two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, that is, the output of the encoding block of the same stage; input v1 and v2 into two parallel branches respectively to obtain features q1 and q2, each branch contains a 1x1 convolution, a normalization layer and a ReLU activation function connected in turn; add features q1 and q2, and perform 1x1 convolution, normalization layer and Sigmoid activation on the added result in turn to obtain feature z; point multiply input feature s1 with z to obtain the output of the gating attention module. 3.The tumor radiotherapy dose prediction method based on a generative adversarial network according to claim 1, wherein, The multi-head self-attention structure described in step (2-a-ⅲ) is a CSWin Transformer structure.

Images