A head and neck cancer radiotherapy dose prediction method based on cross-sensing fusion

Abstract

The application discloses a head and neck cancer radiotherapy dose prediction method based on cross perception fusion, comprising the following steps: (1) establishing a training data set containing a patient CT image, a planning target volume, an organ at risk and a real dose distribution; (2) constructing a generative adversarial network based on cross perception fusion, introducing a cross perception fusion module in the generator, realizing the deep interaction and fusion of local and global features, and introducing a multi-scale cross window self-attention coding network to constrain the difference between the generated and real doses at multiple scales; (3) constructing a hybrid loss function based on the discriminator and the multi-scale cross window self-attention coding network; (4) training the generative adversarial network by using the training data set; and (5) testing the data to be tested by using the trained network, and the output of the generator is the dose prediction result. The application can realize the direct generation of the head and neck cancer radiotherapy dose distribution, assist doctors in formulating accurate radiotherapy plans, and improve the radiotherapy effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, and in particular to a head and neck cancer radiotherapy dose prediction method based on cross perception fusion. BACKGROUND

[0002] Head and neck cancer is a common malignant tumor, which usually occurs in squamous cells on the mucosal surface of the head and neck, such as squamous cells inside the mouth, throat and vocal cords. Radiotherapy is one of the main treatment methods for head and neck cancer, aiming to provide sufficient dose to the planning target volume while protecting the surrounding organs at risk. Quantitative dose prediction of each tissue and organ is the premise of intelligent precision radiotherapy. Traditional manual radiotherapy planning needs to be constantly adjusted and optimized manually by the dose planner in a trial-and-error manner to obtain a dose distribution that meets clinical requirements. This process is not only tedious and time-consuming, but also heavily dependent on the knowledge level and experience of the dose planner.

[0003] In recent years, with the development of deep learning technology, researchers have successively proposed many methods based on convolutional neural networks (CNN) to provide new solutions for predicting dose distribution maps. These deep CNN models can learn the non-linear mapping characteristics between input data (such as CT images, planning target volume (PTV), organs at risk (OAR) masks) and output (dose distribution map), realize automatic prediction of radiotherapy dose, and reduce the cost and time of manual intervention.

[0004] Although CNN-based methods have achieved encouraging results, they all have a common feature that convolution operations are limited to local receptive fields, making it difficult to model long-distance dependencies and effectively extract global features. However, the anatomical structure of the head and neck is complex, surrounded by many important organs and tissues such as the mouth, throat, spinal cord, brainstem, etc., and their dose distribution has global characteristics, while CNN-based networks are difficult to effectively capture global dose information. Given the powerful long-distance modeling capability of Transformer, some studies have begun to apply Transformer to the field of dose distribution prediction.

[0005] However, the existing Transformer-based methods mostly adopt simple stacking or embedding strategies, which have certain limitations. Specifically, stacking multiple Transformer blocks requires multiple self-attention mechanism and feedforward neural network calculations, resulting in excessive consumption of computing resources. Using the method of stacking multiple Transformer blocks to extract dosimetric features is prone to neglecting the local dependency relationship in the dose prediction task, and the simple embedding strategy is difficult to realize the deep understanding and interaction of local and global features. In addition, the existing dose prediction methods usually use global images for model loss calculation. Head and neck radiotherapy involves numerous organs of different sizes, and directly calculating the loss on the whole image cannot perform layered dose constraints according to the size of the organs, thereby resulting in low dose prediction accuracy. SUMMARY

[0006] In view of the defects and shortcomings of the prior art, the present application develops a head and neck radiotherapy dose prediction method based on cross perception fusion. The method uses a generative adversarial network (GAN) as the main framework of the prediction model. Specifically, a cross perception fusion module based on CNN and Transformer technology is designed, and the module is used to build the generator in GAN, thereby improving the ability of the model to extract image features from global and local perspectives. At the same time, adaptive weight loss and multi-scale cross window self-attention loss are introduced to refine the specific dose distribution of various local organs and generate more accurate dose distribution maps.

[0007] The present application is realized by the following scheme:

[0008] (1) Establish a training data set B containing patient CT images, planning target volume, organs at risk and real dose distribution maps;

[0009] (2) Construct a generative adversarial network based on cross perception fusion, denoted as CPFTransGAN, which specifically includes:

[0010] (2-a) Construct a generator network composed of a four-stage encoding-decoding structure, denoted as CPFTransGenerator, and the specific structure is as follows: the encoder includes four stages, each of which is composed of an embedding module and a cross perception fusion module connected in turn. The embedding module converts the size of the input feature to a specified size through a learnable embedding layer, i.e. for an input feature with a size of HxWxC, where H, W and C represent the height, width and channel number of the input feature, respectively, after passing through the embedding module, its size is converted to The cross perception fusion module is taken as an encoding block to perform feature extraction on the feature map output by the embedding module; the decoder also includes four stages, and each stage adopts a cross perception fusion module as a decoding block for decoding. The specific structure includes: in the fourth stage of the decoder, the decoding block adopts the output of the encoding block in the same stage as input for decoding, adds the decoded feature to the feature F4, and inputs the added feature into the cross perception fusion module for decoding to obtain the fourth-stage decoding feature D4. The up-sampling convolution module with a step of 2 is input to obtain the feature F4, and the feature F4 is also input into the gating attention module for feature fusion.

[0011] The 1x1 convolution and the up-sampling with a step of 8 are sequentially performed to obtain the fourth-stage decoding feature D4; in the third stage of the decoder, the gating attention module takes the feature F4 and the output of the encoding block in the same stage as input to perform feature fusion, adds the fused feature to the feature F4, and inputs the added feature into the cross perception fusion module for decoding to obtain the third-stage decoding feature D3. The up-sampling convolution module with a step of 2 is input to obtain the feature F3, and the feature F3 is also input into the gating attention module for feature fusion. The 1x1 convolution and the up-sampling with a step of 4 are sequentially performed to obtain the third-stage decoding feature D3; in the second stage of the decoder, the gating attention module takes the feature F3 and the output of the encoding block in the same stage as input to perform feature fusion, adds the fused feature to the feature F3, and inputs the added feature into the cross perception fusion module for decoding to obtain the second-stage decoding feature D2. The up-sampling convolution module with a step of 2 is input to obtain the feature F2, and the feature F2 is also input into the gating attention module for feature fusion. The 1x1 convolution and the up-sampling with a step of 2 are sequentially performed to obtain the second-stage decoding feature D2; in the first stage of the decoder, the gating attention module takes the feature F2 and the output of the encoding block in the same stage as input to perform feature fusion, adds the fused feature to the feature F2, and inputs the added feature into the cross perception fusion module for decoding to obtain the first-stage decoding feature D1. The 1x1 convolution is performed to obtain the first-stage decoding feature D1; then, the first-stage decoding feature D1, the second-stage decoding feature D2, the third-stage decoding feature D3, and the fourth-stage decoding feature D4 are added and subjected to Sigmoid activation to obtain the dose prediction result.

[0012] (2-a-i) The cross perception fusion module in step (2-a) is composed of N cross perception-based hybrid CNN-Transformer structures, denoted as CPF Transformer, which are cascaded, wherein N is a natural number greater than 0, preferably a natural number between 3 and 8.

[0013] (2-a-ii) The cross-aware hybrid CNN-Transformer structure described in step (2-a-i), denoted as CPF Transformer, specifically includes: constructing two branches, where the first branch consists of a self-attention residual module and two interaction enhancement modules connected together, and the second branch consists of a multi-scale perception residual module and two interaction enhancement modules connected together; introducing cross-connections between the two branches, specifically, for input features... Where H1, W1, and C1 represent the height, width, and number of channels of the input feature map, respectively. First, for... Channel separation is performed to obtain two sub-features. and Sub-feature X 0 and X 1 After passing through the self-attention residual module and the multi-scale perception residual module respectively, feature A is obtained. 0 and feature A 1 Feature A 0 and X 1 Input the first interaction enhancement module of the first branch to obtain feature F 0 Feature A 1 and X 0 Input the first interaction enhancement module of the second branch to obtain feature F 1 ; feature F 0 and A 1 Input the second interaction enhancement module of the first branch to obtain feature T. 0 ; feature F 1 and A 0 Input the second interaction enhancement module of the second branch to obtain feature T. 1 ;

[0014] Finally, feature T 0 and feature T 1 The output of the CPF Transformer is obtained by concatenation; (2-a-iii) The self-attention residual module described in step (2-a-ii) has the following specific structure: For

[0015] Input feature X 0 The feature R is obtained by passing through a normalization layer and a multi-head self-attention structure in sequence. 0 Then, using residual connections, the feature X is... 0 and R 0 Add them together to obtain feature A. 0 ;

[0016] The preferred multi-head self-attention structure is the CSWin Transformer structure.

[0017] (2-a-ⅳ) The multi-scale sensing residual module described in step (2-a-ⅱ) has the following specific structure: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.]

[0018] Input feature X 1 The feature R is obtained by passing through a normalization layer and a multi-scale perceptual fusion structure in sequence. 1 Then, using residual connections, the feature X is... 1 and R 1 Add them together to obtain the features.

[0019] A 1 ;

[0020] (2-a-v) The multi-scale perceptual fusion structure described in step (2-a-v) is denoted as MSPF, and its specific structure is as follows:

[0021] The structure includes: input features X in The inputs are fed into three linear layers to obtain features h, w, and p. Feature h is then fed into a multi-scale hybrid convolutional layer to obtain feature m1, and feature m1 is multiplied by feature w to obtain feature f1. Feature f1 is fed into a multi-scale hybrid convolutional layer to obtain feature m2, and feature m2 is multiplied by feature w to obtain feature f2. Feature f2 is fed into a multi-scale hybrid convolutional layer to obtain feature m3, and feature m3 is multiplied by feature w to obtain feature f3. Feature f3 is fed into a linear layer to obtain feature k, and feature k is multiplied by feature p to obtain the output of the MSPF module.

[0022] Out of R 1 ;

[0023] (2-a-v) The multi-scale hybrid convolutional layer described in step (2-a-v) specifically includes the following structure: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.]

[0024] Input characteristics Channel separation is performed, where H2, W2, and C2 represent the height, width, and number of channels of the input feature q, respectively. That is, the input feature is divided into four groups based on the number of channels, resulting in four sub-features. and The four sub-features are respectively input into depthwise separable convolutions with kernels of 3×3, 5×5, 7×7, and 9×9. The resulting features are then concatenated to obtain the output feature q′ of the multi-scale hybrid convolutional layer.

[0025] (2-a-ⅶ) The interactive enhancement module in step (2-a-ⅱ), denoted as IE, has a specific structure comprising: the IE module accepts two inputs, i.e., the input features from the current branch and the input features from another branch; first, the input features C from the current branch and the input features O from another branch are spliced, and then the spliced features are sequentially input into a normalization layer and a fully connected feedforward network to obtain features H, and then the features H and C are added through a residual connection to obtain output features;

[0026] (2-a-ⅷ) The gating attention module in step (2-a), denoted as AG, has a specific structure comprising: the AG module accepts two inputs v1 and v2, where v1 represents the output of the up-sampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, i.e., the output of the encoding block in the same stage; features q1 and q2 are obtained by inputting v1 and v2 into two parallel branches, respectively, each branch containing a 1×1 convolution, a normalization layer, and a ReLU activation function connected in sequence; the features q1 and q2 are added, and the added result is sequentially executed with a 1×1 convolution, a normalization layer, and a Sigmoid activation to obtain a feature z; the z input feature s1 is point multiplied to obtain the output of the gating attention module;

[0027] (2-b) Constructing a discriminator network composed of an initial convolutional layer, three intermediate convolutional layers, and an output convolutional layer connected in sequence, the specific structure comprising: for the input image where H3, W3, and C3 represent the height, width, and channel number of the input feature I, respectively, and An initial convolutional layer containing a 4x4 convolution with a step of 2 and a LeakyReLU activation function is input to obtain a feature The feature t1 is sequentially input into three intermediate convolutional layers to obtain features Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer, and a LeakyReLU activation function, where the step of the 4x4 convolution of the first two intermediate convolutional layers is 2, and the step of the 4x4 convolution of the third intermediate convolutional layer is 1; the features An output convolutional layer containing a 4x4 convolution with a step of 1 and a Sigmoid activation function is input to obtain a two-dimensional discrimination matrix related to the size of the input image Each element in matrix J represents the probability that an 8×8 local region in the input image is real data;

[0028] (2-c) input the generated dose image G output by the CPFTransGenerator generator network and the real dose image R into the discriminator described in step (2-b) respectively to obtain two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the real dose image R; construct a discriminator loss function, solve the gradient direction conflict problem caused by traditional equal-weight summation by dynamically adjusting the loss weights of real data and generated data; specifically, the discriminator loss function is defined as:

[0029]

[0030] wherein L1 and L2 represent generated data loss and real data loss respectively, N1 and N2 represent the total number of pixels of the discrimination matrices J1 and J2 respectively, J2(p) represents the value of the pth pixel in the discrimination matrix J2, J1(p) represents the value of the pth pixel in the discrimination matrix J1; w1 and w2 are adaptive dynamically adjusted weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgments of the discriminator on the real and generated dose maps, specifically including: first, calculate the included angle of the gradient directions of the generated result loss L1 and the real dose loss L2, if the included angle exceeds 90°, it indicates that the two gradient directions conflict, at this time, the weight is dynamically distributed according to the gradient, specifically:

[0031]

[0032] wherein ε is a small constant value, used to prevent the denominator from being zero or the weight being too small, and the value range of ε is (0.001, 0.1), this weight distribution method can make the total gradient direction orthogonal to the conflict item or avoid reverse optimization interference; if the gradient directions of L1 and L2 are consistent, adjust the weights w1 and w2 in combination with the judgments of the discriminator on the real and generated two types of samples, when the discriminator excessively trusts the real samples, that is, the probability of the discriminator judging the real samples as true is close to 1, then increase the weight w1 of the generated samples to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, then increase the weight w2 of the real samples to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; the main reason for this approach is that when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, it indicates that the generated samples are highly realistic, and the discriminator loses effective antagonistic ability, if the learning of the generated samples is continued to be strengthened, the risk of gradient disappearance and mode collapse will be intensified, therefore, at this time, the weight of the real samples needs to be increased to further improve the discrimination ability of the discriminator;

[0033] (2-d) Construct a multi-scale cross-window self-attention encoding network, denoted as MCENet, the input of the network contains the generated dose map and the real dose map, MCENet network is a four-stage encoding network, the specific structure includes: the generated dose G and the real dose R are respectively input into an encoding block containing a CSWin self-attention module and a maximum pooling layer with a step of 2, to obtain intermediate features g1 and r1, and the mean absolute error is used to calculate the loss of the intermediate features g1 and r1, to obtain the loss of the generated dose map G and the real dose map R in the first stage Where g1(j) and r1(j) represent the jth pixel in the feature map g1 and r1 respectively, T1 represents the total number of pixels in the feature map g1; the features g1 and r1 are respectively input into an encoding block containing a CSWin self-attention module and a maximum pooling layer with a step of 2, to obtain intermediate features g2 and r2, and the mean absolute error is used to calculate the loss of the intermediate features g2 and r2, to obtain the loss of the generated dose map G and the real dose map R in the second stage Where g2(j) and r2(j) represent the jth pixel in the feature map g2 and r2 respectively, T2 represents the total number of pixels in the feature map g2; the features g2 and r2 are respectively input into an encoding block containing a CSWin self-attention module and a maximum pooling layer with a step of 2, to obtain intermediate features g3 and r3, and the mean absolute error is used to calculate the loss of the intermediate features g3 and r3, to obtain the loss of the generated dose map G and the real dose map R in the third stage Where g3(j) and r3(j) represent the jth pixel in the feature map g3 and r3 respectively, T3 represents the total number of pixels in the feature map g3; the features g3 and r3 are respectively input into an encoding block containing a CSWin self-attention module and a maximum pooling layer with a step of 2, to obtain intermediate features g4 and r4, and the mean absolute error is used to calculate the loss of the intermediate features g4 and r4, to obtain the loss of the generated dose map G and the real dose map R in the fourth stage Where g4(j) and r4(j) represent the jth pixel in the feature map g4 and r4 respectively, T4 represents the total number of pixels in the feature map g4; S1, S2, S3 and S4 are weighted and added, and the result is used as the loss of the multi-scale cross-window self-attention encoding network, to constrain the multi-scale difference between the generated dose map and the real dose map; specifically, the MCENet network loss function L mcs is defined as:

[0034]

[0035] Where, λ i is the weight coefficient of the i-th stage loss, g1(j) and r1(j) represent the jth pixel in the intermediate feature map g i and r ithe jth pixel in the image, T i representing the feature map g i representing the total number of pixels.

[0036] (3) According to the discriminator and MCENet network loss, the loss function L of the CPFTransGAN network is constructed:

[0037]

[0038] wherein G(i) and R(i) represent the values of the ith pixel of the generated dose map G and the real dose map R respectively, N3 is the total number of pixels of the generated dose map G, L G representing the loss obtained by calculating the mean absolute error between the measured dose map G and the real dose map R, representing the adaptive weighted loss calculated by the discriminator, L mcs representing the loss calculated by the multi-scale cross window self-attention network;

[0039] (4) The CPFTransGAN network is trained using the training data set B until the loss function L converges, wherein the CT image, the planned target volume and the organ at risk in the training data set B are spliced as the network input, and the real dose distribution map is used for loss calculation;

[0040] (5) The trained network model is used to predict the test data, wherein the test data contains patient CT image, planned target volume and organ at risk map, and the output of CPFTransGenerator generator is the dose prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a framework diagram of the cross perception fusion based generative adversarial network

[0042] Figure 2 is a structure diagram of the generator network

[0043] Figure 3 is a schematic diagram of the cross perception fusion module

[0044] Figure 4 is a schematic diagram of the cross perception based hybrid CNN-Transformer structure

[0045] Figure 5 is a schematic diagram of the multi-scale perception fusion structure

[0046] Figure 6 is a schematic diagram of the multi-scale hybrid convolution layer

[0047] Figure 7 is a schematic diagram of the interactive enhancement module

[0048] Figure 8 Schematic diagram of the gated attention module

[0049] Figure 9 Structure diagram of the discriminator network

[0050] Figure 10 Multi-scale cross window self-attention encoding network

[0051] Figure 11 Dose prediction result example of the embodiment of the present application, Figure 11 (a) ~ Figure 11 (c) are three original CT images randomly selected from the public head and neck cancer dataset of OpenKBP-2020 AAPM Grand Challenge, Figure 11 (d) ~ Figure 11 (f) is the dose prediction result obtained by testing the CT images shown in (a) ~ (c) using the method in Embodiment 1, Figure 11 (a) ~ Figure 11 (c) shown in the CT images, Figure 11 (g) ~ Figure 11 (i) is the dose distribution map of the CT images shown in (a) ~ (c) Figure 11 (a) ~ Figure 11 (c) shown in the CT images DETAILED DESCRIPTION

[0052] Embodiment 1

[0053] A head and neck cancer radiotherapy dose prediction method based on cross perception fusion, the specific implementation steps are as follows:

[0054] (1) Establish a training data set B containing patient CT images, planning target volume, organs at risk and real dose distribution map;

[0055] (2) Construct a cross perception fusion based generative adversarial network, denoted as CPFTransGAN, as shown in Figure 11 , which specifically includes:

[0056] (2-a) Construct a generator network composed of four-stage encoding-decoding structure, denoted as CPFTransGenerator, as shown in Figure 1 , the specific structure is: the encoder contains four stages, each stage is composed of an embedding module and a cross perception fusion module connected in turn, the embedding module converts the size of the input feature to the specified size through a learnable embedding layer, that is, for the input feature with a size of HxWxC, where H, W and C represent the height, width and channel number of the input feature, respectively, after the embedding module, its size is converted to The cross perception fusion module serves as an encoding block to extract features from the feature map output by the embedding module. The decoder also includes four stages, each of which uses a cross perception fusion module as a decoding block for decoding. The specific structure includes: in the fourth stage of the decoder, the decoding block uses the output of the encoding block in the same stage as input for decoding, and the decoded features are added to the features F4 to obtain the fourth-stage decoding features D4. The up-sampling convolution module with a step size of 2 is used to obtain the features F4, and the features F4 are added to the features F4 to obtain the fourth-stage decoding features D4. The 1x1 convolution and the up-sampling with a step size of 8 are sequentially performed to obtain the fourth-stage decoding features D4; in the third stage of the decoder, the gating attention module uses the features F4 and the output of the encoding block in the same stage as input for feature fusion, and the fused features are added to the features F4. The features after addition are input into the cross perception fusion module for decoding, and the decoded features are added to the features F4 to obtain the third-stage decoding features D3. The up-sampling convolution module with a step size of 2 is used to obtain the features F3, and the features F3 are added to the features F3 to obtain the third-stage decoding features D3. The 1x1 convolution and the up-sampling with a step size of 4 are sequentially performed to obtain the third-stage decoding features D3; in the second stage of the decoder, the gating attention module uses the features F3 and the output of the encoding block in the same stage as input for feature fusion, and the fused features are added to the features F3. The features after addition are input into the cross perception fusion module for decoding, and the decoded features are added to the features F3 to obtain the second-stage decoding features D2. The up-sampling convolution module with a step size of 2 is used to obtain the features F2, and the features F2 are added to the features F2 to obtain the second-stage decoding features D2. The 1x1 convolution and the up-sampling with a step size of 2 are sequentially performed to obtain the second-stage decoding features D2; in the first stage of the decoder, the gating attention module uses the features F2 and the output of the encoding block in the same stage as input for feature fusion, and the fused features are added to the features F2. The features after addition are input into the cross perception fusion module for decoding, and the decoded features are added to the features F2 to obtain the first-stage decoding features D1. The 1x1 convolution is performed to obtain the first-stage decoding features D1; then, the decoding features D1, D2, D3, and D4 of each stage are added and subjected to Sigmoid activation to obtain the dose prediction result.

[0057] (2-a-i) The cross perception fusion module of step (2-a) is composed of N cross perception-based hybrid CNN-Transformer structures, denoted as CPF Transformer, and cascaded, as shown in Figure 2 wherein N is a natural number greater than 0, preferably a natural number between 3 and 8, and N = 4 is preferred in this embodiment;

[0058] (2-a-ii) The cross perception-based hybrid CNN-Transformer structure of step (2-a-i) is denoted as CPF Transformer, as shown in Figure 3As shown, the specific structure comprises: constructing two branches, wherein the first branch is connected by one self-attention residual module and two interactive enhancement modules, and the second branch is connected by one multi-scale perception residual module and two interactive enhancement modules; cross connection is introduced between the two branches, and the specific structure is that, for the input feature wherein H1, W1 and C1 represent the height, width and channel number of the input feature map, respectively, first channel separation is performed on to obtain two sub-features and After the sub-features X 0 and X 1 pass through the self-attention residual module and the multi-scale perception residual module, respectively, features A 0 and A 1 are obtained; the features A 0 and X 1 are input into the first interactive enhancement module of the first branch to obtain feature F 0 ; the features A 1 and X 0 are input into the first interactive enhancement module of the second branch to obtain feature F 1 ; the features F 0 and A 1 are input into the second interactive enhancement module of the first branch to obtain feature T 0 ; the features F 1 and A 0 are input into the second interactive enhancement module of the second branch to obtain feature T 1 ; finally, the features T 0 and T 1 are spliced to obtain the output of the CPF Transformer;

[0059] (2-a-ⅲ) The self-attention residual module of step (2-a-ⅱ), the specific structure comprising: for the input feature X 0 , sequentially passing through a normalization layer and a multi-head self-attention structure to obtain feature R 0 , then using a residual connection to add the features X 0 and R 0 to obtain feature A 0 ; the multi-head self-attention structure in this embodiment is preferably a CSWin Transformer structure;

[0060] (2-a-ⅳ) The multi-scale perception residual module of step (2-a-ⅱ), the specific structure comprising: for the input feature X 1 , sequentially passing through a normalization layer and a multi-scale perception fusion structure to obtain feature R

[0061] ​1 Then, the residual connection is used to add the features X 1 and R 1 to obtain the feature

[0062] A 1 ;

[0063] (2-a-v) The multi-scale perception fusion structure described in step (2-a-iv), denoted as MSPF, as shown in Figure 4

[0064] The specific structure includes: inputting the input feature X in into three linear layers respectively to obtain features h, w and p; inputting the feature h into a multi-scale mixed convolution layer to obtain a feature m1, and multiplying the feature m1 by the feature w to obtain a feature f1; inputting the feature f1 into a multi-scale mixed convolution layer to obtain a feature m2, and multiplying the feature m2 by the feature w to obtain a feature f2; inputting the feature f2 into a multi-scale mixed convolution layer to obtain a feature m3, and multiplying the feature m3 by the feature w to obtain a feature f3; inputting the feature f3 into a linear layer to obtain a feature k, and multiplying the feature k by the feature p to obtain

[0065] the output R 1 of the MSPF module;

[0066] (2-a-vi) The multi-scale mixed convolution layer described in step (2-a-v), as shown in Figure 5 The specific structure

[0067] includes: performing channel separation on the input feature , where H2, W2 and C2 represent the height, width and channel number of the input feature q, i.e., the input feature is evenly divided into four groups according to the number of channels to obtain four sub-features

[0068] and inputting the four sub-features into depth separable convolutions with convolution kernels of 3x3, 5x5, 7x7 and 9x9 respectively, and then splicing the obtained features to obtain the output feature q' of the multi-scale mixed convolution layer;

[0069]

[0070] (2-a-vii) The interactive enhancement module described in step (2-a-ii), denoted as IE, as shown in Figure 6 The specific structure includes: the IE module accepts two inputs, i.e., an input feature from a current branch, such as the input feature shown by the solid line in Figure 7 and 7 , and an input feature from another branch, such as the input feature shown by the solid line in Figure 4 and 7 ​The input feature indicated by the dashed line is obtained by first concatenating the input feature C from the current branch and the input feature O from another branch, then inputting the concatenated feature into a normalization layer and a fully connected feedforward network in sequence to obtain feature H, and then adding feature H and C through a residual connection to obtain the output feature;

[0071] (2-a-ⅷ) The gating attention module described in step (2-a), denoted as AG, has a specific structure as shown in the figure, which includes: the AG module accepts two inputs v1 and v2, where v1 represents the output of the up-sampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, i.e. the output of the encoding block in the same stage; inputting v1 and v2 into two parallel branches to obtain features q1 and q2, respectively, each branch contains a 1x1 convolution, a normalization layer and a ReLU activation function connected in sequence; adding features q1 and q2, and performing a 1x1 convolution, a normalization layer and a Sigmoid activation on the added result in sequence to obtain feature z; point-multiplying z with input feature s1 to obtain the output of the gating attention module; Figure 4

[0072] (2-b) Constructing a discriminator network, as shown in the figure, the network is composed of an initial convolutional layer, three intermediate convolutional layers and an output convolutional layer connected in sequence, and the specific structure includes: for an input image Figure 8

[0073]

[0074] where H3, W3 and C3 represent the height, width and channel number of the input feature I, respectively, and the input is input into an initial convolutional layer containing a 4x4 convolution with a stride of 2 and a LeakyReLU activation function to obtain feature Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer and a LeakyReLU activation function, where the stride of the 4x4 convolution of the first two intermediate convolutional layers is 2, and the stride of the 4x4 convolution of the third intermediate convolutional layer is 1; inputting feature

[0075]

[0076]

[0077] ​​​​​​​​(2-c) generating a dose image G output by the CPFTransGenerator generator network and a real dose image R

[0078] The image R is input into the discriminator described in step (2-b) to obtain two-dimensional discrimination matrices J1 and J2 corresponding to the generated dose image G and the real dose image R; a discriminator loss function is constructed, and by dynamically adjusting the loss weights of the real data and the generated data, the gradient direction conflict problem caused by traditional equal-weight summation is solved; specifically, the discriminator loss function is defined as:

[0079]

[0080] wherein L1 and L2 represent the generated data loss and the real data loss, respectively, N1 and N2 represent the total number of pixels in the discrimination matrices J1 and J2, respectively, J2(p) represents the value of the pth pixel in the discrimination matrix J2, and J1(p) represents the value of the pth pixel in the discrimination matrix J1; w1 and w2 are adaptive and dynamically adjusted weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgments of the discriminator on the real and generated dose images, and specifically include: first, the included angle between the gradient directions of the generated result loss L1 and the real dose loss L2 is calculated, and if the included angle exceeds 90°, it indicates that the two gradient directions conflict, and at this time, the weight is dynamically distributed according to the gradient, specifically:

[0081]

[0082] wherein ε is a small constant value, used to prevent the denominator from being zero or the weight from being too small, and the value range of ε is (0.001, 0.1), and preferably ε = 0.01 in this embodiment, and this weight distribution method can make the total gradient direction orthogonal to the conflicting item or If the gradient directions of L1 and L2 are consistent, the weights w1 and w2 are adjusted in combination with the judgment of the discriminator on the real and generated two types of samples, when the discriminator excessively trusts the real samples, that is, the probability of the discriminator judging the real samples as true is close to 1, then the weight w1 of the generated samples is increased to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, then the weight w2 of the real samples is increased to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; the main reason for this approach is that when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as true is close to 1, it indicates that the generated samples are highly realistic, and the discriminator has lost its effective countermeasures, and if the learning of the generated samples is continued to be strengthened, the risk of gradient disappearance and mode collapse will be aggravated, therefore, at this time, the weight of the real samples needs to be increased to further improve the discrimination ability of the discriminator;

[0083] (2-d) Construct a multi-scale cross-window self-attention encoding network, denoted as MCENet, as follows: Figure 9 As shown, the network input includes the generated dose map and the true dose map. The MCENet network is a four-stage encoding network. Its specific structure includes: inputting the generated dose G and the true dose R into an encoding block containing a CSWin self-attention module and a max-pooling layer with a stride of 2, respectively, to obtain intermediate features g1 and r1. The mean absolute error is used to calculate the loss on the intermediate features g1 and r1, resulting in the loss of the generated dose map G and the true dose map R in the first stage. Where g1(j) and r1(j) represent the j-th pixel in feature maps g1 and r1, respectively, and T1 represents the total number of pixels in feature map g1. Features g1 and r1 are input into an encoding block containing a CSWin self-attention module and a max-pooling layer with a stride of 2, respectively, to obtain intermediate features g2 and r2. The mean absolute error is used to calculate the loss of intermediate features g2 and r2, resulting in the loss of the generated dose map G and the true dose map R in the second stage. Where g2(j) and r2(j) represent the j-th pixel in feature maps g2 and r2, respectively, and T2 represents the total number of pixels in feature map g2. Features g2 and r2 are input into an encoding block containing a CSWin self-attention module and a max-pooling layer with a stride of 2, respectively, to obtain intermediate features g3 and r3. The mean absolute error is used to calculate the loss of intermediate features g3 and r3, resulting in the loss of the generated dose map G and the true dose map R in the third stage. Where g3(j) and r3(j) represent the j-th pixel in feature maps g3 and r3, respectively, and T3 represents the total number of pixels in feature map g3. Features g3 and r3 are input into an encoding block containing a CSWin self-attention module and a max-pooling layer with a stride of 2, respectively, to obtain intermediate features g4 and r4. The mean absolute error is used to calculate the loss of intermediate features g4 and r4, resulting in the loss of the generated dose map G and the true dose map R in the fourth stage. Where g4(j) and r4(j) represent the j-th pixel in feature maps g4 and r4, respectively, and T4 represents the total number of pixels in feature map g4; S1, S2, S3, and S4 are weighted and summed, and the result is used as the loss of the multi-scale cross-window self-attention encoding network to constrain the multi-scale difference between the generated dose map and the real dose map; specifically, the MCENet network loss function L mcs Defined as:

[0084]

[0085] Where, λ i Let g1(j) and r1(j) represent the weight coefficients of the loss in the i-th stage, and g1(j) and r1(j) represent the intermediate feature maps g generated in the i-th stage, respectively. i and ri the jth pixel in the image, T i denotes the feature map g i the total number of pixels.

[0086] (3) According to the discriminator and MCENet network loss, the loss function L of the CPFTransGAN network is constructed:

[0087]

[0088] wherein G(i) and R(i) represent the values of the ith pixel of the generated dose map G and the real dose map R respectively, N3 is the total number of pixels of the generated dose map G, L G denotes the loss obtained by calculating the mean absolute error between the measured dose map G and the real dose map R, denotes the adaptive weighted loss calculated by the discriminator, L mcs denotes the loss calculated by the multi-scale cross window self-attention network;

[0089] (4) The CPFTransGAN network is trained using the training data set B until the loss function L converges, wherein the CT image, the planned target volume and the organ at risk in the training data set B are spliced as the network input, and the real dose distribution map is used for loss calculation;

[0090] (5) The trained network model is used to predict the test data, wherein the test data contains patient CT image, planned target volume and organ at risk map, and the output of CPFTransGenerator generator is the dose prediction result.

[0091] Example 2

[0092] The method in Example 1 is used to conduct experiments on the public head and neck cancer data set of OpenKBP-2020AAPM Grand Challenge. The data set contains 340 head and neck cancer patient data, each patient data contains patient CT image, planned target volume map, organ at risk map and real dose distribution map, and the image size is 128x128x128. In the experiment, the data is divided into test data set and training data set according to the ratio of 2:8.

[0093] Figure 10 Fig. 6 shows some dose prediction result examples obtained on the test set. Figure 11 (a) ~ Figure 11 (c) are three patient original CT images randomly selected from the test data, Figure 11 (d) ~ Figure 11 (f) are dose prediction results obtained by using the method in Example 1 on Figure 11 (a) ~ Figure 11(c) the CT image shown in the test, the dose prediction results map, Figure 11 (g)~ Figure 11 (i) is Figure 11 (a)~ Figure 11 Figure 11 (c) the true dose distribution map of the CT image shown, it can be seen that the dose prediction results obtained by the method of the present application are very close to the true dose distribution.

[0094] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-sense fusion-based head and neck cancer radiotherapy dose prediction method, characterized in that, The method comprises the following steps: (1) establishing a training data set B comprising a patient CT image, a planned target volume, an organ at risk and a true dose distribution map; (2) constructing a CPFTransGAN based on cross-perception fusion, specifically comprising: (2-a) constructing a generator network composed of four-stage encoding-decoding structure, denoted as CPFTransGenerator, the specific structure is: the encoder contains four stages, each stage is composed of an embedding module and a cross perception fusion module connected in turn, the embedding module converts the size of the input feature to the specified size through the learnable embedding layer, that is, for the input feature with the size of HxWxC, where H, W and C represent the height, width and channel number of the input feature respectively, after the embedding module, the size is converted to The cross perception fusion module is used as an encoding block to extract features from the feature map output by the embedding module; the decoder also contains four stages, each stage uses a cross perception fusion module as a decoding block for decoding, the specific structure includes: in the fourth stage of the decoder, the decoding block uses the output of the encoding block in the same stage as the input for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F4, and at the same time is sequentially subjected to 1x1 convolution and up-sampling with a step size of 8 to obtain the fourth-stage decoding feature D4; in the third stage of the decoder, the gating attention module takes the feature F4 and the output of the encoding block in the same stage as the input for feature fusion, and the fused feature is added to F4, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F3, and at the same time is sequentially subjected to 1x1 convolution and up-sampling with a step size of 4 to obtain the third-stage decoding feature D3; in the second stage of the decoder, the gating attention module takes the feature F3 and the output of the encoding block in the same stage as the input for feature fusion, and the fused feature is added to F3, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the up-sampling convolution module with a step size of 2 to obtain the feature F2, and at the same time is sequentially subjected to 1x1 convolution and up-sampling with a step size of 2 to obtain the second-stage decoding feature D2; in the first stage of the decoder, the gating attention module takes the feature F2 and the output of the encoding block in the same stage as the input for feature fusion, and the fused feature is added to F2, and the added feature is input into the cross perception fusion module for decoding, and the decoded feature is input into the 1x1 convolution to obtain the first-stage decoding feature D1; then, the decoding features D1, D2, D3 and D4 of each stage are added and subjected to Sigmoid activation to obtain the dose prediction result; (2-a-i) the cross-perception fusion module in step (2-a) is composed of N cross-perception-based hybrid CNN-Transformer structures, denoted as CPF Transformer, in cascade, wherein N is a natural number greater than 0; (2-a-ii) The cross perception based hybrid CNN-Transformer structure of step (2-a-i) is recorded as CPFTransformer, and the specific structure comprises: two branches are constructed, wherein the first branch is composed of one self-attention residual module and two interactive enhancement modules, and the second branch is composed of one multi-scale perception residual module and two interactive enhancement modules; cross connection is introduced between the two branches, and the specific structure is that, for the input feature wherein H1, W1 and C1 represent the height, width and channel number of the input feature map respectively, firstly, channel separation is performed on to obtain two sub-features and After the sub-features X 0 and X 1 respectively pass through the self-attention residual module and the multi-scale perception residual module, the features A 0 and A 1 are obtained; the features A 0 and X 1 are input into the first interactive enhancement module of the first branch to obtain the feature F 0 ; the features A 1 and X 0 are input into the first interactive enhancement module of the second branch to obtain the feature F 1 ; the features F 0 and A 1 are input into the second interactive enhancement module of the first branch to obtain the feature T 0 ; the features F 1 and A 0 are input into the second interactive enhancement module of the second branch to obtain the feature T 1 ; finally, the features T 0 and T 1 are spliced to obtain the output of the CPFTransformer; (2-a-iii) the self-attention residual module of step (2-a-ii), and the specific structure comprises: for input feature X 0 , sequentially passing through a normalization layer and a multi-head self-attention structure to obtain feature R 0 , then using a residual connection to add features X 0 and R 0 to obtain feature A 0 ; (2-a-iv) the multi-scale perception residual module of step (2-a-ii), and the specific structure comprises: for input feature X 1 , sequentially passing through a normalization layer and a multi-scale perception fusion structure to obtain feature R 1 , then using a residual connection to add features X 1 and R 1 to obtain feature A 1 ; (2-a-Ⅴ) The multi-scale perception fusion structure described in step (2-a-Ⅳ) is denoted as MSPF, and the specific structure comprises: inputting the input feature X in into three linear layers respectively to obtain features h, w and p; inputting the feature h into a multi-scale mixed convolution layer to obtain a feature m1, and multiplying the feature m1 by the feature w to obtain a feature f1; inputting the feature f1 into a multi-scale mixed convolution layer to obtain a feature m2, and multiplying the feature m2 by the feature w to obtain a feature f2; inputting the feature f2 into a multi-scale mixed convolution layer to obtain a feature m3, and multiplying the feature m3 by the feature w to obtain a feature f3; inputting the feature f3 into a linear layer to obtain a feature k, and multiplying the feature k by the feature p to obtain an output R of the MSPF module 1 ; (2-a-Ⅴ) The multi-scale mixed convolutional layer described in step (2-a-Ⅳ) has the following specific structure: the input feature Channel separation is performed, where H2, W2, and C2 represent the height, width, and channel number of the input feature q, that is, the input feature is evenly divided into four groups according to the number of channels, and four sub-features are obtained and The four sub-features are respectively input into depth separable convolutions with convolution kernels of 3×3, 5×5, 7×7, and 9×9, and then the obtained features are spliced to obtain the output feature q' of the multi-scale mixed convolutional layer. (2-a-ⅶ) the interactive enhancement module in step (2-a-ⅱ), denoted as IE, is characterized in that the specific structure comprises: the IE module accepts two inputs, namely the input features from the current branch and the input features from another branch; first, the input features C from the current branch and the input features O from another branch are spliced, and then the spliced features are sequentially input into a normalization layer and a fully connected feedforward network to obtain features H, and then the features H and C are added through a residual connection to obtain output features; (2-b) constructing a discriminator network, which is sequentially connected by an initial convolutional layer, three intermediate convolutional layers and an output convolutional layer, and the specific structure includes: for the input image wherein H3, W3, C3 represent the height, width and channel number of the input feature I respectively, and inputting an initial convolutional layer containing a 4x4 convolution with a step of 2 and a LeakyReLU activation function to obtain a feature inputting the feature t1 into three intermediate convolutional layers in sequence to obtain features Each intermediate convolutional layer contains a 4x4 convolution, a normalization layer and a LeakyReLU activation function, wherein the step of the 4x4 convolution of the first two intermediate convolutional layers is 2, and the step of the 4x4 convolution of the third intermediate convolutional layer is 1; inputting the features inputting an output convolutional layer containing a 4x4 convolution with a step of 1 and a Sigmoid activation function to obtain a two-dimensional discrimination matrix related to the size of the input image (2-c) inputting the generated dose image G and the real dose image R output by the CPFTransGenerator generator network into the discriminator described in step (2-b) respectively to obtain the two-dimensional discrimination matrix J1 corresponding to the generated dose image G and the two-dimensional discrimination matrix J2 corresponding to the real dose image R; constructing a discriminator loss function, which is specifically defined as: Wherein, L1 and L2 represent the generated data loss and the real data loss respectively, N1 and N2 represent the total number of pixels of the discrimination matrices J1 and J2 respectively, J2(p) represents the value of the pth pixel in the discrimination matrix J2, J1(p) represents the value of the pth pixel in the discrimination matrix J1; w1 and w2 are adaptive dynamic adjustment weight coefficients, which are adjusted by analyzing whether the gradient directions of L1 and L2 conflict and combining the true and false judgment of the discriminator on the real and generated dose maps, specifically including: first, calculate the included angle of the gradient directions of the generated result loss L1 and the real dose loss L2, if the included angle exceeds 90°, it indicates that the two gradient directions conflict, at this time, the weight is dynamically allocated according to the gradient, specifically: wherein ε is a small constant value, used to prevent the denominator from being zero or the weight being too small, and the value range of ε is (0.001, 0.1), and this weight distribution method makes the total gradient direction orthogonal to the conflict item or , avoiding interference of reverse optimization; if the gradient directions of L1 and L2 are consistent, the weights w1 and w2 are adjusted in combination with the judgment of the discriminator on the real and generated samples, when the discriminator excessively trusts the real samples, that is, the probability of the discriminator judging the real samples as real is close to 1, the weight w1 of the generated samples is increased to strengthen the learning of the generated samples; when the discriminator cannot distinguish the generated samples, that is, the probability of the generated samples being judged as real is close to 1, the weight w2 of the real samples is increased to strengthen the learning of the real samples and improve the discrimination ability of the discriminator; (2-d) Construct a multi-scale cross window self-attention encoding network, denoted as MCENet, the network input contains the generated dose map and the real dose map, MCENet network is a four-stage encoding network, the specific structure includes: the generated dose G and the real dose R are respectively input into an encoding block containing CSWin self-attention module and maximum pooling layer with step 2, to obtain intermediate features g1 and r1, and the mean absolute error is used to calculate the loss of the intermediate features g1 and r1, to obtain the loss of the generated dose map G and the real dose map R in the first stage Where g1(j) and r1(j) represent the jth pixel in the feature maps g1 and r1 respectively, T1 represents the total number of pixels in the feature map g1; the features g1 and r1 are respectively input into an encoding block containing CSWin self-attention module and maximum pooling layer with step 2, to obtain intermediate features g2 and r2, and the mean absolute error is used to calculate the loss of the intermediate features g2 and r2, to obtain the loss of the generated dose map G and the real dose map R in the second stage Where g2(j) and r2(j) represent the jth pixel in the feature maps g2 and r2 respectively, T2 represents the total number of pixels in the feature map g2; the features g2 and r2 are respectively input into an encoding block containing CSWin self-attention module and maximum pooling layer with step 2, to obtain intermediate features g3 and r3, and the mean absolute error is used to calculate the loss of the intermediate features g3 and r3, to obtain the loss of the generated dose map G and the real dose map R in the third stage Where g3(j) and r3(j) represent the jth pixel in the feature maps g3 and r3 respectively, T3 represents the total number of pixels in the feature map g3; the features g3 and r3 are respectively input into an encoding block containing CSWin self-attention module and maximum pooling layer with step 2, to obtain intermediate features g4 and r4, and the mean absolute error is used to calculate the loss of the intermediate features g4 and r4, to obtain the loss of the generated dose map G and the real dose map R in the fourth stage Where g4(j) and r4(j) represent the jth pixel in the feature maps g4 and r4 respectively, T4 represents the total number of pixels in the feature map g4; S1, S2, S3 and S4 are weighted and added, and the result is used as the loss of the multi-scale cross window self-attention encoding network, to constrain the multi-scale difference between the generated dose map and the real dose map; specifically, the MCENet network loss function L mcs is defined as: wherein λ i is a weight coefficient of the loss of the i-th stage, g1(j) and r1(j) represent the j-th pixel in the intermediate feature map g i and r i generated by the i-th stage, respectively, T i represents the total number of pixels of the feature map g i . (3) constructing the loss function L of the CPFTransGAN network according to the discriminator and the MCENet network loss: where G(i) and R(i) represent the value of the i-th pixel of the generated dose map G and the real dose map R, respectively, N3is the total number of pixels of the generated dose map G, L G represents the loss obtained by calculating the mean absolute error between the measured dose map G and the real dose map R, represents the adaptive weighted loss calculated by the discriminator, L mcs represents the loss calculated by the multi-scale cross-window self-attention network; (4) training the CPFTransGAN network using the training data set B until the loss function L converges, wherein the CT image, the planned target volume and the organ at risk in the training data set B are spliced as the network input, and the true dose distribution map is used for loss calculation; (5) using the trained network model to predict the test data, wherein the test data comprises a patient CT image, a planned target volume and an organ at risk map, and the output of the CPFTransGenerator generator is the dose prediction result.

2. The head and neck cancer radiotherapy dose prediction method based on cross-sense fusion of claim 1, wherein, The gating attention module described in step (2-a) is denoted as AG, and the specific structure includes: the AG module accepts two inputs v1 and v2, where v1 represents the output of the upsampling layer connected to the current AG module, and v2 represents the output of the encoding block connected to the current AG module, that is, the output of the encoding block of the same stage; input v1 and v2 into two parallel branches respectively to obtain features q1 and q2, each branch contains a 1x1 convolution, a normalization layer and a ReLU activation function connected in turn; add features q1 and q2, and perform 1x1 convolution, normalization layer and Sigmoid activation on the added result in turn to obtain feature z; point multiply input feature s1 with z to obtain the output of the gating attention module.

3. The head and neck cancer radiotherapy dose prediction method based on cross-sense fusion of claim 1, wherein, The multi-head self-attention structure described in step (2-a-ⅲ) is a CSWin Transformer structure.

Images