Dose prediction method for nasopharyngeal carcinoma radiotherapy planning based on progressive feature fusion network

Abstract

The present invention provides a method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network. The method acquires image data, which includes CT images, annotation information of target areas and organs at risk, and dose distribution of radiotherapy plan, and processes the image data into an input image of a preset size; inputs the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model; inputs a medical image to be identified into the target dose prediction model and outputs a dose prediction result of nasopharyngeal carcinoma radiotherapy plan. Specifically, progressive feature fusion and a pixel attention guided fusion strategy based on wavelet convolution are introduced into the dose prediction model, which can effectively extract discriminative semantic features and enhance the semantic discrimination ability of the network model. At the same time, it also effectively fuses deep-level semantic discrimination information and shallow-level spatial positioning details, and enhances the multi-level information fusion ability of the network model.

Description

Technical Field

[0001] The present invention belongs to the field of medical technology, and in particular relates to a method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network. Background Art

[0002] Radiotherapy (RT) is one of the important means of cancer treatment, using high-energy radiation to control the spread of cancer cells and kill malignant cells. In recent years, with the introduction of intensity modulated radiation therapy (IMRT) and volumetric modulated arc therapy (VMAT), the quality of treatment plans has been significantly improved. However, this has led to a significant increase in the complexity of treatment planning for complex areas, which usually requires multiple attempts and a lot of time to optimize the quality of the plan.

[0003] The automatic planning design (KBP) method based on prior knowledge is a solution. The above methods mainly rely on the patient's historical treatment plan data to automatically create dose distribution plans for new clinical treatments. Dose volume histogram (DVH) is a commonly used clinical evaluation indicator in the KBP method, which helps to formulate automated radiotherapy plans. However, the KBP method still has some limitations. First, they are highly dependent on manually designed features related to dose distribution, feature extraction is time-consuming, and the accuracy is insufficient. Second, due to the compression and asymmetric mapping of three-dimensional (3D) dose information, the features extracted from DVH lack detailed information on spatial dose distribution. In addition, the above methods fail to consider the anatomical structure and position information around the tumor, ignoring the influence of surrounding organs.

[0004] In recent years, convolutional neural networks (CNNs) have made significant breakthroughs in the field of medical image analysis and have been widely used in the field of radiotherapy dose prediction. Accurate and fast dose prediction results make automatic planning based on dose prediction possible. Among them, 3D UNet and its variants are representative radiotherapy dose prediction models.

[0005] Since the dose prediction of nasopharyngeal carcinoma radiotherapy plan is a voxel-level dense regression problem, it requires both deep semantic discriminant features in the deep neural network model and shallow spatial positioning details. The existing methods can constrain the network model to a certain extent through a simple attention mechanism, so that it can focus on the extraction of features related to dose prediction, and perform preliminary multi-level information fusion of semantic discriminant features and spatial location details through a hierarchical densely connected network structure. However, the existing methods do not consider how to further enhance the discriminant feature extraction ability of the dose prediction network model, resulting in the inability of the existing methods to accurately understand the deep semantic information of medical images, such as the accurate distinction between tumors and normal tissues; moreover, the existing dose prediction network model does not fully integrate the deep semantic discriminant features and spatial location details in medical images, resulting in the inability of the existing methods to accurately understand the anatomical structure around the tumor area; in addition, the existing methods cannot distinguish and locate the areas with difficult dose prediction and the areas with simple dose prediction, resulting in the lack of clinical relevance of the dose prediction results; finally, the main purpose of most existing methods is to improve the accuracy of dose prediction, but ignores the quantitative study of how dose prediction affects the plan quality, and accurate plan quality assessment is crucial for high-quality plan design. Summary of the invention

[0006] Based on this, an embodiment of the present invention provides a method and system for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network, aiming to solve at least one of the above problems.

[0007] A first aspect of an embodiment of the present invention provides a method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network, the method comprising:

[0008] Acquiring image data, the image data including CT images, target volume and organ at risk annotation information, and dose distribution of a radiotherapy plan, and processing the image data into an input image of a preset size;

[0009] Inputting the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model;

[0010] Inputting the medical image to be identified into the target dose prediction model, and outputting the dose prediction result of the nasopharyngeal carcinoma radiotherapy plan;

[0011] The dose prediction model is composed of two progressive UNet architectures, wherein the first UNet architecture is used for preliminary feature extraction, and then the features extracted by the four upsampling layers are converted into attention weights through the channel attention module, and the attention weights are applied to the second UNet architecture, the first UNet architecture is a four-layer structure, and the second UNet architecture is a five-layer structure;

[0012] The number of channels of each layer of the second UNet architecture is twice that of the corresponding layer of the first UNet architecture, and the input of each layer of the second UNet architecture includes features from the previous layer and corresponding features generated by the upsampling process of the first UNet architecture;

[0013] In addition, a pixel attention guided fusion module based on wavelet convolution is set between the third layer and the fourth layer in the downsampling process of the second UNet architecture. The encoders of the first UNet architecture and the second UNet architecture both use a convolution layer with a step size of 2 to replace the maximum pooling layer for downsampling operations. The encoder of the first UNet architecture uses a standard 3D convolution module, and the encoder of the second UNet architecture uses a residual dense block.

[0014] In the decoders of the first UNet architecture and the second UNet architecture, upsampling uses trilinear interpolation, and feature fusion is achieved through a residual module, and finally a 1×1×1 convolutional layer is used to generate the final nasopharyngeal carcinoma radiotherapy plan dose prediction result.

[0015] Furthermore, the channel attention module is used to output a part of the input after passing through the first convolution layer, the global pooling layer, the second convolution layer, the third convolution layer and the Sigmoid activation function in sequence, and then multiply it with another part of the input for final output, wherein the first convolution layer includes a convolution kernel and an activation function of size 3×3×3, the second convolution layer includes a convolution kernel and an activation function of size 1×1×1, and the third convolution layer includes a convolution kernel of size 1×1×1.

[0016] Furthermore, the pixel attention guided fusion module based on wavelet convolution is used to complete wavelet convolution and normalization processing on the third layer feature image of the second UNet architecture branch, and adjust the feature map size through convolution operation, and at the same time complete wavelet convolution and normalization processing on the fourth layer feature image of the first UNet architecture branch. In addition, the pixel attention guided fusion module based on wavelet convolution obtains the selective weight factor through the Sigmoid function calculation of two branch vectors.

[0017] Furthermore, the first convolution operation of the two-branch feature maps in the wavelet convolution-based pixel attention guided fusion module is set to wavelet convolution. The wavelet convolution adds the result image after the wavelet convolution kernel and the inverse wavelet convolution kernel to the result image of the direct convolution operation, so as to increase the feature receptive field size exponentially while linearly increasing the training parameters.

[0018] Furthermore, the residual dense block includes two branches, the upper branch contains three convolution modules, each convolution module concatenates the original features with the transformed features generated by the current convolution operation, and each convolution module extracts local features through 3×3×3 convolution; the lower branch extracts local features through 1×1×1 convolution; the outputs of the two branches are fused by element-by-element addition, and finally a 1×1×1 convolution is performed.

[0019] Furthermore, the residual module includes two convolution groups, each convolution group is composed of a series of convolution layers, and the convolution layer is a 3×3×3 convolution kernel, wherein a part of the input passes through the first convolution group and is added to another part of the input to obtain a first output, and a part of the first output passes through the second convolution group and is multiplied with another part of the first output to obtain a second output.

[0020] Furthermore, in the step of inputting the input image into the dose prediction model to train the dose prediction model and determine the target dose prediction model, the total loss function used in the training is composed of the mean absolute error loss, the difficult area learning loss and the auxiliary segmentation loss, which is expressed as:

[0021]

[0022] in, is the total loss function, is the mean absolute error loss, Learn the loss for the hard region, is the weight used to adjust the auxiliary segmentation loss, is the auxiliary segmentation loss.

[0023] A second aspect of an embodiment of the present invention provides a nasopharyngeal carcinoma radiotherapy plan dose prediction system based on a progressive feature fusion network, which is used to implement the nasopharyngeal carcinoma radiotherapy plan dose prediction method based on a progressive feature fusion network provided in the first aspect, and the system includes:

[0024] An acquisition module, configured to acquire image data, the image data including CT images, target volume and organ at risk annotation information, and dose distribution of a radiotherapy plan, and process the image data into an input image of a preset size;

[0025] A training module, used for inputting the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model;

[0026] An input module, used to input the medical image to be identified into the target dose prediction model, and output the dose prediction result of the nasopharyngeal carcinoma radiotherapy plan;

[0027] The dose prediction model is composed of two progressive UNet architectures, wherein the first UNet architecture is used for preliminary feature extraction, and then the features extracted by the four upsampling layers are converted into attention weights through the channel attention module, and the attention weights are applied to the second UNet architecture, the first UNet architecture is a four-layer structure, and the second UNet architecture is a five-layer structure;

[0028] The number of channels of each layer of the second UNet architecture is twice that of the corresponding layer of the first UNet architecture, and the input of each layer of the second UNet architecture includes features from the previous layer and corresponding features generated by the upsampling process of the first UNet architecture;

[0029] In addition, a pixel attention guided fusion module based on wavelet convolution is set between the third layer and the fourth layer in the downsampling process of the second UNet architecture. The encoders of the first UNet architecture and the second UNet architecture both use a convolution layer with a step size of 2 to replace the maximum pooling layer for downsampling operations. The encoder of the first UNet architecture uses a standard 3D convolution module, and the encoder of the second UNet architecture uses a residual dense block.

[0030] In the decoders of the first UNet architecture and the second UNet architecture, upsampling uses trilinear interpolation, and feature fusion is achieved through a residual module, and finally a 1×1×1 convolutional layer is used to generate the final nasopharyngeal carcinoma radiotherapy plan dose prediction result.

[0031] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network provided in the first aspect.

[0032] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network provided in the first aspect is implemented.

[0033] A method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network is provided in an embodiment of the present invention. The method acquires image data, which includes CT images, annotation information of target areas and organs at risk, and dose distribution of radiotherapy plans, and processes the image data into an input image of a preset size; inputs the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model; inputs the medical image to be identified into the target dose prediction model and outputs a dose prediction result of nasopharyngeal carcinoma radiotherapy plan. Specifically, progressive feature fusion and a pixel attention guided fusion strategy based on wavelet convolution are introduced into the dose prediction model, which can effectively extract discriminative semantic features and enhance the semantic distinction ability of the network model. At the same time, it can also effectively fuse deep-level semantic discrimination information and shallow-level spatial positioning details to enhance the multi-level information fusion ability of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network provided in the first embodiment of the present invention;

[0035] Figure 2 Schematic diagram of the structure of two progressive UNet architectures for dose prediction models;

[0036] Figure 3 It is a schematic diagram of the structure of the channel attention module;

[0037] Figure 4 The schematic diagram of the structure of the pixel attention guided fusion module based on wavelet convolution;

[0038] Figure 5 This is a schematic diagram of the principle of wavelet convolution performed only once;

[0039] Figure 6 Schematic diagram of the structure of the residual dense block;

[0040] Figure 7 It is a structural diagram of the residual module;

[0041] Figure 8 Schematic diagram of the structure of learning loss for difficult regions;

[0042] Fig. 9 Dose distribution diagram;

[0043] Fig.10 is a high-dose area map;

[0044] Fig.11 Schematic diagram of dose distribution of the true value (GT) and the predicted result;

[0045] Fig.12Dose-volume histogram (DVH) of the true value (GT) and predicted result of PTV55;

[0046] Fig.13 Dose-volume histogram (DVH) of the true value (GT) of PTV60 and the predicted result;

[0047] Fig.14 Dose-volume histogram (DVH) of the true value (GT) of PTV70 and the predicted result;

[0048] Fig.15 A structural block diagram of a nasopharyngeal carcinoma radiotherapy plan dose prediction system based on a progressive feature fusion network provided in Embodiment 3 of the present invention;

[0049] Fig.16 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0050] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0051] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0053] Embodiment 1

[0054] See also Figure 1 , Figure 1 A flowchart of an implementation of a method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network provided in Embodiment 1 of the present invention is shown. The method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network specifically includes steps S01 to S03.

[0055] Step S01 : acquiring image data, the image data including CT images, target volume and organ at risk annotation information, and dose distribution of radiotherapy plan, and processing the image data into an input image of a preset size.

[0056] In this embodiment, the CT images and dose distribution data of each patient are rescaled to 96 × 128 × 128. Subsequently, the CT values ​​are truncated to the range of (-1024, 2048) and normalized to (0, 1), and finally all values ​​in the dose distribution data are divided by 70 Gray.

[0057] Step S02: input the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model.

[0058] See also Figure 2 , which is a schematic diagram of the structure of two progressive UNet architectures of the dose prediction model. Specifically, the first UNet architecture is used for preliminary feature extraction, and then the features extracted from the four upsampling layers are converted into attention weights through the channel attention module, and the attention weights are applied to the second UNet architecture. The first UNet architecture is a four-layer structure, and the second UNet architecture is a five-layer structure. For more details, please refer to Figure 3 , is a schematic diagram of the structure of the channel attention module, which is used to output a part of the input after passing through the first convolution layer, the global pooling layer, the second convolution layer, the third convolution layer and the Sigmoid activation function in sequence, and then multiply it with another part of the input for final output, wherein the first convolution layer includes a convolution kernel of size 3×3×3 and an activation function, the second convolution layer includes a convolution kernel of size 1×1×1 and an activation function, and the third convolution layer includes a convolution kernel of size 1×1×1;

[0059] The number of channels in each layer of the second UNet architecture is twice that of the corresponding layer of the first UNet architecture. The input of each layer of the second UNet architecture includes the features from the previous layer and the corresponding features generated by the upsampling process of the first UNet architecture.

[0060] In addition, in order to ensure the effective fusion of context information and spatial detail information, and avoid the problem of image boundary pixels being corroded by surrounding pixels due to larger spatial detail information covering smaller spatial context information, a pixel attention guidance fusion module based on wavelet convolution is set between the third and fourth layers in the downsampling process of the second UNet architecture to help the fourth layer of the first UNet architecture selectively learn the third layer detail information of the second UNet architecture. It can be understood that the dose prediction model of the embodiment of the present invention achieves more effective feature extraction and fusion in both network depth and width. Please refer to Figure 4, which is a structural diagram of a pixel attention guided fusion module based on wavelet convolution, wherein the pixel attention guided fusion module based on wavelet convolution is used to complete wavelet convolution and normalization processing on the third layer feature image of the second UNet architecture, and adjust the size of the feature map through convolution operation, and at the same time complete wavelet convolution and normalization processing on the fourth layer feature image of the first UNet architecture. In addition, the pixel attention guided fusion module based on wavelet convolution calculates the selective weight factor through the Sigmoid function through two branch vectors. It should be noted that the larger the selective weight factor, the more the module output trusts the third layer of the second UNet architecture, and vice versa, the more it trusts the fourth layer of the first UNet architecture.

[0061] Furthermore, the first convolution operation of the two-branch feature map in the wavelet convolution-based pixel attention guided fusion module is set to wavelet convolution, which adds the result image after the wavelet convolution kernel and the inverse wavelet convolution kernel to the result image of the direct convolution operation, so as to increase the feature receptive field size exponentially while linearly increasing the training parameters, thereby more effectively extracting features with stronger semantic discrimination ability in the input medical image. Figure 5 , which is a schematic diagram of the principle of wavelet convolution performed only once.

[0062] Furthermore, the encoders of the first and second UNet architectures use convolutional layers with a stride of 2 instead of the maximum pooling layer for downsampling. The encoder of the first UNet architecture uses a standard 3D convolutional module, while the encoder of the second UNet architecture uses a residual dense block to enhance feature extraction capabilities. Figure 6 , is a schematic diagram of the structure of the residual dense block. It should be noted that the residual dense block consists of two branches. The upper branch contains three convolution modules. Each convolution module concatenates the original features with the transformed features generated by the current convolution operation. Each convolution module extracts local features through 3×3×3 convolution; the lower branch extracts local features through 1×1×1 convolution; the outputs of the two branches are fused through element-by-element addition, and finally a 1×1×1 convolution is performed. In this way, the global features of the upper branch are integrated with the local features of the lower branch, and all convolution operations are followed by instance normalization and ReLU activation functions.

[0063] In the decoder of the first UNet architecture and the second UNet architecture, upsampling uses trilinear interpolation, and feature fusion is achieved through the residual module. Finally, a 1×1×1 convolutional layer is used to generate the final NPC radiotherapy plan dose prediction result. Please refer to Figure 7, is a structural diagram of the residual module. Specifically, the residual module includes two convolution groups, each convolution group is composed of a series of convolution layers, and the convolution layer is a 3×3×3 convolution kernel, wherein a part of the input passes through the first convolution group and is added to another part of the input to obtain a first output, and a part of the first output passes through the second convolution group and is multiplied with another part of the first output to obtain a second output.

[0064] To facilitate the understanding of the scheme, each processing link after the input image is input into the dose prediction model will be introduced one by one according to the processing steps:

[0065] 1. The input data includes CT images, target volume and organs at risk (OARs) annotation information, and the dose distribution of the radiotherapy plan. The data is processed according to the previously set size, and the final input reference image size is (Dr, Hr, Wr).

[0066] 2. The image is first subjected to preliminary feature extraction through the first UNet architecture. The four-layer downsampling structure of the first UNet architecture uses a convolution layer with a stride of 2 to replace the traditional maximum pooling layer for downsampling operations. Specifically, a standard 3D convolution module is used to extract features in both spatial and temporal dimensions to better capture the local information of the input data. The output sizes of the four downsamplings are (Dr / 2, Hr / 2, Wr / 2), (Dr / 4, Hr / 4, Wr / 4), (Dr / 8, Hr / 8, Wr / 8), and (Dr / 16, Hr / 16, Wr / 16).

[0067] 3. The first UNet architecture four-layer upsampling structure model is implemented by bilinear interpolation, and the feature fusion of the corresponding stages of downsampling and upsampling is realized through the residual network. In this way, the decoder can gradually restore the spatial resolution of the features. During the upsampling process, the size of the feature map is restored to (Dr / 8, Hr / 8, Wr / 8), (Dr / 4, Hr / 4, Wr / 4), (Dr / 2, Hr / 2, Wr / 2).

[0068] 4. The channel attention module extracts features from the four upsampling layers of the first UNet and converts these features into attention weights , , and (Bottom-to-top order). The extracted attention weights are then applied to the second UNet architecture to guide its more accurate feature extraction and fusion.

[0069] 5. The image features after the first UNet architecture are taken together with the image features after the first downsampling as the input of the second UNet architecture. The second UNet architecture has a five-layer structure, and the number of channels in each layer is twice the number of channels in the corresponding layer of the first UNet architecture. Its input includes the attention weight features extracted from the first UNet architecture ( , , and ) and features from the previous layer output. The second UNet architecture uses a convolutional layer with a stride of 2 to replace the traditional maximum pooling layer for downsampling. It uses residual dense blocks to enhance the feature transfer and fusion capabilities through skip connections and dense connections. The sizes of the five downsampling output results are (Dr / 2, Hr / 2, Wr / 2), (Dr / 4, Hr / 4, Wr / 4), (Dr / 8, Hr / 8, Wr / 8), (Dr / 16, Hr / 16, Wr / 16), and (Dr / 32, Hr / 32, Wr / 32).

[0070] 6. The pixel attention guided fusion module based on wavelet convolution is used to guide the results of the third downsampling of the second UNet architecture and the first UNet architecture. After processing, the output is then sent to the second UNet architecture for the fourth downsampling.

[0071] 7. The five-layer upsampling of the second UNet architecture is also achieved through bilinear interpolation, and the feature fusion of the corresponding stages of downsampling and upsampling is achieved through the residual network. During the upsampling process, the size of the feature map is restored to (Dr / 16, Hr / 16, Wr / 16), (Dr / 8, Hr / 8, Wr / 8), (Dr / 4, Hr / 4, Wr / 4), (Dr / 2, Hr / 2, Wr / 2).

[0072] 8. The residual dense block combines the residual connection of ResNet and the dense connection of DenseNet. It consists of two branches. The upper branch contains three convolution modules, each of which concatenates the original features with the transformed features generated by the current convolution operation; the lower branch extracts local features through 1×1×1 convolution; the outputs of the two branches are fused through element-by-element addition, and finally 1×1×1 convolution is performed. In this way, the global features of the upper branch are integrated with the local features of the lower branch.

[0073] 9. The pixel attention guided fusion module based on wavelet convolution assists the fourth layer of the first UNet architecture branch to selectively learn key semantic discriminative features from the output of the third layer of the second UNet architecture branch, and merges the selected features with the output features of the third layer of the second UNet architecture branch into the subsequent stage. The module completes wavelet convolution and normalization processing on the feature map of the third layer of the second UNet architecture, adjusts the size of the feature map through convolution operation, and completes wavelet convolution and normalization processing on the feature map of the fourth layer of the first UNet architecture. The principle expression is:

[0074]

[0075]

[0076] in, is the corresponding vector of the third layer feature map of the second UNet architecture, It is the corresponding vector of the fourth layer feature map of the first UNet architecture. The two vectors are calculated by the Sigmoid function to obtain the selective weight factor , for After convolution processing, the features for After convolution processing, the features It is the feature output after the channel attention mechanism is processed.

[0077] 10. The wavelet convolution process consists of two branches. The first branch directly performs convolution operation on the input. The second branch uses the wavelet convolution kernel to extract the four feature maps of the input, and then restores the size of the feature map through inverse wavelet transform. The feature maps of the two branches are added to obtain the final wavelet convolution output. Specifically, the input is represented as , the direct convolution result of the first branch is , the second branch wavelet convolution kernel is:

[0078]

[0079] enter After convolution with the wavelet convolution kernel, we get , , , Four feature maps of the same size, which contain rich low-frequency information. The four feature maps are obtained through convolution operation. , , , , and then the output of the second branch is obtained through inverse wavelet convolution ,same After adding, we get the final wavelet convolution module output The specific formula is as follows:

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] 11. Finally, based on the output features of step 7, the network generates dose prediction results through a 1×1×1 convolutional layer, which further abstracts the previous fused features and outputs the final voxel-level dose regression prediction.

[0086] All parameters in the dose prediction model need to be learned through the training process, and the optimizer is continuously adjusted according to the feedback of the loss function to find the best parameter configuration. After the model training is completed, the parameters are fixed, and the final pixel-level regression results can be obtained directly by using the above steps. The network parameters are initialized by the Kaiming method, and the Adam optimizer is used for model training and parameter optimization under the guidance of the loss function. The initial learning rate is set to 0.0003. During the training process, the cosine annealing strategy is used to gradually reduce the learning rate. Cosine annealing adjusts the learning rate in the form of a cosine function. As the number of training rounds increases, the learning rate gradually decreases according to the cosine curve.

[0087] Dose prediction is a voxel-level regression task. Under the currently widely used loss functions (such as mean absolute error loss), convolutional neural networks (CNNs) tend to focus more on easy-to-learn areas and ignore the more difficult but more important areas. To solve this problem, this scheme introduces the difficult area learning loss to explicitly guide the model to pay more attention to these difficult areas, such as Figure 8 As shown, it is a schematic diagram of the structure of the difficult area learning loss.

[0088] The embodiment of the present invention uses the mean square error (MSE) to calculate the difference between the model prediction value and the true value (GT), which is denoted as Next, four dropout outputs with different probability distributions are used to replace the standard output of the model. The purpose of this process is to make the model produce four outputs with more diversity and randomness in one forward propagation iteration. These four dropout outputs are processed by two branches respectively. In the first branch, the four dropout outputs are processed by the function In the second branch, the average of the four dropout outputs is calculated and subtracted from the true value (GT), and then the function Get the second weight map. Then, use the mean square error loss Calculate the difference between the two generated weight maps to get the final difficult area learning loss .

[0089]

[0090]

[0091]

[0092] in, represents the number of samples, , , , is the result of four dropout predictions, D represents , , , A collection of represents the predicted value, represents the true value, is the variance of D, is the average value of D, and is a hyperparameter used to tune the normalization mapping and focus more on the difficult areas. and Set to 0.5 and 0.05 respectively.

[0093] For the dose distribution map, in some areas, the dose value exceeds the prescribed dose, and these areas need to be assigned more attention weights. Therefore, a semantic segmentation loss function is used to explicitly require the model to pay attention to high-dose areas. Fig. 9 is a dose distribution diagram, where the prescription dose of PTVnx is 70 Gray. In the embodiment of the present invention, the area with a dose greater than or equal to 70 Gray is marked as a high-dose area. Fig.10 This is a high-dose area map. By combining the binary cross entropy (BCE) and DICE loss function, the difference between the dose prediction result and the true value (GT) in these high-dose areas is calculated:

[0094]

[0095] Here, represents the region segmentation loss function, represents the total number of high-dose regions, n represents the nth high-dose region, represents the BCE loss of the nth high-dose region, represents the DICE loss of the nth high-dose region.

[0096] Final loss is the mean absolute error (MAE) loss , Difficult Area Learning Loss and auxiliary segmentation loss Combination of:

[0097]

[0098] Among them, the parameters The weight used to adjust the auxiliary segmentation loss is set to 0.1 to prevent the auxiliary segmentation loss weight from being too large, thereby reducing the final dose prediction performance.

[0099] Step S03, inputting the medical image to be identified into the target dose prediction model, and outputting the dose prediction result of the nasopharyngeal carcinoma radiotherapy plan.

[0100] In summary, the method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network in the above-mentioned embodiment of the present invention obtains image data, the image data includes CT images, annotation information of target areas and organs at risk, and dose distribution of radiotherapy plan, and processes the image data into an input image of a preset size; inputs the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model; inputs the medical image to be identified into the target dose prediction model, and outputs the dose prediction result of nasopharyngeal carcinoma radiotherapy plan. Specifically, the progressive feature fusion and the pixel attention guided fusion strategy based on wavelet convolution are introduced into the dose prediction model, which can effectively extract discriminative semantic features and enhance the semantic distinction ability of the network model. At the same time, it also effectively fuses deep-level semantic discrimination information and shallow-level spatial positioning details, and enhances the multi-level information fusion ability of the network model.

[0101] Embodiment 2

[0102] In order to train the dose prediction model, data of 131 patients were prepared. The data of each patient included CT images, annotation information of target areas and organs at risk (OARs), and dose distribution of radiotherapy plans. All CT images and dose distribution data were rescaled to a size of 96×128×128. Subsequently, the CT values ​​were truncated to the range of (-1024, 2048) and normalized to (0, 1). For the dose distribution data, all values ​​were divided by 70 Gray for processing. Subsequently, the model training was performed according to the method provided in Example 1 of the present invention. After the training was completed and the model parameters were determined, the test data was processed, the model reasoning was performed, and the result evaluation was completed.

[0103] To evaluate the performance of the model, this study used multiple dosimetric indicators, including , , , and These indicators are used to evaluate PTV70, PTV60 and PTV55. PTV70, PTV60 and PTV55 represent different tumor target volumes. Among them, PTV70 represents the dose of the corresponding target volume is 7000cGy, PTV60 represents the dose of the corresponding target volume is 6000cGy, and PTV55 represents the dose of the corresponding target volume is 5500cGy. and It is used to evaluate OAR (organs at risk). Indicates coverage of PTV x % absorbed dose, and Dmean represents the mean dose of PTV.

[0104] To directly quantify the dose difference between GT and model predictions, this protocol calculated the MAE:

[0105]

[0106] in, n represents the total number of voxels, and Represent GT and predicted value respectively.

[0107] See also Figures 11 to 14 , Fig.11 Schematic diagram of dose distribution of the true value (GT) and the predicted result; Fig.12 The dose-volume histogram (DVH) of the true value (GT) and predicted result of PTV55, where the solid line represents the true value and the dotted line represents the predicted value; Fig.13 The dose-volume histogram (DVH) of the true value (GT) and predicted result of PTV60, where the solid line represents the true value and the dotted line represents the predicted value; Fig.14 The dose-volume histogram (DVH) of the true value (GT) of PTV70 and the predicted result, where the solid line represents the true value and the dotted line represents the predicted value. Figures 11 to 14 It can be seen that the difference between the true value and the predicted result is small.

[0108] Embodiment 3

[0109] See also Fig.15 , Fig.151 is a structural block diagram of a nasopharyngeal carcinoma radiotherapy plan dose prediction system based on a progressive feature fusion network provided in Embodiment 3 of the present invention. The nasopharyngeal carcinoma radiotherapy plan dose prediction system 200 based on a progressive feature fusion network includes: an acquisition module 21, a training module 22 and an input module 23, wherein:

[0110] An acquisition module 21 is used to acquire image data, the image data including CT images, target volume and organ at risk annotation information, and dose distribution of a radiotherapy plan, and process the image data into an input image of a preset size;

[0111] The training module 22 is used to input the input image into the dose prediction model to train the dose prediction model and determine the target dose prediction model, wherein the total loss function used in the training is composed of the mean absolute error loss, the difficult area learning loss and the auxiliary segmentation loss, which is expressed as:

[0112]

[0113] in, is the total loss function, is the mean absolute error loss, Learn the loss for the hard region, is the weight used to adjust the auxiliary segmentation loss, is the auxiliary segmentation loss;

[0114] An input module 23 is used to input the medical image to be identified into the target dose prediction model and output the dose prediction result of the nasopharyngeal carcinoma radiotherapy plan;

[0115] The dose prediction model is composed of two progressive UNet architectures, wherein the first UNet architecture is used for preliminary feature extraction, and then the features extracted by the four upsampling layers are converted into attention weights through a channel attention module, and the attention weights are applied to the second UNet architecture, the first UNet architecture is a four-layer structure, and the second UNet architecture is a five-layer structure, and the channel attention module is used to sequentially pass a part of the input through the first convolution layer, the global pooling layer, the second convolution layer, the third convolution layer and the Sigmoid activation function, and then multiply it with another part of the input, and finally output it, wherein the first convolution layer includes a convolution kernel and an activation function of a size of 3×3×3, the second convolution layer includes a convolution kernel and an activation function of a size of 1×1×1, and the third convolution layer includes a convolution kernel of a size of 1×1×1;

[0116] The number of channels of each layer of the second UNet architecture is twice that of the corresponding layer of the first UNet architecture, and the input of each layer of the second UNet architecture includes features from the previous layer and corresponding features generated by the upsampling process of the first UNet architecture;

[0117] In addition, a pixel attention guided fusion module based on wavelet convolution is set between the third layer and the fourth layer during the downsampling process of the second UNet architecture. The encoders of the first UNet architecture and the second UNet architecture both use a convolution layer with a step size of 2 to replace the maximum pooling layer for downsampling operations. The encoder of the first UNet architecture uses a standard 3D convolution module, and the encoder of the second UNet architecture uses a residual dense block. The pixel attention guided fusion module based on wavelet convolution is used to complete wavelet convolution and normalization processing on the third layer feature image of the second UNet architecture branch, and adjust the size of the feature map through convolution operation. At the same time, wavelet convolution and normalization processing are completed on the fourth layer feature image of the first UNet architecture branch. In addition, the pixel attention guided fusion based on wavelet convolution The fusion module obtains the selective weight factor by calculating the two branch vectors through the Sigmoid function. The first convolution operation of the two branch feature maps in the pixel attention guided fusion module based on wavelet convolution is set to wavelet convolution. The wavelet convolution adds the result image after the wavelet convolution kernel and the inverse wavelet convolution kernel to the result image of the direct convolution operation, so as to increase the feature receptive field size exponentially while linearly increasing the training parameters. The residual dense block includes two branches, the upper branch includes three convolution modules, each convolution module connects the original feature with the transformed feature generated by the current convolution operation, and each convolution module extracts local features through 3×3×3 convolution; the lower branch extracts local features through 1×1×1 convolution; the outputs of the two branches are fused by element-by-element addition, and finally a 1×1×1 convolution is performed;

[0118] In the decoders of the first UNet architecture and the second UNet architecture, upsampling uses trilinear interpolation, and feature fusion is achieved through a residual module, and finally a 1×1×1 convolutional layer is used to generate the final nasopharyngeal carcinoma radiotherapy plan dose prediction result. The residual module includes two convolution groups, each convolution group is composed of a series of convolutional layers, and the convolutional layer is a 3×3×3 convolution kernel, wherein a part of the input passes through the first convolution group and is added to another part of the input to obtain a first output, and a part of the first output passes through the second convolution group and is multiplied by another part of the first output to obtain a second output.

[0119] Embodiment 4

[0120] Another aspect of the present invention provides an electronic device, see Fig.16, shown is an electronic device in Embodiment 4 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on the progressive feature fusion network as described above is implemented.

[0121] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0122] The memory 20 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of an electronic device, such as a hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Further, the memory 20 may also include both an internal storage unit and an external storage device of the electronic device. The memory 20 may be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is to be output.

[0123] It should be pointed out that Fig.16 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than those shown in the figure, or combine certain components, or arrange the components differently.

[0124] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on the progressive feature fusion network.

[0125] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0126] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0127] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0129] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network, characterized in that: The method comprises: Acquiring image data, the image data including CT images, target volume and organ at risk annotation information, and dose distribution of a radiotherapy plan, and processing the image data into an input image of a preset size; Inputting the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model; Inputting the medical image to be identified into the target dose prediction model, and outputting the dose prediction result of the nasopharyngeal carcinoma radiotherapy plan; The dose prediction model is composed of two progressive UNet architectures, wherein the first UNet architecture is used for preliminary feature extraction, and then the features extracted by the four upsampling layers are converted into attention weights through the channel attention module, and the attention weights are applied to the second UNet architecture, the first UNet architecture is a four-layer structure, and the second UNet architecture is a five-layer structure; The number of channels of each layer of the second UNet architecture is twice that of the corresponding layer of the first UNet architecture, and the input of each layer of the second UNet architecture includes features from the previous layer and corresponding features generated by the upsampling process of the first UNet architecture; In addition, a pixel attention guided fusion module based on wavelet convolution is set between the third layer and the fourth layer in the downsampling process of the second UNet architecture. The encoders of the first UNet architecture and the second UNet architecture both use a convolution layer with a step size of 2 to replace the maximum pooling layer for downsampling operations. The encoder of the first UNet architecture uses a standard 3D convolution module, and the encoder of the second UNet architecture uses a residual dense block. In the decoders of the first UNet architecture and the second UNet architecture, upsampling uses trilinear interpolation, and feature fusion is achieved through a residual module, and finally a 1×1×1 convolutional layer is used to generate the final nasopharyngeal carcinoma radiotherapy plan dose prediction result.

2. The method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on progressive feature fusion network according to claim 1, characterized in that: The channel attention module is used to output a part of the input after passing it through the first convolution layer, the global pooling layer, the second convolution layer, the third convolution layer and the Sigmoid activation function in sequence, and then multiply it with another part of the input for final output, wherein the first convolution layer includes a convolution kernel and an activation function of size 3×3×3, the second convolution layer includes a convolution kernel and an activation function of size 1×1×1, and the third convolution layer includes a convolution kernel of size 1×1×1.

3. The method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on progressive feature fusion network according to claim 1, characterized in that: The pixel attention guided fusion module based on wavelet convolution is used to complete wavelet convolution and normalization processing on the third layer feature image of the second UNet architecture branch, and adjust the size of the feature map through convolution operation, and at the same time complete wavelet convolution and normalization processing on the fourth layer feature image of the first UNet architecture branch. In addition, the pixel attention guided fusion module based on wavelet convolution obtains the selective weight factor through the Sigmoid function calculation of two branch vectors.

4. The method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on progressive feature fusion network according to claim 3, characterized in that: The first convolution operation of the two-branch feature map in the wavelet convolution-based pixel attention guided fusion module is set to wavelet convolution. The wavelet convolution adds the result image after the wavelet convolution kernel and the inverse wavelet convolution kernel to the result image of the direct convolution operation, so as to increase the feature receptive field size exponentially while linearly increasing the training parameters.

5. The method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on progressive feature fusion network according to claim 1, characterized in that: The residual dense block includes two branches, the upper branch includes three convolution modules, each convolution module concatenates the original features with the transformed features generated by the current convolution operation, and each convolution module extracts local features through 3×3×3 convolution; The lower branch extracts local features through 1×1×1 convolution; the outputs of the two branches are fused by element-wise addition and finally subjected to 1×1×1 convolution.

6. The method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on progressive feature fusion network according to claim 1, characterized in that: The residual module includes two convolution groups, each convolution group is composed of a series of convolution layers, and the convolution layer is a 3×3×3 convolution kernel, wherein a part of the input passes through the first convolution group and is added to another part of the input to obtain a first output, and a part of the first output passes through the second convolution group and is multiplied with another part of the first output to obtain a second output.

7. The method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on progressive feature fusion network according to claim 1, characterized in that: In the step of inputting the input image into the dose prediction model to train the dose prediction model and determine the target dose prediction model, the total loss function used in the training is composed of a mean absolute error loss, a difficult region learning loss, and an auxiliary segmentation loss, and is expressed as: in, is the total loss function, is the mean absolute error loss, Learn the loss for the hard region, is the weight used to adjust the auxiliary segmentation loss, is the auxiliary segmentation loss.

8. A nasopharyngeal carcinoma radiotherapy planning dose prediction system based on progressive feature fusion network, characterized in that: A method for predicting nasopharyngeal carcinoma radiotherapy plan dose based on a progressive feature fusion network according to any one of claims 1 to 7, the system comprising: An acquisition module, configured to acquire image data, the image data including CT images, target volume and organ at risk annotation information, and dose distribution of a radiotherapy plan, and process the image data into an input image of a preset size; A training module, used for inputting the input image into a dose prediction model to train the dose prediction model and determine a target dose prediction model; An input module, used to input the medical image to be identified into the target dose prediction model, and output the dose prediction result of the nasopharyngeal carcinoma radiotherapy plan; The dose prediction model is composed of two progressive UNet architectures, wherein the first UNet architecture is used for preliminary feature extraction, and then the features extracted by the four upsampling layers are converted into attention weights through the channel attention module, and the attention weights are applied to the second UNet architecture, the first UNet architecture is a four-layer structure, and the second UNet architecture is a five-layer structure; The number of channels of each layer of the second UNet architecture is twice that of the corresponding layer of the first UNet architecture, and the input of each layer of the second UNet architecture includes features from the previous layer and corresponding features generated by the upsampling process of the first UNet architecture; In addition, a pixel attention guided fusion module based on wavelet convolution is set between the third layer and the fourth layer in the downsampling process of the second UNet architecture. The encoders of the first UNet architecture and the second UNet architecture both use a convolution layer with a step size of 2 to replace the maximum pooling layer for downsampling operations. The encoder of the first UNet architecture uses a standard 3D convolution module, and the encoder of the second UNet architecture uses a residual dense block. In the decoders of the first UNet architecture and the second UNet architecture, upsampling uses trilinear interpolation, and feature fusion is achieved through a residual module, and finally a 1×1×1 convolutional layer is used to generate the final nasopharyngeal carcinoma radiotherapy plan dose prediction result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for predicting the dose of nasopharyngeal carcinoma radiotherapy plan based on a progressive feature fusion network as described in any one of claims 1 to 7 is implemented.

Images