Advanced nasopharynx cancer multi-target dose prediction system based on deep learning

Abstract

The invention relates to the technical field of medical drug dose analysis, in particular to a deep learning-based advanced nasopharyngeal carcinoma multi-target dose prediction system, which comprises a data acquisition module, an automatic sketching module, a dose prediction model construction module, a diversified dose generation module, a model training and optimization module, a result output and visualization module and the like. The automatic sketching module is used for realizing accurate and automatic sketching by using a multi-modal fusion and knowledge distillation framework; the dose prediction module accurately extracts features based on a UNetr network model and generates a multi-channel dose prediction result; the diversified dose generation module generates diversified complementary dose distribution through a specific method and a constraint mechanism; the model training and optimizing module effectively improves the model performance; the result output and visualization module presents results in various visual modes for doctors to make decisions; the nasopharynx cancer treatment dose prediction accuracy and individuation degree can be improved, the automatic sketching precision and efficiency can be improved, the clinical decision process can be optimized, and development of related technologies can be promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of medical drug dosage analysis, and in particular to a multi-target dosage prediction system for advanced nasopharyngeal carcinoma based on deep learning. Background Art

[0002] Nasopharyngeal carcinoma is a common malignant tumor of the head and neck. Radiotherapy is one of the main treatments for nasopharyngeal carcinoma and plays a key role in controlling tumor growth and improving patient survival. However, the treatment of advanced nasopharyngeal carcinoma faces many challenges, among which accurate dose prediction and precise delineation of tumor target areas and organs at risk are key factors in developing an effective radiotherapy plan.

[0003] Traditional dose prediction and delineation methods have the following limitations:

[0004] Reliance on manual experience: Traditional dose prediction and delineation of tumor target areas and organs at risk mainly rely on the doctor's manual experience and manual operation. Doctors need to determine the appropriate radiotherapy dose and delineate the range of tumors and organs at risk based on the patient's imaging data (such as CT, MR, etc.) as well as their own professional knowledge and clinical experience. This method is not only time-consuming and laborious, but there may be large subjective differences between different doctors, resulting in increased uncertainty in treatment effects.

[0005] Limited accuracy: When manually outlining tumor target areas and organs at risk, it is difficult to achieve very precise definitions due to the complexity of imaging data (such as unclear tumor boundaries, similarities between normal tissue and diseased tissue in imaging, etc.), and outlining errors are prone to occur. Similarly, in terms of dose prediction, empirical methods are difficult to fully consider individual differences among patients (such as tumor heterogeneity, physical condition of patients, etc.), so that the predicted dose may not achieve the best treatment effect, or may cause unnecessary damage to normal tissues, thereby increasing the risk of complications.

[0006] Therefore, to address the above problems, a multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning was proposed. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning includes:

[0010] Data acquisition module: The data acquisition module acquires the multimodal imaging data and clinical information of the patient;

[0011] Automatic outlining module: The automatic outlining module receives multimodal imaging data, uses multimodal information fusion technology to integrate the particularity and complementarity of multimodal imaging data, and uses the multimodal segmentation network framework based on the knowledge distillation framework to guide the student model training using the soft target of the teacher model, automatically outlining the structure of the multimodal imaging data, and obtaining the automatic outlining results of the tumor target area and organs at risk;

[0012] Dose prediction model building module; The dose prediction model building module receives the automatic delineation results, extracts features through the encoder of the UNetr network model based on the fusion Transformer module and UNet advantages, and the decoder fuses the features and generates multiple dose prediction channels including uncertainty weights and predicted dose results, and each dose prediction channel has two branches, which respectively generate uncertainty weights and predicted dose results;

[0013] Diversified dose generation module: The diversified dose generation module includes a multi-branch prediction method based on shared trunk features and a triplet loss function constraint mechanism. The consistency of multiple branch dose prediction results is guided by trunk feature information sharing. Each branch learns diversified results through a convolutional layer. At the same time, the triplet loss function is used to constrain the multi-branch model learning process to generate diversified dose distributions.

[0014] Model training and optimization module: The model training and optimization module uses training algorithms and optimization strategies to train and optimize the dose prediction model. During the training process, the training set data is used to update the model parameters, and the validation set data is used to evaluate the model performance and adjust the hyperparameters.

[0015] Result output and visualization module: The result output and visualization module receives automatic delineation results and diversified dose distribution for visualization. The visualization methods include dose distribution curve, three-dimensional dose cloud map, and tumor and organ-at-risk delineation contour display.

[0016] As a preferred solution, in the automatic outlining module, the distillation loss function of the knowledge distillation framework is defined as ,in, Represents the number of feature dimensions output by the teacher model and the student model, represents the index of the sample, and are the outputs of the teacher model and the student model respectively, and the segmentation network based on the encoder-decoder architecture is integrated in the two processing branches of the network framework, and the cross entropy loss is used as the segmentation loss function. The cross entropy loss function is defined as ,in, is the sample size, is the true label, is the predicted label.

[0017] As a preferred solution, in the multi-branch prediction method based on shared trunk features, trunk feature information sharing ensures that multiple branch dose prediction results can achieve diversified learning on the basis of commonality.

[0018] As a preferred solution, the steps of dose prediction by the dose prediction model building module are as follows:

[0019] Step S1, receiving the automatic delineation result outputted from the automatic delineation module, the automatic delineation result includes the delineation information of the tumor target area and the organ at risk, the delineation information of the tumor target area and the organ at risk is presented in the form of an image, wherein each pixel corresponds to the labeling information of whether it belongs to a tumor or a specific organ at risk, and the automatic delineation result is used as input data and sent to the UNetr network model;

[0020] Step S2: In the encoder part of the UNetr network model, the Transformer module is used to extract features from the input data. The Transformer module captures the long-distance dependencies and global context information in the input data based on its self-attention mechanism. Suppose the input data is X, the self-attention calculation process in the Transformer module is as follows:

[0021] The input X is mapped into query vector Q, key vector K and value vector V respectively through linear transformation, the formula is: , , ,in, , , is a learnable weight matrix;

[0022] Step S3: Calculate attention score , the formula is: ;in, is the key vector The dimension of is the key vector The transpose of The function is used to normalize the attention score to interval, so that the sum of each row is 1;

[0023] Step S4: Perform weighted summation of the value vectors using the attention score to obtain the output processed by the self-attention mechanism , the formula is: After being processed by multiple layers of Transformer modules, an intermediate feature representation with rich global feature information is obtained. ;

[0024] Step S5: While the Transformer module extracts features, the encoder of the UNetr network model also follows the UNet architecture style to perform downsampling operations. Downsampling is achieved by combining convolutional layers and pooling layers. After each downsampling, the features extracted from different levels are fused. The features obtained after downsampling are , and compare it with the features extracted by the Transformer module at the previous level Fusion is performed by concatenation or element-level addition to obtain the fused features ;

[0025] Step S6: In the decoder part, the features extracted from different levels of the encoder are transferred to the corresponding level of the decoder using skip connections. The characteristics of the level are It is passed to the corresponding level of the decoder through the skip connection, which is recorded in the decoder as ;

[0026] Step S7: In the decoder, upsampling is performed using a convolutional layer operation to restore the resolution of the data. After each upsampling, the upsampled features are combined with the features of the corresponding level transmitted through the jump connection. Fusion is performed to obtain the fused features , through multiple upsampling and feature fusion operations, the features are restored to a resolution close to the original input data, and feature information at different levels is fused;

[0027] Step S8: After the decoder operation, the final feature representation that combines features at different levels is obtained. , based on the final feature representation , multiple dose prediction channels are generated through a series of fully connected layers or convolutional layers. Let the number of generated dose prediction channels be , then for each dose prediction channel , , the output of the channel is obtained through the corresponding network layer operation;

[0028] Step S9: For each dose prediction channel , its output has two branches, which are used to generate uncertainty weights and predicted dose results :

[0029] Step S9-1: Generate uncertainty weights Branch: The final feature representation is obtained through a sub-network structure of a small network containing several fully connected layers or convolutional layers. Processing to obtain uncertainty weight ;

[0030] Step S9-2: Generate predicted dose results Branch: The final feature representation is achieved through the sub-network structure Processing to obtain the predicted dose results , predicted dose results is the radiation therapy dose value predicted by the model for each voxel point.

[0031] As a preferred solution, in the triple loss function constraint mechanism, the triple loss function is ,in, represents anchor data, represents the dose prediction that matches the anchor data, represents the dose prediction that matches the anchor data, represents a preset interval, , Represents the calculation method of a set of data feature distances. When the calculation method is the Euclidean distance calculation method, the formula for the Euclidean distance calculation is: ,in, and are two data points, and Represents data points and In the The value of the feature dimension, It is a data point and The number of features.

[0032] It can be seen from the technical solution provided by the present invention that the multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning provided by the present invention has the following beneficial effects: by adopting the UNetr network model that integrates the advantages of Transformer and UNet for dose prediction, its encoder uses the Transformer module to capture the deep features and global information of three-dimensional CT volume data, the decoder uses jump connections to fuse features of different levels and resolutions, and each dose prediction channel sets two branches to generate uncertainty weights and predicted dose results respectively. This architecture design helps to predict radiotherapy doses more accurately and provide more reliable data support for clinical treatment; the diversified dose generation module uses a multi-branch prediction method based on shared trunk features and a triple loss function constraint mechanism to generate diversified and complementary dose distributions for each patient; the automatic delineation module uses multimodal information fusion technology to integrate the particularity and complementarity of multiple modal images such as CT, MR, and PET, and can more comprehensively and accurately identify tumor areas and endangered organs; the multimodal segmentation network framework based on the knowledge distillation framework uses the soft target of the teacher model to guide the student model training, and can obtain a reasonable automatic delineation structure when single-modal, dual-modal and multi-modal inputs. In this way, not only can the accuracy of automatic delineation be improved, but also the training complexity and computing resource consumption of the model can be reduced to a certain extent, and the delineation efficiency can be improved; the result output and visualization module visualizes the diverse dose prediction results generated by the model and the automatic delineation results in various ways, such as dose distribution curves, three-dimensional dose cloud maps, and delineation contours of tumors and organs at risk, etc. It also supports comparative analysis and display of different dose prediction results. These intuitive visualization methods can enable clinicians to have a clearer and more comprehensive understanding of the patient's condition and the effects of different dose regimens, so as to make more scientific and reasonable treatment decisions;

[0033] The present invention integrates multiple functional modules such as data acquisition, automatic delineation, dose prediction, diversified generation and visualization to form a complete multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning. This integrated system helps to promote the development of nasopharyngeal carcinoma radiotherapy technology in a more intelligent, precise and personalized direction, and provides strong support for improving the treatment quality and quality of life of nasopharyngeal carcinoma patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the overall structure of the multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning in the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0037] like Figure 1 As shown, an embodiment of the present invention provides a multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning, including a data acquisition module, an automatic delineation module, a dose prediction model construction module, a diversified dose generation module, a model training and optimization module, and a result output and visualization module.

[0038] In this embodiment, the data acquisition module acquires the multimodal imaging data and clinical information of the patient;

[0039] Among them, the data sources of multimodal imaging data and clinical information include the hospital's picture archiving and transmission system (PACS), electronic medical record system (EMR), clinical laboratory information system (LIS), etc.

[0040] In this embodiment, the automatic delineation module receives multimodal image data, uses multimodal information fusion technology to integrate the particularity and complementarity of multimodal image data, uses a multimodal segmentation network framework based on a knowledge distillation framework, uses the soft target of the teacher model to guide the student model training, automatically delineates the structure of the multimodal image data, and obtains the automatic delineation results of the tumor target area and the organ at risk, which specifically includes the following steps:

[0041] 1. Multimodal image data reception and preprocessing

[0042] Receiving data: The automatic delineation module first receives multimodal image data acquired from the data acquisition module, including images of different modalities such as CT, MR, and PET;

[0043] Data preprocessing:

[0044] Normalization: Normalize the image data of each modality and map the pixel values ​​to a specific interval, such as [0,1]. ( Represents different modalities, such as CT, MR, PET), assuming that its original pixel value range is , then the normalization formula is: ;

[0045] Cropping and scaling: According to the subsequent network processing requirements and the area of ​​interest (such as the nasopharynx and surrounding areas that may be involved), the image is cropped to remove irrelevant areas, and then scaled to make its size meet the network input requirements; for example, the image size is reduced from the original size to the image size through methods such as bilinear interpolation. (height and width) resize to target size ;

[0046] 2. Application of multimodal information fusion technology

[0047] Feature extraction:

[0048] For the preprocessed image data of each modality, a convolutional neural network (CNN) layer or other appropriate feature extraction methods are used to extract features; taking CT images as an example, the preprocessed CT image data is assumed to be , through a convolutional layer (the convolution kernel size is , the step length is , , filled with , the convolution kernel weight is , the bias is ) to extract features and obtain feature maps , the calculation formula is:

[0049] in, represents the coordinate position in the feature map, Represents the index of the convolution kernel in the height direction, Represents the index of the convolution kernel in the width direction; similarly, corresponding feature extraction is performed on MR and PET images to obtain and ;

[0050] Integration strategy determination and implementation:

[0051] Select an appropriate fusion strategy. Here, we assume that a mid-term fusion strategy is adopted, that is, fusion is performed in the middle layer of the network.

[0052] After a certain number of layers of feature extraction, the features of different modalities are fused; for example, the feature maps extracted from CT, MR, and PET images are combined. , , The fused feature map is obtained by element-wise addition. , the calculation formula is: ;

[0053] 3. Multimodal segmentation network framework construction and teacher model training based on knowledge distillation framework

[0054] Network framework construction:

[0055] Construct a multimodal segmentation network based on the knowledge distillation framework, which consists of a teacher model and a student model. Both the teacher model and the student model use a similar segmentation network structure based on the encoder-decoder architecture, but the teacher model is relatively more complex, has more parameters, and has stronger learning ability.

[0056] Encoder part: for the input fused feature map , the encoder performs downsampling operations through a series of convolutional layers and pooling layers, gradually reducing the data resolution and extracting more abstract features; for example, after a convolution operation through a convolutional layer, it is downsampled through a maximum pooling layer; let the feature obtained after the i-th downsampling be ;

[0057] Decoder part: Corresponding to the encoder, the decoder performs upsampling operation through the transposed convolution layer (or upsampling convolution layer), gradually restores the data resolution, and combines the features passed from the encoder through the jump connection to fuse the features of different levels to generate the final segmentation result; let the feature obtained after the upsampling be , the corresponding feature passed from the encoder through the skip connection is , and fuse them (such as by concatenation or element-level addition) to obtain the fused features ;

[0058] Teacher model training:

[0059] Data preparation: A large amount of multimodal imaging data with accurate tumor target and organ at risk delineation annotations is collected as the training set of the teacher model, denoted as ,in , , They are CT, MR, and PET image training data, It is the actual delineation and annotation of the corresponding tumor target area and organs at risk (represented by a binary image, with the tumor area as 1 and the normal tissue as 0);

[0060] Define the loss function: Use the cross entropy loss function as the loss function of the teacher model; for the input multimodal image data , the predicted outline result is obtained through the teacher model network , cross entropy loss function The calculation formula is: in, is the number of training samples, is the true outline annotation of the sample, is the predicted outline result of the sample;

[0061] Training process: Use stochastic gradient descent (SGD) or its variants (such as Adam, etc.) to optimize the parameters of the teacher model according to the calculated loss value. Update; taking the Adam algorithm as an example, the update formula is as follows: ; ; ; ; ;

[0062] in, , is the first-order moment estimate and second-order moment estimate in the Adam algorithm, , is a hyperparameter (usually , ), is the learning rate, A very small number (to prevent division by zero errors, usually 10 -8 ), Indicates the number of training steps; the teacher model training process continues until the preset stop condition is reached on the validation set, such as the validation set loss no longer decreases or the maximum number of training rounds is reached;

[0063] 4. Student model training (soft target guidance based on teacher model)

[0064] Data preparation: Multimodal image data is also used as input, that is, ;

[0065] Soft target acquisition: During the teacher model training process, in addition to the final prediction and outlining results In addition, a soft target is generated in the middle layer of the network (set as the first layer), that is, the feature representation ;

[0066] Define the loss function:

[0067] Define the distillation loss function To measure the difference between the soft target of the student model and the teacher model, the calculation formula is: ;

[0068] in, is the number of training samples, It is The dimension of the layer feature representation (e.g., the number of pixels in the feature map, etc.);

[0069] At the same time, define the cross entropy loss function It is used to measure the difference between the final prediction result of the student model and the true annotation. The calculation formula is the same as the cross entropy loss function of the teacher model: ;

[0070] Define the total loss function To comprehensively consider the distillation loss and cross entropy loss, the calculation formula is: ;

[0071] in, is the weight coefficient used to balance the importance of distillation loss and cross entropy loss;

[0072] Training process: Use stochastic gradient descent (SGD) or its variants (such as Adam, etc.) to optimize the parameters of the student model according to the calculated total loss value. Update; taking the Adam algorithm as an example, the update formula is as follows: ; ; ; ; ;

[0073] The student model training process continues until the preset stopping condition is reached on the validation set, such as the validation set loss no longer decreasing or the maximum number of training rounds is reached;

[0074] 5. Automatically generate outline results

[0075] After the student model training is completed, the new multimodal image data Input into the trained student model, and after the network processing of the student model, the automatic delineation results of the tumor target area and organs at risk can be obtained. ;

[0076] These automatic delineation results can be binary images (such as tumor area is 1, normal tissue is 0), or segmented images with probability values, showing the probability of each pixel belonging to tumor or normal tissue.

[0077] In this embodiment, the dose prediction model construction module receives the automatic delineation result, extracts features through the encoder of the UNetr network model based on the fusion Transformer module and the UNet advantage, and the decoder fuses the features and generates multiple dose prediction channels including uncertainty weights and predicted dose results, and each dose prediction channel has two branches, respectively generating uncertainty weights and predicted dose results, including the following steps:

[0078] 1. Receive the automatic delineation results output by the automatic delineation module. The automatic delineation results include the delineation information of the tumor target area and the organs at risk. The delineation information of the tumor target area and the organs at risk is presented in the form of an image, in which each pixel corresponds to the annotation information of whether it belongs to a tumor or a specific organ at risk. The automatic delineation results are used as input data and sent to the UNetr network model;

[0079] 2. In the encoder part of the UNetr network model, the Transformer module is used to extract features from the input data. The Transformer module captures the long-distance dependencies and global context information in the input data based on its self-attention mechanism. Suppose the input data is X, the self-attention calculation process in the Transformer module is as follows:

[0080] The input X is mapped into query vector Q, key vector K and value vector V respectively through linear transformation, the formula is: , , ,in, , , is a learnable weight matrix;

[0081] 3. Calculate the attention score , the formula is: ;in, is the key vector The dimension of is the key vector The transpose of The function is used to normalize the attention score to interval, so that the sum of each row is 1;

[0082] 4. Perform weighted summation of the value vectors using the attention score to obtain the output processed by the self-attention mechanism , the formula is: After being processed by multiple layers of Transformer modules, an intermediate feature representation with rich global feature information is obtained. ;

[0083] 5. While the Transformer module extracts features, the encoder of the UNetr network model also follows the UNet architecture style to perform downsampling operations. Downsampling is achieved by combining convolutional layers and pooling layers. After each downsampling, the features extracted from different levels are fused. The features obtained after downsampling are , and compare it with the features extracted by the Transformer module at the previous level Fusion is performed by concatenation or element-level addition to obtain the fused features ;

[0084] 6. In the decoder part, the features extracted from different levels of the encoder are transferred to the corresponding levels of the decoder using skip connections. The characteristics of the level are It is passed to the corresponding level of the decoder through the skip connection, which is recorded in the decoder as ;

[0085] 7. In the decoder, the convolution layer operation is used for upsampling to restore the resolution of the data. After each upsampling, the upsampled features are compared with the features of the corresponding level passed through the jump connection. Fusion is performed to obtain the fused features , through multiple upsampling and feature fusion operations, the features are restored to a resolution close to the original input data, and feature information at different levels is fused;

[0086] 8. After the decoder operation, the final feature representation that combines features at different levels is obtained , based on the final feature representation , multiple dose prediction channels are generated through a series of fully connected layers or convolutional layers. Let the number of generated dose prediction channels be , then for each dose prediction channel , , the output of the channel is obtained through the corresponding network layer operation;

[0087] 9. For each dose prediction channel , its output has two branches, which are used to generate uncertainty weights and predicted dose results :

[0088] 9-1. Generate uncertainty weights Branch: The final feature representation is obtained through a sub-network structure of a small network containing several fully connected layers or convolutional layers. Processing to obtain uncertainty weight ;

[0089] 9-2. Generate predicted dose results Branch: The final feature representation is achieved through the sub-network structure Processing to obtain the predicted dose results , predicted dose results is the radiation therapy dose value predicted by the model for each voxel point.

[0090] In this embodiment, the diversified dose generation module includes a multi-branch prediction method based on shared trunk features and a triplet loss function constraint mechanism. The consistency of multiple branch dose prediction results is guided by sharing trunk feature information. Each branch learns diversified results through a convolutional layer. At the same time, the triplet loss function is used to constrain the multi-branch model learning process to generate diversified dose distributions. Specifically, the following steps are included:

[0091] 1. Shared backbone feature extraction

[0092] Data preparation:

[0093] First, obtain the feature data output from the dose prediction model building module; these feature data have undergone a series of processing, such as extracting and fusing features of different levels and resolutions through the Transformer module and UNet architecture related operations in the encoder, including information related to the tumor target area, organs at risk, and the overall patient condition;

[0094] Backbone feature extraction:

[0095] The input feature data is further processed using a shared convolutional neural network (CNN) backbone network to extract shared backbone features; this backbone network can be composed of multiple convolutional layers, pooling layers, etc., and its structure and parameters are fixed, and is used to mine representative and universal features from the input data;

[0096] Assume the input feature data is After a series of convolution and pooling operations in the shared backbone network, the shared backbone features are obtained. ; For example, after a convolution layer (the convolution kernel size is , the step length is , filled with , the convolution kernel weight is , the bias is ) operation, the calculation formula is as follows:

[0097] in, Represents the coordinate position in the feature map; through multiple layers of such operations, more abstract and representative shared backbone features are gradually extracted ;

[0098] 2. Multi-branch prediction based on shared trunk features

[0099] Branch network construction:

[0100] Construct multiple branch networks, each of which is based on shared trunk features Further learning and prediction are performed; these branch networks can have different structures and parameter settings so that diversified results can be learned; for example, some branch networks can be set to focus on the high-dose distribution in the tumor area, while other branch networks can be set to focus on the low-dose distribution that protects the organs at risk, etc.;

[0101] Each branch network is also composed of a series of convolutional layers, fully connected layers, etc., and is designed according to its specific prediction target;

[0102] Diverse results learning:

[0103] For each branch network, the trunk features will be shared The input is further extracted and learned through the convolutional layers specific to each branch;

[0104] Taking one of the branch networks as an example, let the first convolution layer parameter of the branch network be (convolution kernel size , step length ,filling , convolution kernel weight , bias ), the shared backbone features of the input Perform convolution operation to obtain a new feature map , the calculation formula is:

[0105] Through multiple such convolutional layers and possible fully connected layers, each branch network gradually learns the features related to its own target and finally generates different dose prediction results. Let the dose prediction result generated by branch network n be D n This result is based on the unique learning and processing of the shared trunk features by the branch network. The D n It will present diverse characteristics, such as different dose distribution patterns, different focuses (high dose to tumors or low dose to organs at risk, etc.);

[0106] 3. Application of triplet loss function constraint mechanism

[0107] Data preparation:

[0108] Select appropriate samples from the training data set as anchor data, positive samples, and negative samples; the anchor data is a randomly selected sample, the positive sample is a dose prediction that matches the anchor data, usually a sample that belongs to the same category as the anchor data (for example, for the same patient, a different but reasonable high-dose prediction scheme), and the negative sample is a dose prediction that does not match the anchor data, that is, a sample of a different category from the anchor data (for example, for the same patient, a dose prediction scheme that is too low or too high and unreasonable);

[0109] Feature distance calculation:

[0110] For each sample (anchor data, positive sample, negative sample), calculate the feature distance between it and other samples; here we use Euclidean distance as an example to calculate the data feature distance. and samples , whose characteristic dimension is , then the Euclidean distance The calculation formula is:

[0111] Calculate the distance between anchor data and positive samples respectively , the distance between anchor data and negative samples ;

[0112] Triplet loss function calculation and model update:

[0113] Define triplet loss function ,in, is a preset interval, which is a hyperparameter used to control the distance difference between positive and negative samples;

[0114] According to the calculated distance value, the value of the triplet loss function is calculated; during the training process, the learning process of the multi-branch model is constrained by minimizing the triplet loss function; the stochastic gradient descent (SGD) or its variants (such as Adam, etc.) optimization algorithm is used to update the parameters of the multi-branch model according to the calculated triplet loss function value; taking the Adam algorithm as an example, the update formula is as follows: ; ; ; ; ;

[0115] in, , are the first-order moment estimation and second-order moment estimation in the algorithm, , is a hyperparameter (usually , ), is the learning rate, is a very small number (to prevent division by zero errors, usually ), (t represents the number of training steps; by continuously adjusting the model parameters, the distance between the positive sample and the anchor data is made as small as possible, and the distance between the negative sample and the anchor data is made as large as possible, while maintaining a reasonable difference from the anchor sample, thereby prompting the model to generate more diverse and complementary dose distributions;

[0116] 4. Generate diverse dose distributions

[0117] During the training process, as the multi-branch model continues to learn and optimize, the diversified results learned by each branch network based on the shared trunk features and the constraints of the triplet loss function eventually generate a diversified dose distribution;

[0118] These diverse dose distributions include different dose schemes generated by different branch networks for different objectives (such as tumor control, protection of endangered organs, etc.), and maintain a certain rationality and complementarity through the constraints of the ternary loss function as a whole. They can provide clinicians with a variety of optional dose schemes to better meet the personalized treatment needs of different patients.

[0119] In this embodiment, the model training and optimization module uses a training algorithm and an optimization strategy to train and optimize the dose prediction model. During the training process, the training set data is used to update the model parameters, and the performance of the model is evaluated and the hyperparameters are adjusted through the validation set data, including the following steps:

[0120] 1. Data preparation

[0121] Collect relevant data and divide it into training set, validation set and test set according to a certain ratio;

[0122] 2. Selection and initialization

[0123] Select a suitable training algorithm (such as Adam, etc.) and initialize the dose prediction model parameters (such as He initialization, etc.);

[0124] 3. Update parameters using training set

[0125] Set training parameters such as batch size and initial learning rate;

[0126] Perform forward propagation on each batch of data in the training set to obtain the prediction results, and calculate the loss function value accordingly;

[0127] Use the selected training algorithm to update the model parameters through back propagation according to the loss value to complete rounds of training;

[0128] 4. Evaluate and adjust using the validation set

[0129] Use validation set data to evaluate model performance every certain number of rounds and calculate performance indicators such as MSE and MAE;

[0130] If the performance is not satisfactory, adjust the hyperparameters (such as learning rate, network architecture related parameters, etc.);

[0131] Repeat the training and evaluation adjustment process until the model achieves satisfactory performance indicators on the validation set.

[0132] In this embodiment, the result output and visualization module receives the automatic delineation results and the diversified dose distribution for visualization, and the visualization method includes displaying the delineation contours of the dose distribution curve, the three-dimensional dose cloud map, and the tumor and the organ at risk, including the following steps:

[0133] 1. Data reception and preparation

[0134] Receiving data: The result output and visualization module first receives the automatic contouring results from the automatic contouring module and the diversified dose distribution data from the diversified dose generation module; the automatic contouring results are usually presented in the form of images, with the contour information of the tumor target area and the organs at risk marked; the diversified dose distribution data includes the dose information under different schemes, such as the dose value corresponding to each voxel point, etc.

[0135] Data format checking and preprocessing:

[0136] Check whether the received data format meets the requirements of subsequent visualization processing; if the data format is inconsistent, perform necessary conversion operations; for example, convert image data in different formats into a format that can be recognized by commonly used visualization libraries (such as Matplotlib, Mayavi, etc.), such as numpy array format, etc.;

[0137] For dose distribution data, normalization is required to make its value range suitable for visual display;

[0138] 2. Dose distribution curve visualization

[0139] Data extraction and organization:

[0140] Extract the dose value of a specific area (such as a tumor area, a specific organ at risk area, or an overall area of ​​interest) from the diversified dose distribution data, and the data on the changes of the dose value with the spatial position (such as along a certain coordinate axis) or other related factors (such as time, if dynamic dose analysis is involved); for example, for the tumor area, the dose values ​​of different points on the tumor centerline can be extracted, or the dose values ​​of each grid point can be extracted according to a certain grid division;

[0141] Organize the extracted data in a certain order to form a data structure suitable for drawing curves, such as forming a two-dimensional array of the coordinates of each point and the corresponding dose value;

[0142] Draw a curve:

[0143] Use a suitable visualization library (such as Matplotlib) to draw curves. First, create a drawing object, and then draw the dose distribution curve by calling the relevant functions of the drawing library (such as the plot function) based on the organized data. For example, if the dose value of the tumor area changes along a certain coordinate axis, the coordinate value in the coordinate axis direction can be used as the horizontal coordinate, and the corresponding dose value can be used as the vertical coordinate for drawing.

[0144] Set the curve's appearance attributes such as color, line type (such as solid line, dashed line, etc.), and mark (such as circle, triangle, etc.) to make the curve visually clearer and more beautiful; at the same time, you can add axis labels (such as "position" on the horizontal axis and "dose value" on the vertical axis), titles (such as "tumor area dose distribution curve") and other annotation information to better understand the content represented by the curve;

[0145] 3. 3D dose cloud visualization

[0146] Data preparation and gridding:

[0147] Gridding the diversified dose distribution data, that is, dividing the dose values ​​in the three-dimensional space into grid units according to certain rules, and each grid unit corresponds to a dose value; the purpose of gridding is to facilitate the subsequent three-dimensional visualization processing, so that the dose values ​​can be presented more regularly in the three-dimensional space; some special gridding algorithms (such as Delaunay triangulation, etc.) can be used to achieve this, or the gridding function provided by the visualization library can be used;

[0148] Determine the coordinate range of the three-dimensional space and set the appropriate coordinate axis range (such as the value range of the x, y, and z axes) according to the actual situation, so as to fully display the three-dimensional form of the dose distribution in the visualization;

[0149] Create a 3D visualization object:

[0150] Use a suitable 3D visualization library (such as Mayavi) to create a 3D visualization object; create a new 3D scene by calling the relevant functions of the library (such as the mlab.figure function), in which the 3D dose cloud map will be displayed;

[0151] Draw a 3D dose cloud map:

[0152] The gridded dose data and the corresponding three-dimensional spatial coordinate information are passed to the three-dimensional visualization object, and the three-dimensional dose cloud map is drawn by calling the relevant functions of the library (such as the mlab.points3d function); during the drawing process, the color mapping of the cloud map (such as using the heat map color mapping to make the high-dose area appear red and the low-dose area appear blue, etc.), transparency (such as setting a certain degree of transparency in some low-dose areas to better observe the internal structure) and other properties can be set to enhance the visual effect;

[0153] Add annotation information such as axis labels (such as axis labeling "lateral position", axis labeling "longitudinal position", axis labeling "depth position"), titles (such as "three-dimensional dose cloud map"), etc., to better understand the content represented by the cloud map;

[0154] 4. Outlining of tumors and organs at risk

[0155] Image preparation:

[0156] For the automatically outlined result image, make sure it has been converted into a format suitable for visual display (such as numpy array format, etc.); if necessary, perform some basic processing on the image, such as adjusting the image contrast and brightness, to make the outline more clear and visible;

[0157] Show outlines:

[0158] Use a suitable visualization library (such as Matplotlib) to draw contours. For binary images of automatic contouring results (such as tumor area and normal tissue), you can draw contours by calling related functions of the library (such as contour function) to clearly display the contours of the tumor and organs at risk on the image. If the automatic contouring result is a segmented image with probability values, you can convert it into a binary image according to the set threshold and then draw contours.

[0159] Set the appearance attributes of the contour line, such as the color of the contour line of the tumor area (for example, red for the contour line of the tumor area, green for the contour line of the organ at risk, etc.), line type (for example, solid line, dashed line, etc.), and add annotation information such as axis labels (for example, "horizontal position" for the horizontal axis and "vertical position" for the vertical axis), titles (for example, "outline of tumor and organ at risk"), so as to better understand the content represented by the image;

[0160] 5. Integrate and output visualization results

[0161] Integrate the visualization results such as the drawn dose distribution curve, 3D dose cloud map, and the outline of the tumor and organs at risk, for example, by combining them in one visualization interface (such as using the subplot function of Matplotlib to combine different visualization elements in one figure, or using specialized visualization software to integrate different visualization results in one window), so that clinicians can view and compare different aspects of information at the same time;

[0162] The integrated visualization results can be output to a computer screen for real-time viewing, or saved as an image file (such as PNG, JPEG, etc.) or video file (such as MP4 format, if dynamic visualization is involved) for subsequent viewing, analysis, and archiving.

[0163] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning, characterized by: include: Data acquisition module; the data acquisition module acquires multimodal imaging data and clinical information of the patient; Automatic delineation module; the automatic delineation module receives multimodal imaging data, uses multimodal information fusion technology to integrate the particularity and complementarity of multimodal imaging data, uses a multimodal segmentation network framework based on a knowledge distillation framework, uses the soft target of the teacher model to guide the student model training, automatically delineates the structure of the multimodal imaging data, and obtains the automatic delineation results of the tumor target area and the organs at risk; A dose prediction model construction module; the dose prediction model construction module receives the automatic delineation result, extracts features through an encoder based on a UNetr network model that integrates the Transformer module and the UNet advantage, and a decoder fuses the features and generates multiple dose prediction channels including uncertainty weights and predicted dose results, and each dose prediction channel has two branches, which respectively generate uncertainty weights and predicted dose results; A diversified dose generation module; the diversified dose generation module includes a multi-branch prediction method based on shared trunk features and a triplet loss function constraint mechanism, which guides the consistency of multiple branch dose prediction results through trunk feature information sharing, and each branch learns diversified results through a convolutional layer. At the same time, the triplet loss function is used to constrain the multi-branch model learning process to generate diversified dose distributions; Model training and optimization module: The model training and optimization module uses training algorithms and optimization strategies to train and optimize the dose prediction model. During the training process, the training set data is used to update the model parameters, and the validation set data is used to evaluate the performance of the model and adjust the hyperparameters. Result output and visualization module: The result output and visualization module receives the automatic delineation results and the diversified dose distribution for visualization, and the visualization methods include dose distribution curve, three-dimensional dose cloud map, and the delineation contour display of tumors and organs at risk.

2. The multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning according to claim 1, characterized in that: In the automatic outlining module, the distillation loss function of the knowledge distillation framework is defined as ,in, Represents the number of feature dimensions output by the teacher model and the student model, represents the index of the sample, and are the outputs of the teacher model and the student model respectively, and the segmentation network based on the encoder-decoder architecture is integrated in the two processing branches of the network framework, and the cross entropy loss is used as the segmentation loss function. The cross entropy loss function is defined as ,in, is the sample size, is the true label, is the predicted label.

3. The multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning according to claim 2, characterized in that: In the multi-branch prediction method based on shared trunk features, trunk feature information sharing ensures that multiple branch dose prediction results can achieve diversified learning on the basis of commonality.

4. The multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning according to claim 3 is characterized by: The steps of the dose prediction model construction module for dose prediction are as follows: Step S1, receiving the automatic delineation result outputted from the automatic delineation module, the automatic delineation result includes the delineation information of the tumor target area and the organ at risk, the delineation information of the tumor target area and the organ at risk is presented in the form of an image, wherein each pixel corresponds to the labeling information of whether it belongs to a tumor or a specific organ at risk, and the automatic delineation result is used as input data and sent to the UNetr network model; Step S2: In the encoder part of the UNetr network model, the Transformer module is used to extract features from the input data. The Transformer module captures the long-distance dependencies and global context information in the input data based on its self-attention mechanism. Suppose the input data is X, the self-attention calculation process in the Transformer module is as follows: The input X is mapped into query vector Q, key vector K and value vector V respectively through linear transformation, the formula is: , , ,in, , , is a learnable weight matrix; Step S3: Calculate attention score , the formula is: ;in, is the key vector The dimension of is the key vector The transpose of The function is used to normalize the attention score to interval, so that the sum of each row is 1; Step S4: Perform weighted summation of the value vectors using the attention score to obtain the output processed by the self-attention mechanism , the formula is: After being processed by multiple layers of Transformer modules, an intermediate feature representation with rich global feature information is obtained. ; Step S5: While the Transformer module extracts features, the encoder of the UNetr network model also follows the UNet architecture style to perform downsampling operations. Downsampling is achieved by combining convolutional layers and pooling layers. After each downsampling, the features extracted from different levels are fused. The features obtained after downsampling are , and compare it with the features extracted by the Transformer module at the previous level Fusion is performed by concatenation or element-level addition to obtain the fused features ; Step S6: In the decoder part, the features extracted from different levels of the encoder are transferred to the corresponding level of the decoder using skip connections. The characteristics of the level are It is passed to the corresponding level of the decoder through the skip connection, which is recorded in the decoder as ; Step S7: In the decoder, upsampling is performed using a convolutional layer operation to restore the resolution of the data. After each upsampling, the upsampled features are combined with the features of the corresponding level transmitted through the jump connection. Fusion is performed to obtain the fused features , through multiple upsampling and feature fusion operations, the features are restored to a resolution close to the original input data, and feature information at different levels is fused; Step S8: After the decoder operation, the final feature representation that combines features at different levels is obtained. , based on the final feature representation , multiple dose prediction channels are generated through a series of fully connected layers or convolutional layers. Let the number of generated dose prediction channels be , then for each dose prediction channel , , the output of the channel is obtained through the corresponding network layer operation; Step S9: For each dose prediction channel , its output has two branches, which are used to generate uncertainty weights and predicted dose results : Step S9-1: Generate uncertainty weights Branch: The final feature representation is obtained through a sub-network structure of a small network containing several fully connected layers or convolutional layers. Processing to obtain uncertainty weight ; Step S9-2: Generate predicted dose results Branch: The final feature representation is achieved through the sub-network structure Processing to obtain the predicted dose results , predicted dose results is the radiation therapy dose value predicted by the model for each voxel point.

5. The multi-target dose prediction system for advanced nasopharyngeal carcinoma based on deep learning according to claim 4, characterized in that: In the triple loss function constraint mechanism, the triple loss function is ,in, represents anchor data, represents the dose prediction that matches the anchor data, represents the dose prediction that matches the anchor data, represents a preset interval, , Represents the calculation method of a set of data feature distances. When the calculation method is the Euclidean distance calculation method, the formula for the Euclidean distance calculation is: ,in, and are two data points, and Represents data points and In the The value of the feature dimension, It is a data point and The number of features.

Images