Head and neck cancer local recurrence prediction method based on clinical data text coding and multi-mechanism domain adversarial learning

Abstract

The invention discloses a head and neck cancer local recurrence prediction method based on clinical data text coding and multi-mechanism domain adversarial learning. The method specifically comprises the steps of sample set division and image preprocessing; expanding multi-view data; performing text coding on the clinical data; constructing a multi-mechanism domain adversarial learning network based on fusion text coding; training the constructed multi-mechanism domain adversarial learning network to obtain a coding network of the CT image; according to the obtained coding network, coding the CT image data and carrying out pooling coding; a classifier network based on multi-modal decoupling bilinear pooling fusion is constructed; the classifier is trained; a multi-modal domain adversarial learning mode is adopted to train an encoder, auxiliary training in a domain discriminator is used for capturing domain invariant features needing to be used in a prediction task, visual features are enhanced by including cross-modal interaction in a classifier, class invariant features used in the prediction task are better captured, and the prediction efficiency is improved. And meanwhile, the trained feature extraction network can be used in a plurality of downstream tasks, so that the prediction capability is better, and the generalization capability is stronger.

Description

Technical Field

[0001] The present invention belongs to the technical field of clinical auxiliary prediction, and in particular relates to a method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning. Background Art

[0002] Many existing methods for predicting the prognosis of head and neck cancer are mainly used on single-institution datasets. However, due to the small amount of medical data, data from many institutions often have to be used for training and generalized to new datasets. However, differences in instruments or operators in different institutions lead to differences in data, which affects the generalization effect. At the same time, in the existing multimodal methods for predicting the prognosis of head and neck cancer, the use of multimodal data generally adopts a simple superposition of CT images and PET images or a simple superposition of features. This simple superposition method cannot effectively extract the complementary features of different modalities, and it is easy to extract many redundant features. It also ignores the importance of clinical data, which affects the generalization results of the final classification.

[0003] Local regional recurrence is one of the main reasons for treatment failure in head and neck cancer. After chemoradiotherapy, 15%-50% of head and neck cancer patients will still experience local recurrence. Accurately predicting local recurrence after radiotherapy is crucial to achieving better treatment outcomes for head and neck cancer patients by developing personalized treatment strategies. Due to the heterogeneity and complexity of the anatomical structure of head and neck cancer, non-invasive artificial intelligence tools that help doctors make clinical decisions will better help personalized medicine.

[0004] When predicting local recurrence of head and neck cancer, the amount of available annotated medical data is small, so the prediction model cannot learn more sufficient features and the generalization ability is not high. At the same time, the medical data types are rich, and how to mine and utilize the complementary information of multimodal medical images to design a robust, efficient and accurate head and neck cancer prognosis prediction model also has important theoretical research value and good clinical application prospects.

[0005] A method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-institutional domain adversarial learning (publication number: CN115272230A) uses data from three modalities: computed tomography (CT), positron emission tomography (PET) and clinical data. First, two-dimensional views are extracted from the original three-dimensional CT and PET image data. Then, the output obtained after the first fusion of the CT and PET views is fused with the clinical data and input into the end-to-end neural network for training. The disadvantage of this method is that the encoding of clinical data is simple hard coding, which requires manual pre-coding, and the encoded data is discrete. The compatibility with the increase in clinical data fields is not strong enough, and the scalability is insufficient. At the same time, this method directly mixes the data of different institutions, and the generalization performance of the network is insufficient, and the generalization performance of the network trained under multi-institutional data under realistic conditions cannot be guaranteed. . Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-institutional domain adversarial learning, which effectively integrates CT images and clinical data information, and extracts domain generalization prediction methods of inter-domain invariant features to improve the performance of head and neck cancer prognosis prediction models under multiple institutions; the encoder is trained in a multimodal domain adversarial learning manner, and auxiliary training is performed in the domain discriminator module to capture domain invariant features needed in prediction tasks. In the classifier network, visual features are enhanced by including cross-modal interactions to better capture class invariant features used in prediction tasks. At the same time, the trained feature extraction network can be used in multiple downstream tasks, with better prediction ability and stronger generalization ability.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-institutional domain adversarial learning, specifically comprising the following steps:

[0009] Step S1, preprocessing of multi-institutional data sets: taking a number of positive samples of CT image data of head and neck cancer local recurrence and a number of negative samples of CT image data of head and neck cancer without local recurrence from A+1 medical institutions, dividing the positive and negative samples into a training set and a validation set, wherein the positive and negative samples of A institutions are used as the training set, and the positive and negative samples of the remaining institution are used as the validation set; balancing the positive and negative samples by resampling; converting the grayscale values ​​of the CT image data in the positive and negative samples into CT values ​​respectively and normalizing them; resampling the normalized CT image data to the same resolution by using a linear interpolation method;

[0010] Step S2, multi-view expansion of the multi-institutional data set: obtaining CT three-dimensional image views and corresponding contour label views at different angles along the z-axis, i.e., the vertical axis, of the CT image data processed in step S1, and projecting all three-dimensional image views and contour label views into two-dimensional images; based on the contour label view, intercepting the projected two-dimensional image view, unifying the size of the intercepted two-dimensional image view, and using it as the input of the feature extraction network;

[0011] Step S3, text encoding the patient's clinical data: Based on the collected patient clinical data, the clinical data encoding of each patient sample is constructed through the CLIP network, and the clinical data encoding is used as the text modality input of the classifier;

[0012] Step S4, constructing a multi-institutional domain adversarial learning network based on fusion text coding according to the clinical data coding outputted in step S3;

[0013] Step S5, encoding the clinical data outputted in step S3 and the CT image data processed in step S2, and training the two together to obtain an encoding network for the CT image;

[0014] Step S6, according to the encoding network obtained in step S5, the CT image data processed in step S2 is pooled and encoded as the CT image modality input of the classifier;

[0015] Step S7, constructing a classifier network based on multimodal decoupling bilinear pooling fusion: fusing the feature vectors of the text modality of the classifier obtained in step S3 and the CT image modality of the classifier obtained in step S6 to obtain a final classification result of whether the head and neck cancer has locally recurred;

[0016] Step S8: Train the classifier by using the Leave-one-out cross-validation method, select all samples in one domain as the validation set in turn, and use the positive and negative samples of the remaining A domains for training; during each training process, save the model with the highest AUC value on the validation set as the optimal model; obtain the prediction probability of the entire A+1 validation set P=P1+P2+P3…P from the corresponding optimal model A ,P A+1 , which is the predicted probability of the entire data set.

[0017] The specific method of step S1 is: obtaining m positive samples of local recurrence of head and neck cancer and n negative samples of no local recurrence of head and neck cancer from patient data of A+1 medical institutions, dividing the positive and negative samples into training set and validation set, wherein the positive and negative samples of A institutions are used as training set, and the positive and negative samples of the remaining institution are used as validation set; balancing the positive and negative samples by resampling; converting the grayscale values ​​of the CT image data in the positive and negative samples into CT values ​​respectively and normalizing them; the normalization formula is as follows:

[0018]

[0019] Wherein, R represents the CT image data after normalization, I represents the CT image data before normalization, max(I) and min(I) represent the maximum value and minimum value operations respectively; the CT image data is resampled to the same resolution using linear interpolation.

[0020] The specific method of step S2 is:

[0021] S201, performing data enhancement on the CT image data processed in step S1 to expand the training data, so as to avoid overfitting caused by using a small number of samples for training; using the original three-dimensional data for multi-view extraction; not only expanding the training samples, but also obtaining more tumor information; obtaining a CT three-dimensional image view and a corresponding contour label view of the CT image data processed in step S1;

[0022] The three-dimensional image view and the corresponding contour label view are respectively rotated along the Z axis, i.e., the vertical axis, to obtain CT three-dimensional image views and corresponding contour label views at different angles;

[0023] S202, projecting all CT three-dimensional image views and contour label views into two-dimensional images, and averaging corresponding pixel values ​​of the projected two-dimensional images along the horizontal axis of the CT three-dimensional image views and corresponding contour label views of different angles obtained in step S201;

[0024] S203: Finally, each sample obtains a two-dimensional image view and a contour label view. Based on the contour label view, the two-dimensional image view is cut into a uniform input sample size.

[0025] In step S201, the three-dimensional CT image and the contour label are rotated along the Z axis, ie, the vertical axis, from -75 degrees to 90 degrees.

[0026] The specific method of step S3 is:

[0027] Construct clinical data coding for each patient sample: collect the patient's clinical data, including age, primary site (including larynx, nasopharynx, oropharynx, hypopharynx), HPV status (including positive, negative, unknown status), treatment method (including radiotherapy, chemotherapy), T stage (including T1, T2, T3, T4) and N stage (including N0, N1, N2, N3), process the clinical data of each patient through Python script, and input it into a multimodal pre-trained neural network Contrastive Language-Image Pre-Training (hereinafter referred to as CLIP) in a unified and standardized form to encode the clinical data; output the dimensional features of each clinical data as the clinical data coding result of each patient.

[0028] The specific method of step S4 is:

[0029] Construct a multi-domain adversarial learning network based on fusion text encoding; specifically, the following steps are included:

[0030] S401, constructing an encoder module of a multi-institution image network, inputting the CT image data of multiple medical institutions processed in step S2 into an encoding network based on a residual network, and obtaining corresponding feature vectors through a global average pooling layer;

[0031] S402, constructing a feature dimension reduction module including 3 fully connected layers, reducing the dimension features obtained after processing the encoder module in S401, so as to calculate the loss in a low-dimensional space, and outputting the CT image features for subsequent fusion with the clinical data features;

[0032] S403, construct a domain discriminator module including a gradient reversal layer, fuse the CT image feature vector output in S401 with the dimensional features of the clinical data output in S3, and use them as the input of the domain discriminator module including a gradient reversal layer; the domain discriminator module including a gradient reversal layer contains two fully connected layers, and outputs the features of the domain discriminant feature vector; complete the construction of a multi-institutional domain adversarial learning network based on fused text encoding.

[0033] The step S5 specifically includes the following steps:

[0034] According to the clinical data encoding result outputted in step S3 and the CT image data processed in step S2, the two are used together to train the multi-institutional domain adversarial learning network based on fusion text encoding constructed in step S4; during the training process, the classification loss function, the domain classification loss function, the inter-domain covariance alignment loss function and the risk variance extrapolation loss function are used together as the objective function;

[0035] Classification loss function L class for:

[0036]

[0037] Where N is the total number of samples, y i is the true label of the i-th sample, is the predicted probability distribution output for the i-th sample;

[0038] Domain classification loss function L domain for:

[0039]

[0040] Among them, d i is the domain label to which the i-th sample belongs, The probability distribution of the predicted domain output by the i-th sample;

[0041] Inter-domain covariance alignment loss function L CORAL for:

[0042]

[0043] in, represents the Frobenius norm, C S and C T The calculation is as follows:

[0044]

[0045] Among them, D S is the source domain sample set, n s is the number of source domain samples, D T is the target domain sample set, n T is the number of samples in the target domain, d is the data dimension, 1 is a column vector with all elements equal to 1;

[0046] Risk variance extrapolation loss function L V-REx for:

[0047]

[0048] Among them, Var(·) calculates the variance, θ is the network parameter, m is the number of domains, R(θ) is the risk function, and l(·) is the common cross entropy loss, which is calculated as follows:

[0049]

[0050] The final total loss function is:

[0051] L total =L class +L domain +λ1L CORAL +λ2LV-REx

[0052] Among them, λ1 and λ2 are the proportions of the inter-domain covariance alignment loss function and the risk variance extrapolation loss function respectively;

[0053] During the training process, the stochastic gradient descent method is used for optimization. First, the partial derivative of θ is obtained using the loss function J(θ). The parameter θ is updated in the direction of the negative gradient, θ′ is the updated network parameter, θ j is the network parameter before updating, σ is the learning rate, is the training data input to the network, h θ (x i ) is the weight of the training set, y i is the label corresponding to the training set, m is the number of samples input for each training, a group of samples are randomly selected from the training set, and updated according to the gradient descent rule after each training.

[0054] The specific method of step S6 is:

[0055] Step S601, extracting features from different views of each CT image data obtained in step S2 through the encoding network trained in step S5, and extracting the CT image data obtained in step S2 through the corresponding encoding network to obtain feature vectors of the dimensions of the corresponding views;

[0056] Step S602: construct a view pooling layer, and perform view maximum pooling fusion on the feature vector of the dimension of the corresponding view obtained in step S601, that is, for each element of the feature vector of the dimension, take the maximum value in the corresponding view to obtain the dimension feature after maximum pooling fusion; the fusion method is expressed as follows:

[0057]

[0058] in, represents the i-th eigenvalue of the eigenvector of the j-th view, m i is the i-th eigenvalue of the fused feature vector, and the final fused feature m is:

[0059] m=[m1,m2,...,m 2048 ].

[0060] The specific method of step S7 is:

[0061] S701, using the clinical data encoding result obtained in step S3 as the text modality of the classifier and the feature vector of the CT image modality of the classifier obtained in step S6 to obtain the final classification result; modeling the local pairwise feature interaction through bilinear pooling and translation invariance, using second-order statistical information, and performing classification in a higher dimension when the first-order information is inseparable;

[0062] S702, construct a CT and clinical code multimodal decoupling bilinear pooling fusion (MFB) module, and reduce the dimension of the features of the fused CT image obtained in step S602 by a multilayer perceptron (MLP) to obtain a feature vector; the obtained feature vector and the clinical code obtained in step S301 are used as inputs of MFB to obtain fused dimensional features, and the final input logical value is obtained by passing through one layer of MLP;

[0063] The construction method of the multimodal decoupling bilinear pooling fusion module is as follows:

[0064] The module is divided into a feature expansion stage and a contraction stage. Assume that the multimodal decoupled bilinear pooling module fuses the feature vectors of the two modes into an o-dimensional vector. In the feature expansion stage, the bilinear pooling method will bring about the problems of high feature dimension and high computational complexity. Therefore, it is necessary to decouple it into two low-rank matrices based on the projection matrix of the original bilinear pooling method. The calculation formula is as follows:

[0065]

[0066] in, is the output of the bilinear model, x and y are the input feature vectors of the two modes, and is the decoupled learnable weight matrix, k is the dimension, For Hadamard, Is a full 1 vector, in order to get the final output vector The corresponding third-order vector needs to be learned and Simply resize U and V to 2D matrices and get:

[0067]

[0068] in, is the intermediate feature after taking the Hadamard product; after obtaining z′, a dropout layer is added to avoid overfitting;

[0069] In the feature shrinkage stage, the sum pooling operation is performed, and the formula is as follows:

[0070]

[0071] Among them, SumPooling(x, k) means using a one-dimensional window of size k without overlap to perform sum pooling on the input x;

[0072] A power regularization layer (z←sign(z)|z| 0.5 ) and a two-norm regularization layer (z←z / ||z||).

[0073] In the process of training the classifier network in step S8, the cross entropy loss function is used. Train the classifier network, defined as:

[0074]

[0075] Among them, y is the label and p is the predicted probability of the classifier network.

[0076] Compared with the prior art, the present invention has the following advantages:

[0077] 1. Design of encoder network: A novel text encoding and domain generalization learning method is adopted. The clinical data is encoded through a text encoder. Then, CT and clinical data are used as input data to extract domain-invariant features in domain adversarial learning and class-invariant features in the classification network. On the basis of generalizing the domain-invariant features extracted by single-modal domain, the text modality of clinical data is added. The mutual guidance of feature extraction between modalities can extract complementary information between multiple modalities, enhance visual features, and better capture class-invariant features used in prediction tasks. At the same time, the trained feature extraction network can be used in multiple downstream tasks.

[0078] 2. Construction of domain discriminator module: We found that clinical data itself is independent of the CT image domains of different institutions. Therefore, the features encoded by the clinical data are integrated into the domain discriminator. The process of integrating the features encoded by the clinical data into the domain discriminator can assist the discriminator in learning domain-independent features in image features, which can improve the generalization ability of the network. The present invention pays more attention to solving the problem of poor generalization ability of the network in reality, and the proposed multi-institutional domain adversarial learning training method is more effective.

[0079] 3. Design of classifier network: The decoupled bilinear pooling fusion method is used to perform decoupled bilinear pooling fusion on the features extracted from CT and clinical data. At the same time, the fused image features and clinical features are again decoupled bilinear pooling fusion. The local pairwise feature interaction is modeled in a translation-invariant manner. Compared with simple splicing operations, second-order statistical information is used, and classification can be performed in a higher dimension when the first-order information is inseparable.

[0080] 4. The method proposed in the present invention achieves better generalization prediction effect. Compared with other inventions that integrate CT, PET and clinical data, without the assistance of PET information, the average AUC reaches 0.74 and the average ACC reaches 0.81.

[0081] In summary, the present invention adopts a multimodal domain adversarial learning method to train the encoder, and auxiliary training in the domain discriminator is used to capture the domain invariant features needed in the prediction task. In the classifier, the visual features are enhanced by including cross-modal interactions to better capture the class invariant features used in the prediction task. At the same time, the trained feature extraction network can be used in multiple downstream tasks, with better prediction ability and stronger generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a flow chart of the present invention.

[0083] Figure 2 It is a flow chart of the encoder network and classifier network of the present invention.

[0084] Figure 3 This is a flow chart of a module for generating multi-view data in the present invention.

[0085] Figure 4 This is a flow chart of the multimodal decoupling bilinear pooling fusion module in the present invention. DETAILED DESCRIPTION

[0086] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0087] See attached Figure 1 , a local recurrence prediction method for head and neck cancer based on clinical data text encoding and multi-institutional domain adversarial learning, comprising the following steps:

[0088] S1. Preprocessing of multi-institutional datasets

[0089] Matlab software was used to process 562 patient data from 6 institutions (hospitals), including 100 positive samples with local recurrence and 462 negative samples without local recurrence. The 6 institutions are CHUM (6 positive, 56 negative), CHUS (14 positive, 79 negative), HGJ (12 positive, 65 negative), HMR (6 positive, 13 negative), UTSW (49 positive, 157 negative), HN1 (13 positive, 92 negative). Each time the validation is performed, one of the institution data is taken as the validation set, and the other five institution data are taken as the training set. During the training process, the positive samples are resampled 5 times to balance the positive and negative samples.

[0090] The grayscale values ​​of the CT data are converted into CT values ​​and normalized. The normalization formula is as follows:

[0091]

[0092] Wherein, R represents the CT image after normalization, I represents the CT image before normalization, max(I) and min(I) represent the maximum value and minimum value operations respectively.

[0093] The images were resampled to the same resolution of 1×1×1mm using linear interpolation. 3 .

[0094] S2. Multi-view expansion of multi-institutional datasets

[0095] Due to the scarcity of data samples, the CT image data processed in step S1 is augmented to expand the training data to avoid overfitting caused by using few samples for training. Using traditional random data augmentation will not add extra information. Using the original 3D data for multi-view extraction not only expands the training samples, but also obtains more tumor information. Please refer to Figure 3 , specifically including the following steps:

[0096] S201, rotating the three-dimensional CT image and the contour label along the Z axis (vertical axis) from -75 degrees to 90 degrees at intervals of 15 degrees, to obtain 12 CT three-dimensional image views and corresponding contour label views at different angles;

[0097] S202, projecting the three-dimensional image view and the corresponding contour label view into a two-dimensional image, and averaging the corresponding pixel values ​​of the projected two-dimensional images along the horizontal axis of the 12 CT three-dimensional image views and the corresponding contour label views obtained in step S201 at different angles;

[0098] S203. Finally, each sample obtains 12 two-dimensional image views and contour label views. Based on the contour label view, the image view is cut into a size of 200×200 to unify the size of the input sample.

[0099] S3. Text encoding of clinical data

[0100] Construct clinical data encoding for each patient sample as input to the classifier. Clinical data include age, primary site (including larynx, nasopharynx, oropharynx, hypopharynx), HPV status (including positive, negative, unknown status), treatment (including radiotherapy, chemotherapy), T stage (including T1, T2, T3, T4) and N stage (including N0, N1, N2, N3); the clinical data of each patient is processed by Python script and standardized as "A{age}-year-old patient with head and neck cancer has a tumor primary site located in{primarySite}, whose tumorstage is{tStage}, the node stage is{nStage}, the treatment is{therapy}, and theHPV status is{hpvStatus}." Input into a multimodal pre-trained neural network Contrastive Language-Image Pre-Training, hereinafter referred to as CLIP, to encode clinical data, and each data output becomes a 512×1 dimensional feature.

[0101] S4. Constructing a multi-domain adversarial learning network based on fusion text encoding

[0102] In order to extract domain-invariant features and use auxiliary information from other modalities when extracting modality features, a multi-institutional domain adversarial network method based on fused clinical data is used to construct an encoding network. Figure 2 The specific steps include:

[0103] S401, construct an encoding network module for a multi-institutional image network, input the CT image data of multiple medical institutions processed in step S2 into the encoding network based on the residual network (ResNet50), and obtain the corresponding feature vector after the global average pooling layer.

[0104] S402: Build a feature dimension reduction module containing three fully connected layers, reduce the dimension of a 2048-dimensional feature obtained after processing by the S401 encoder module to calculate the loss in a low-dimensional space, and output a feature of size 512×1. The CT image features output by this step will be fused with the clinical data features in the multimodal decoupling bilinear pooling module in S7.

[0105] S403, construct a domain discriminator module including a gradient reversal layer, fuse the CT image feature vector output in S401 with the dimensional features of the clinical data output in S3, and use them as the input of the domain discriminator module including a gradient reversal layer; the domain discriminator module including a gradient reversal layer contains two fully connected layers, and outputs features of the number of organizations, i.e., 6×1 dimensional features; complete the construction of a multi-organization domain adversarial learning network based on fused text encoding.

[0106] The 2048×1-dimensional features obtained from the CT modality are reduced in dimension to output features of size 512×1 for subsequent fusion with clinical data features. The features encoded in S402 are used in the inference stage, and the domain discriminator module in S403 is not used. This module only assists network training and learns domain-independent features in the data. The process of fusing the features encoded from clinical data into the domain discriminator can assist the discriminator in learning domain-independent features in image features, which can improve the generalization ability of the network.

[0107] S5. Train the encoding network constructed in S4

[0108] According to the clinical data encoding result outputted in step S3 and the CT image data processed in step S2, the two are used together to train the encoding network constructed in step S4; during the training process, the classification loss function, the domain classification loss function, the inter-domain covariance alignment loss function and the risk variance extrapolation loss function are used together as the objective function;

[0109] Classification loss function L class for:

[0110]

[0111] Where N is the total number of samples, y i is the true label of the i-th sample, is the predicted probability distribution output for the i-th sample.

[0112] Domain classification loss function L domain for:

[0113]

[0114] Among them, d i is the domain label to which the i-th sample belongs, The probability distribution of the predicted domain output for the i-th sample.

[0115] Inter-domain covariance alignment loss function L CORAL for:

[0116]

[0117] in, represents the Frobenius norm, C S and C T The calculation is as follows:

[0118]

[0119] Among them, D S is the source domain sample set, n s is the number of source domain samples, D T is the target domain sample set, n T is the number of samples in the target domain, d is the data dimension, and 1 is a column vector with all elements equal to 1.

[0120] Risk variance extrapolation loss function L V-REx for:

[0121]

[0122] Among them, Var(·) calculates the variance, θ is the network parameter, m is the number of domains, R(θ) is the risk function, and l(·) is the common cross entropy loss, which is calculated as follows:

[0123]

[0124] The final total loss function is:

[0125] L total =L class +L domain +λ1L CORAL +λ2L V-REx

[0126] Among them, λ1 and λ2 are the proportions of the inter-domain covariance alignment loss function and the risk variance extrapolation loss function, respectively.

[0127] During the training process, the stochastic gradient descent method is used for optimization. First, the partial derivative of θ is calculated using the loss function J(θ). The parameter θ is updated in the direction of the negative gradient, θ′ is the updated network parameter, θ j is the network parameter before updating, σ is the learning rate, is the training data input to the network, h θ (x i ) is the weight of the training set, y i is the label corresponding to the training set, m is the number of samples input for each training, a group of samples are randomly selected from the training set, and updated according to the gradient descent rule after each training.

[0128] S6. Pooling encoding of multi-view data

[0129] The image data is encoded by the encoding network trained in Example S5. Specifically, the following steps are included:

[0130] Step S601, extracting features from different views of each CT image data obtained in step S2 through the encoding network trained in step S5, and after the CT image data obtained in step S2 is extracted through the corresponding encoding network, 2048×1-dimensional feature vectors of 12 views are obtained respectively;

[0131] S602, construct a view pooling layer, and perform view maximum pooling fusion on the 2048×1-dimensional features of the 12 views of the CT modality obtained in step S601, that is, for each element of the 2048×1-dimensional, take the maximum value of the 12 views at the corresponding position to obtain the 2048×1-dimensional features after maximum pooling fusion. The fusion method is shown as follows:

[0132]

[0133] in, represents the i-th eigenvalue of the eigenvector of the j-th view, m i is the i-th eigenvalue of the fused feature vector, and the final fused feature m is:

[0134] m=[m1,m2,...,m 2048 ]

[0135] S7. Construct a classifier network based on multimodal decoupling and bilinear pooling fusion

[0136] Construct a multimodal decoupled bilinear pooling fusion classifier network, use the clinical data encoding results obtained in Example S3 as the text mode of the classifier, and the feature vector fusion of the CT image mode of the classifier obtained in S6 to obtain the final classification result. The bilinear pooling method models the local pairwise feature interaction in a translationally invariant manner. Compared with the simple splicing operation, it uses second-order statistical information, and can be classified in a higher dimension when the first-order information is inseparable. Specifically, it includes the following steps:

[0137] S701, construct a multimodal decoupling bilinear pooling fusion (MFB) module for CT images and clinical codes, and reduce the 2048×1-dimensional features of the CT image obtained in step S6 by a multilayer perceptron (MLP) to obtain 512×1-dimensional features. The 512-dimensional image feature vector and the 512-dimensional clinical code obtained in step S301 are used as inputs of MFB to obtain fused 10×1-dimensional features, which are passed through 1 layer of MLP to obtain the final input logical value.

[0138] See also Figure 4, the multi-modal decoupling bilinear pooling fusion module operates as follows:

[0139] The module is divided into the feature expansion stage and the contraction stage. Assume that the multimodal decoupled bilinear pooling module fuses the feature vectors of the two modes into an o-dimensional vector. In the feature expansion stage, the bilinear pooling method will bring about the problems of high feature dimension and high computational complexity, so it is necessary to decouple it into two low-rank matrices based on the projection matrix of the original bilinear pooling method. The calculation formula is as follows:

[0140]

[0141] in, is the output of the bilinear model, x and y are the input feature vectors of the two modes, and is the decoupled learnable weight matrix, is the dimension, For Hadamard, Is a full 1 vector, in order to get the final output vector The corresponding third-order vector needs to be learned and Simply resize U and V to 2D matrices and get:

[0142] in, is the intermediate feature after taking the Hadamard product. After obtaining z′, a dropout layer is added to avoid overfitting.

[0143] In the feature shrinkage stage, the sum pooling operation is performed, and the formula is as follows:

[0144]

[0145] Among them, SumPooling(x, k) means using a one-dimensional window with no overlap of size k to perform sum pooling on the input x.

[0146] Because the element-wise Hadamard product layer is added, the values ​​of the neurons at the final output may be very different, which may lead to a local optimum. Therefore, a power regularization layer (z←sign(z)|z| 0.5 ) and a two-norm regularization layer (z←z / ||z||).

[0147] In step S701 of the embodiment, k=3, o=10.

[0148] S8. Train the classifier using the Leave-one-out cross-validation method, select all samples in one domain as the validation set in turn, and use the positive and negative samples of the remaining five domains for training; during each training process, save the model with the highest AUC value on the validation set as the optimal model; from the corresponding optimal model, the predicted probability of the six validation sets as a whole is p = [p1, p2, p3, p4, p5, p6], that is, the predicted probability of the entire data set. The final AUC mean of 0.76 and ACC mean of 0.81 are calculated by p. The model has a good performance in predicting the accuracy of local recurrence classification results of head and neck cancer in different institutions; during the training of the classifier, the loss function uses the cross entropy loss function Defined as:

[0149]

[0150] Among them, y is the label and p is the predicted probability of the classifier network.

[0151] The classic models in the field of domain generalization and the models proposed in recent years are used to compare with the prediction method proposed in the present invention. The name of the institution in the table indicates the model performance results with the institution as the test domain and the remaining other institutions as the training domain. Table 1 shows the model accuracy, hereinafter referred to as ACC; Table 2 shows the area under the ROC curve of the model, Area Under the Curve. Hereinafter referred to as AUC. The model results with the best performance in each institution have been bolded. It can be seen from Tables 1 and 2 that the performance of the method proposed in the present invention far exceeds that of other models in multiple institutions. Although the performance in individual institutions is not optimal, it is not much different from the optimal, and the average ACC and average AUC are the highest, reaching 0.81 and 0.76 respectively. It is worth mentioning that the present invention is based on the second comparison method DANN, the full name of which is Domain-Adversarial Neural Networks. The results show that the multi-institutional domain adversarial learning network trained after combining clinical data text encoding has the best performance in the prediction of local recurrence of head and neck cancer, and improves the results of the head and neck cancer prognosis prediction task with different data sources.

[0152] Table 1 Comparison results of ACC experiments

[0153]

[0154] Table 2 AUC experimental comparison results

[0155]

[0156]

[0157] The present invention provides a method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-institutional domain adversarial learning. The clinical data modality is added to the original domain adversarial network method, and the extracted image features and clinical features are respectively fused using a decoupled bilinear pooling method, thereby improving the results of prognosis prediction of head and neck cancer from institutions with different data sources.

[0158] The model in the present invention adopts a modular design concept. The basic network structures in the feature extractor and the classifier can be replaced. Modules can be added or modified according to the deficiencies of the basic network in the prognosis prediction task. With the development of emerging technologies and the proposal of better network modules, the present invention can be iteratively updated at any time to improve model performance.

Claims

1. A method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-institutional domain adversarial learning, characterized in that: The specific steps include: Step S1, preprocessing of multi-institutional data sets: taking a number of positive samples of CT image data of head and neck cancer local recurrence and a number of negative samples of CT image data of head and neck cancer without local recurrence from A+1 medical institutions, dividing the positive and negative samples into a training set and a validation set, wherein the positive and negative samples of A institutions are used as the training set, and the positive and negative samples of the remaining institution are used as the validation set; balancing the positive and negative samples by resampling; converting the grayscale values ​​of the CT image data in the positive and negative samples into CT values ​​respectively and normalizing them; resampling the normalized CT image data to the same resolution by using a linear interpolation method; Step S2, multi-view expansion of the multi-institutional data set: obtaining CT three-dimensional image views and corresponding contour label views at different angles along the z-axis, i.e., the vertical axis, of the CT image data processed in step S1, and projecting all three-dimensional image views and contour label views into two-dimensional images; based on the contour label view, intercepting the projected two-dimensional image view, unifying the size of the intercepted two-dimensional image view, and using it as the input of the feature extraction network; Step S3, text encoding the patient's clinical data: Based on the collected patient clinical data, the clinical data encoding of each patient sample is constructed through the CLIP network, and the clinical data encoding is used as the text modality input of the classifier; Step S4, constructing a multi-institutional domain adversarial learning network based on fusion text coding according to the clinical data coding outputted in step S3; Step S5, encoding the clinical data outputted in step S3 and the CT image data processed in step S2, and training the two together to obtain an encoding network for the CT image; Step S6, according to the encoding network obtained in step S5, the CT image data processed in step S2 is pooled and encoded as the CT image modality input of the classifier; Step S7, constructing a classifier network based on multimodal decoupling bilinear pooling fusion: fusing the feature vectors of the text modality of the classifier obtained in step S3 and the CT image modality of the classifier obtained in step S6 to obtain a final classification result of whether the head and neck cancer has locally recurred; Step S8: Train the classifier by using the Leave-one-out cross-validation method, select all samples in one domain as the validation set in turn, and use the positive and negative samples of the remaining A domains for training; during each training process, save the model with the highest AUC value on the validation set as the optimal model; obtain the prediction probability of the entire A+1 validation set P=P1+P2+P3…P from the corresponding optimal model A ,P A+1 , which is the predicted probability of the entire data set.

2. According to claim 1, a method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning is characterized in that: The specific method of step S1 is: obtaining m positive samples of local recurrence of head and neck cancer and n negative samples of no local recurrence of head and neck cancer from patient data of A+1 medical institutions, dividing the positive and negative samples into training set and validation set, wherein the positive and negative samples of A institutions are used as training set, and the positive and negative samples of the remaining institution are used as validation set; balancing the positive and negative samples by resampling; converting the grayscale values ​​of the CT image data in the positive and negative samples into CT values ​​respectively and normalizing them; the normalization formula is as follows: Wherein, R represents the CT image data after normalization, I represents the CT image data before normalization, max(I) and min(I) represent the maximum and minimum operations respectively; the CT image data is resampled to the classic resolution of 1*1*1mm3 using linear interpolation.

3. According to claim 1, a method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning is characterized in that: The specific method of step S2 is: S201, performing data enhancement on the CT image data processed in step S1 to expand the training data, so as to avoid overfitting caused by using a small number of samples for training; The original three-dimensional data is used to perform multi-view extraction; the training samples are expanded, and more tumor information is obtained; the CT three-dimensional image view and the corresponding contour label view of the CT image data processed in step S1 are obtained; The three-dimensional image view and the corresponding contour label view are respectively rotated along the Z axis, i.e., the vertical axis, to obtain CT three-dimensional image views and corresponding contour label views at different angles; S202, projecting all CT three-dimensional image views and contour label views into two-dimensional images, and averaging corresponding pixel values ​​of the projected two-dimensional images along the horizontal axis of the CT three-dimensional image views and corresponding contour label views of different angles obtained in step S201; S203: Finally, each sample obtains a two-dimensional image view and a contour label view. Based on the contour label view, the two-dimensional image view is cut into a uniform input sample size.

4. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: In step S201, the three-dimensional CT image and the contour label are rotated along the Z axis, ie, the vertical axis, from -75 degrees to 90 degrees.

5. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: The specific method of step S3 is: Construct clinical data coding for each patient sample: collect the patient's clinical data, including age, primary site (including larynx, nasopharynx, oropharynx, hypopharynx), HPV status (including positive, negative, unknown status), treatment method (including radiotherapy, chemotherapy), T stage (including T1, T2, T3, T4) and N stage (including N0, N1, N2, N3), process the clinical data of each patient through Python script, and input it into a multimodal pre-trained neural network Contrastive Language-Image Pre-Training (hereinafter referred to as CLIP) in a unified and standardized form to encode the clinical data; output the dimensional features of each clinical data as the clinical data coding result of each patient.

6. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: The step S4 constructs a multi-domain adversarial learning network based on fusion text encoding, which specifically includes the following steps: S401, constructing an encoder module of a multi-institution image network, inputting the CT image data of multiple medical institutions processed in step S2 into an encoding network based on a residual network, and obtaining corresponding feature vectors through a global average pooling layer; S402, constructing a feature dimensionality reduction module including three fully connected layers, reducing the dimensionality of the dimensional features obtained after processing by the encoder module in step S401, so as to calculate the loss in a low-dimensional space, and outputting the CT image features for subsequent fusion with the clinical data features; S403, constructing a domain discriminator module including a gradient reversal layer, fusing the CT image feature vector output in step S401 with the dimensional features of the clinical data output in step S3, and using the fusion as the input of the domain discriminator module including a gradient reversal layer; the domain discriminator module including a gradient reversal layer contains two fully connected layers, and outputs the features of the domain discriminant feature vector; Complete the construction of a multi-domain adversarial learning network based on fusion text encoding.

7. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: The step S5 specifically includes the following steps: According to the clinical data encoding result outputted in step S3 and the CT image data processed in step S2, the two are used together to train the multi-organization domain adversarial learning network based on fusion text encoding constructed in step S4; During the training process, the classification loss function, domain classification loss function, inter-domain covariance alignment loss function and risk variance extrapolation loss function are used as the objective function; Classification loss function L class for: Where N is the total number of samples, y i is the true label of the i-th sample, is the predicted probability distribution output for the i-th sample; Domain classification loss function L domain for: Among them, d i is the domain label to which the i-th sample belongs, The probability distribution of the predicted domain output by the i-th sample; Inter-domain covariance alignment loss function L CORAL for: in, represents the Frobenius norm, C S and C T The calculation is as follows: Among them, D S is the source domain sample set, n s is the number of source domain samples, D T is the target domain sample set, n T is the number of samples in the target domain, d is the data dimension, 1 is a column vector with all elements equal to 1; Risk variance extrapolation loss function L V-REx for: Among them, Var(·) calculates the variance, θ is the network parameter, m is the number of domains, R(θ) is the risk function, and l(·) is the common cross entropy loss, which is calculated as follows: The final total loss function is: L total =L class +L domain +λ1L CORAL +λ2L V-REx Among them, λ1 and λ2 are the proportions of the inter-domain covariance alignment loss function and the risk variance extrapolation loss function respectively; During the training process, the stochastic gradient descent method is used for optimization. First, the partial derivative of θ is calculated using the loss function J(θ). The parameter θ is updated in the direction of the negative gradient, θ′ is the updated network parameter, θ j is the network parameter before updating, σ is the learning rate, is the training data input to the network, h θ (x i ) is the weight of the training set, y i is the label corresponding to the training set, m is the number of samples input for each training, a group of samples are randomly selected from the training set, and updated according to the gradient descent rule after each training.

8. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: The specific method of step S6 is: Step S601, extracting features from different views of each CT image data obtained in step S2 through the encoding network trained in step S5, and extracting the CT image data obtained in step S2 through the corresponding encoding network to obtain feature vectors of the dimensions of the corresponding views; Step S602: construct a view pooling layer, and perform view maximum pooling fusion on the feature vector of the dimension of the corresponding view obtained in step S601, that is, for each element of the feature vector of the dimension, take the maximum value in the corresponding view to obtain the dimension feature after maximum pooling fusion; the fusion method is expressed as follows: in, represents the i-th eigenvalue of the eigenvector of the j-th view, m i is the i-th eigenvalue of the fused feature vector, and the final fused feature m is: m=[m1,m2,...,m 2048 ]。 9. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: The specific method of step S7 is: S701, using the clinical data encoding result obtained in step S3 as the text modality of the classifier and the feature vector of the CT image modality of the classifier obtained in step S6 to obtain the final classification result; modeling the local pairwise feature interaction through bilinear pooling and translation invariance, using second-order statistical information, and performing classification in a higher dimension when the first-order information is inseparable; S702, construct a CT and clinical code multimodal decoupling bilinear pooling fusion (MFB) module, and reduce the dimension of the features of the fused CT image obtained in step S602 by a multilayer perceptron (MLP) to obtain a feature vector; the obtained feature vector and the clinical code obtained in step S301 are used as inputs of MFB to obtain fused dimensional features, and the final input logical value is obtained by passing through one layer of MLP; The construction method of the multimodal decoupling bilinear pooling fusion module is as follows: The multimodal decoupled bilinear pooling fusion module is divided into a feature expansion stage and a contraction stage. It is assumed that the multimodal decoupled bilinear pooling module fuses the feature vectors of the two modes into an o-dimensional vector. In the feature expansion stage, the bilinear pooling method will bring about the problems of high feature dimension and high computational complexity. Therefore, it is necessary to decouple it into two low-rank matrices based on the projection matrix of the original bilinear pooling method. The calculation formula is as follows: in, is the output of the bilinear model, x and y are the input feature vectors of the two modes, and is the decoupled learnable weight matrix, k is the dimension, For Hadamard, is a vector of all 1s. In order to obtain the final output vector The corresponding third-order vector needs to be learned and Simply resize U and V to 2D matrices and get: in, is the intermediate feature after taking the Hadamard product; after obtaining z′, a dropout layer is added to avoid overfitting; In the feature shrinkage stage, the sum pooling operation is performed, and the formula is as follows: Among them, SumPooling(x, k) means using a one-dimensional window of size k without overlap to perform sum pooling on the input x; A power regularization layer (z←sign(z)|z| 0.5 ) and a two-norm regularization layer (z←z / ||z||).

10. The method for predicting local recurrence of head and neck cancer based on clinical data text encoding and multi-domain adversarial learning according to claim 1, characterized in that: In the process of training the classifier network in step S8, the cross entropy loss function is used. Train the classifier network, defined as: Among them, y is the label and p is the predicted probability of the classifier network.

Images