A head and neck cancer local recurrence information acquisition method based on multi-modal supervised contrast learning

Abstract

A head and neck cancer local recurrence information acquisition method based on multi-modal supervised contrast learning, specifically comprising: sample set division and image preprocessing, multi-view data enhancement, constructing an image coding network based on multi-modal supervised contrast learning, training the constructed coding network, view pooling fusion image coding, encoding clinical data according to an attribute-based hybrid coding strategy, constructing a classifier network based on multi-modal decoupling bilinear pooling fusion, and training the constructed classifier network; the original contrast learning method is increased based on the multi-modal structure, the intra-modal and inter-modal contrast losses are optimized together, label information is added, the decoupling bilinear pooling mode is used to fuse the extracted image features, and the accuracy of the head and neck cancer recurrence signal acquisition result is improved.

Description

Technical Field

[0001] This invention belongs to the field of clinical auxiliary prediction technology, and in particular relates to a method for obtaining information on local recurrence of head and neck cancer based on multimodal supervised contrastive learning. Background Technology

[0002] Local recurrence is one of the leading causes of treatment failure in head and neck cancers. Between 15% and 50% of head and neck cancer patients still experience local recurrence after radiotherapy and chemotherapy. Accurately predicting local recurrence after radiotherapy is crucial for achieving better treatment outcomes for head and neck cancer patients through personalized treatment strategies. Due to the heterogeneity and complexity of the anatomy in head and neck cancer, non-invasive artificial intelligence tools that assist physicians in clinical decision-making will better support personalized medicine.

[0003] When predicting local recurrence in head and neck cancer, the limited availability of labeled medical data prevents predictive models from learning sufficient features, resulting in poor generalization ability. Meanwhile, the rich variety of medical data necessitates the mining and utilization of complementary information from multimodal medical images to design robust, efficient, and accurate prognostic prediction models for head and neck cancer.

[0004] Existing methods for predicting the prognosis of head and neck cancer typically employ a simple overlay of raw data or features for multimodal data. This simple overlay approach fails to effectively extract complementary features from different modalities and easily extracts many redundant features, affecting the final classification results.

[0005] A method for predicting local recurrence in head and neck cancer based on a multi-modal, multi-view convolutional neural network (Journal No.: Guo, Jinkun, Rongfang Wang, Zhiguo Zhou, Kai Wang, Rongbin Xu, and Jing Wang. "Multi-Modality and Multi-View 2DCNN to Predict Locoregional Recurrence in Head & Neck Cancer." In 2021 International Joint Conference on Neural Networks (IJCNN), pp.1-7. IEEE, 2021.) utilizes data from three modalities: computed tomography (CT), positron emission tomography (PET), and clinical data. First, two-dimensional views are extracted from the raw 3D CT and PET images. Then, the CT and PET views are superimposed and input along with the clinical data into an end-to-end neural network for training. A limitation of this method is that simple superposition of image data from different modalities is not sufficient to effectively extract complementary features from these different modalities. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to propose a method for acquiring local recurrence information of head and neck cancer based on multimodal supervised contrastive learning. On the basis of the original contrastive learning method, a multimodal structure is added to jointly optimize the contrast loss within and between modalities. At the same time, label information is added, and the extracted image features, as well as image and clinical features, are fused using a decoupled bilinear pooling method, thereby improving the results of acquiring head and neck cancer recurrence signals.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for acquiring local recurrence information in head and neck cancer based on multimodal supervised contrastive learning, specifically including the following steps:

[0009] S1. Divide the sample set of head and neck cancer patient data and preprocess CT and PET images;

[0010] S2. Perform multi-view data augmentation on the image processed in step S1;

[0011] S3. Construct an image coding network based on multimodal supervised contrastive learning;

[0012] S4, the encoding network constructed in training step S3;

[0013] S5, View Pooling Fusion Image Encoding;

[0014] S6. Encode the clinical data according to an attribute-based hybrid coding strategy;

[0015] S7. Construct a classifier network based on multimodal decoupling bilinear pooling fusion;

[0016] S8. Train the classifier network constructed in step S7.

[0017] The specific method of step S1 is as follows: using Matlab software, the positive samples of head and neck cancer patients with local recurrence and the negative samples of head and neck cancer patients without local recurrence are randomly and equally divided into training set and validation set, and then cross-validated multiple times.

[0018] The grayscale values ​​of CT and PET data are converted into CT values ​​and SUV values ​​respectively, and then normalized. The normalization formula is as follows:

[0019]

[0020] Where R represents the normalized CT or PET image, I represents the CT or PET image before normalization, and max(I) and min(I) represent the maximum and minimum value operations, respectively; the images are resampled using linear interpolation to obtain the same resolution.

[0021] The specific method for step S2 is as follows:

[0022] Data augmentation is performed on the images processed in step S1 to expand the training data and avoid overfitting caused by training with few samples. Traditional random data augmentation is used, which does not add extra information. Multi-view extraction is performed using the original 3D data, which not only expands the training samples but also obtains more tumor information. The specific steps include the following:

[0023] S201. Rotate the 3D CT and PET images and contour labels along the Z-axis, i.e., the vertical axis, to obtain 3D image views of CT and PET and corresponding contour label views at different angles.

[0024] S202. Average the corresponding pixel values ​​of all slices along the horizontal axis for the obtained three-dimensional image view and the corresponding outline label view, and project all three-dimensional image views and outline label views into two-dimensional images.

[0025] S203. Finally, each sample obtains a two-dimensional image view and a contour label view. Based on the contour label view, the image view is cropped to the size of the input sample.

[0026] The specific method for step S3 is as follows:

[0027] The input image after S2 processing is used to construct an encoding network using a class-based multimodal contrastive learning method, specifically including the following steps:

[0028] S301. Construct an input processing module for multimodal contrast learning. For the input two-dimensional image view of CT or PET, generate an enhanced image through random data augmentation methods, such as one or more combinations of cropping, rotation, flipping, and grayscale changes.

[0029] S302. Construct the encoder module for multimodal contrast learning. The "dual tower" structure of simCLR is used in both CT and PET modalities. Taking the contrast learning network in the CT modal as an example, the enhanced images generated in step S301 are input into the encoder based on the layer residual network (ResNet50). After passing through the global average pooling layer, two feature vectors are obtained. The encoding process of the PET modal is the same as that of the CT modal.

[0030] S303. Construct a feature dimensionality reduction module containing a fully connected layer to reduce the dimensionality of the two features obtained from CT and PET modalities respectively, so as to calculate the loss in the low-dimensional space.

[0031] Dimensionality reduction is performed on the two features obtained from CT and PET modalities respectively to calculate the intra-modal and inter-modal contrast loss. In the intra-modal case, an image is used as an anchor point to find all images of the same class in the same batch and treat them as positive samples. All images of different classes are treated as negative samples, so that positive samples are closer to each other and negative samples are further apart. In the inter-modal case, all images of different classes in the same batch are found, so that the distance between images of different classes is greater than the distance between images of the same class. During the inference stage, only the encoding module in S302 is used, and the dimensionality reduction module is not used.

[0032] The specific method for step S4 is as follows:

[0033] The image coding network based on multimodal supervised contrastive analysis constructed in step S3 is trained to obtain coding networks for CT and PET respectively; during the training process, the sum of intra-modal and inter-modal supervised contrastive loss functions is used as the objective function;

[0034] Intramodal supervised contrastive loss function for:

[0035]

[0036] Where N is the total number of samples, Let be the label of the sample, l(·) be the indicator function, which is 1 when the condition in (·) is met, otherwise it is 0, i is the current sample, j is the sample with the same label as i but different from i, z is the feature vector in the loss space, k is the sample among all samples that is different from i, and τ is the temperature parameter.

[0037] Intermodal supervised comparison loss function for:

[0038]

[0039] Where α is the marginal parameter, c represents the CT sample, p represents the PET sample, and s(c, p) is the similarity function between the CT sample and the PET sample, calculated as follows:

[0040]

[0041]

[0042]

[0043] Wherein, f(c; W) f θ φ )and The joint embedding space mappings for CT and PET are φ(c; θ). φ )and The feature representations extracted from CT and PET are θ, respectively. φ and W represents network parameters. f and W g As weight;

[0044] The final total loss function is:

[0045]

[0046] Wherein, λ1 and λ2 are the proportions of intramodal and intermodal losses, respectively;

[0047] During training, stochastic gradient descent is used for optimization; first, the partial derivative of the loss function J(θ) with respect to θ is calculated. The parameter θ is updated in the direction of the negative gradient. θ′ represents the updated network parameters, θ j These are the network parameters before the update, where σ is the learning rate. h is the training data input to the network. θ (x i y represents the weights of the training set. i The labels are the labels corresponding to the training set, and m is the number of samples input for each training iteration. A set of samples is randomly selected from the training set and updated according to the gradient descent rule after each training iteration.

[0048] Step S5 encodes the image data using the encoding network trained in step S4, specifically including the following steps:

[0049] S501. The different views of the CT and PET modal of each sample obtained in step S2 are used to extract features through the encoder trained in step S4. After the CT or PET modal is extracted by the corresponding encoder, the feature vectors of CT and PET are obtained respectively.

[0050] S502. Construct a view pooling layer and perform view max pooling fusion on the features of the obtained CT or PET modality. That is, for each element of the feature vector, take the maximum value in the 12 views at the corresponding position to obtain the feature after max pooling fusion.

[0051] The fusion method is represented as follows:

[0052]

[0053] in, Let m represent the i-th eigenvalue of the eigenvector of the j-th view. iLet be the i-th feature value of the fused feature vector. The final fused feature m is:

[0054] m = [m1, m2, ..., m 2048 ]

[0055] The specific method for step S6 is as follows:

[0056] Each patient sample's clinical data is encoded and used as input to the classifier. Clinical data includes age, primary site (including larynx, nasopharynx, oropharynx, hypopharynx), HPV status (including positive, negative, and unknown status), treatment method (including radiotherapy and chemotherapy), T stage (including T1, T2, T3, T4), and N stage (including N0, N1, N2, N3). Age is a numerical variable, and its encoding is directly divided by 100 to normalize it. Primary site, HPV status, and treatment method are categorical variables, using one-hot encoding. T stage and N stage are ordinal variables; their original order is first converted to numerical order using integer encoding, and then normalized to a single-bit code by dividing by the maximum value of the corresponding feature.

[0057] The specific method of step S7 is as follows: The feature vectors of the three modalities obtained in steps S5 and S6 are fused to obtain the final classification result; the bilinear pooling method models local pairwise feature interactions in a translation-invariant manner, using second-order statistical information, and performing classification in a higher dimension when first-order information is inseparable, specifically including the following steps:

[0058] S701. Construct a multimodal decoupled bilinear pooling fusion (MFB) module for CT and PET image coding. Use the features of CT and PET modalities obtained in step S5 as input to the MFB to obtain the fused features. After dimensionality reduction by a multilayer perceptron (MLP), the feature vector after image modality fusion is obtained.

[0059] S702. Construct a multimodal decoupled bilinear pooling fusion module for image coding and clinical coding. Take the image feature vector and the clinical coding obtained in step S5 as input, and pass through two layers of MLP to obtain the final input logical value.

[0060] The construction of the multimodal decoupled bilinear pooling fusion module in steps S701 and S702 is as follows: The multimodal decoupled bilinear pooling fusion module operation is as follows:

[0061] The module is divided into a feature expansion stage and a contraction stage. It is assumed that the multimodal decoupled bilinear pooling module fuses the feature vectors of two modes into an o-dimensional vector. In the feature expansion stage, bilinear pooling leads to high feature dimensionality and computational complexity. Therefore, based on the projection matrix of the original bilinear pooling method, it needs to be decoupled into two low-rank matrices, calculated as follows:

[0062]

[0063] in, This is the output of the bilinear model, where x and y are the input feature vectors of the two modes, respectively. and The learnable weight matrix is ​​for decoupling, where k is the dimension. For Hadama accumulation, It is a vector of all 1s, in order to obtain the final output vector The corresponding third-order vector needs to be learned. and Simply adjust the dimensions of U and V to form a 2D matrix. and get:

[0064]

[0065] in, To obtain the intermediate features after the Hadamard product; after obtaining z′, a dropout layer is added to avoid overfitting;

[0066] During the feature shrinking phase, pooling operations are performed, as shown in the following formula:

[0067]

[0068] Wherein, SumPooling(x, k) represents sum pooling on the input x using a one-dimensional non-overlapping window of size k;

[0069] An exponential regularization layer (z←sign(z)|z|) was added after the SumPooling layer. 0.5 ) and the L2 regularization layer (z←z / ||z||).

[0070] The specific method for step S8 is as follows:

[0071] During the training of the classifier network, the cross-entropy loss function is used. Defined as:

[0072]

[0073] Where y is the label and p is the predicted probability of the classifier network.

[0074] Compared with the prior art, the present invention has the following advantages:

[0075] 1. The encoder network design adopts a novel multimodal supervised contrastive learning method, using CT and PET modalities as input data. Label information is embedded in the contrastive learning process, which can extract class-invariant features, making them better suited for classification tasks. Contramodal and intermodal contrastive learning are performed separately. Based on the extraction of classification-friendly features from single-modal contrastive learning, intermodal feature extraction is mutually guided, which can extract complementary information between multiple modalities, effectively improving the separability of data encoding. At the same time, the trained feature extraction network can be used in multiple downstream tasks.

[0076] 2. In the design of the classifier network, the decoupled bilinear pooling fusion method is used to perform decoupled bilinear pooling fusion on the features extracted from CT and PET. At the same time, the fused image features and clinical features are decoupled bilinear pooling fusion again. Local pairwise feature interactions are modeled in a translation-invariant manner. Compared with simple stitching operation, second-order statistical information is used. When first-order information is inseparable, classification can be performed in a higher dimension.

[0077] 3. The model in this invention adopts a modular design approach. The basic network structures in the feature extractor and classifier can be replaced. Modules can be added or modified according to the shortcomings of the basic network in the prognostic prediction task. With the development of emerging technologies and the proposal of better network modules, this invention can be iteratively updated at any time to improve model performance. Attached Figure Description

[0078] Figure 1 This is a flowchart of the present invention;

[0079] Figure 2 This is a flowchart of the encoder network and classifier network of the present invention;

[0080] Figure 3 Here is a flowchart of the multi-view data generation module in this invention;

[0081] Figure 4 This is a flowchart of the multimodal decoupling bilinear pooling fusion module in this invention; Detailed Implementation

[0082] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0083] See appendix Figure 1 A method for obtaining local recurrence information in head and neck cancer based on multimodal supervised contrastive learning includes the following steps:

[0084] S1. Divide the sample set of head and neck cancer patients and preprocess CT and PET images.

[0085] Using Matlab software, data from 206 head and neck cancer patients were randomly divided into 5 equal parts, such as [10; 10; 10; 10; 9] and [31; 32; 32; 31; 31]. During each cross-validation, 4 positive and 1 negative samples were used as the training set and 1 positive and 1 negative sample was used as the validation set.

[0086] The grayscale values ​​of CT and PET data are converted into CT values ​​and SUV values ​​respectively, and then normalized. The normalization formula is as follows:

[0087]

[0088] Where R represents the normalized CT or PET image, I represents the CT or PET image before normalization, and max(I) and min(I) represent the maximum and minimum value operations, respectively.

[0089] The image was resampled to the same resolution of 1×1×1mm using linear interpolation. 3 ;

[0090] S2. Perform multi-view data augmentation on the image processed in step S1.

[0091] Due to the scarcity of data samples, data augmentation is necessary to expand the training data and avoid overfitting caused by training with few samples. Traditional random data augmentation does not add extra information; however, using the original 3D data for multi-view extraction not only expands the training samples but also obtains more tumor information. Please refer to [link to relevant documentation]. Figure 3 Specifically, it includes the following steps:

[0092] S201. Rotate the 3D CT and PET images and contour labels at 15-degree intervals along the Z-axis (vertical axis) from -75 degrees to 90 degrees to obtain a total of 12 different angle 3D image views of CT and PET and corresponding contour label views.

[0093] S202. Average the corresponding pixel values ​​of all slices along the horizontal axis for the obtained 12 three-dimensional image views and corresponding contour label views, and project all three-dimensional image views and contour label views into two-dimensional images.

[0094] S203. Finally, each sample will have 12 two-dimensional image views and contour label views. Based on the contour label views, the image views will be cropped to a size of 200×200, and the size of the input sample will be uniform.

[0095] S3. Construct an image coding network based on multimodal supervised contrastive learning.

[0096] To extract class-invariant features and utilize auxiliary information from other modalities during modality feature extraction, a class-based multimodal contrastive learning method is employed to construct the encoding network; please refer to [link to relevant documentation]. Figure 2 Specifically, it includes the following steps:

[0097] S301. Construct an input processing module for multimodal contrast learning. For a two-dimensional image view of an input CT or PET image, generate two enhanced images through random data augmentation methods, such as one or more combinations of cropping, rotation, flipping, and grayscale changes.

[0098] S302. Construct the encoder module for multimodal contrastive learning. The "dual tower" structure of simCLR is used in both CT and PET modalities. Taking the contrastive learning network in the CT modal as an example, the two enhanced images generated in step S301 are respectively input into the encoder based on a 50-layer residual network (ResNet50). After passing through a global average pooling layer, two 2048×1-dimensional features are obtained. The encoding process of the PET modal is the same as that of the CT modal.

[0099] S303. Construct a feature dimensionality reduction module containing a fully connected layer to reduce the dimensionality of the two 2048×1-dimensional features obtained from CT and PET modalities respectively, so as to calculate the loss in the low-dimensional space and output two features of size 128×1-dimensional.

[0100] The encoded feature vectors retain information related to data augmentation transformations. The role of the feature dimensionality reduction module is to remove this information, allowing the learned representation to revert to the essence of the data and preventing the loss function for similarity calculation from losing some important features during training. Dimensionality reduction is performed on the two 2048×1 dimensional features obtained from the CT and PET modalities respectively, resulting in two 128×1 feature vectors for each modality, used to calculate intra-modal and inter-modal contrast loss. Within a modality, an image is used as an anchor point; all images of the same class within the same batch are identified and treated as positive samples, while images of different classes are treated as negative samples, bringing positive samples closer together and negative samples further apart. Between modalities, all images of different classes within the same batch are identified, ensuring that the distance between images of different classes is greater than the distance between images of the same class. During the inference phase, only the encoding module in S302 is used to obtain the 2048-dimensional features; the dimensionality reduction module is not used.

[0101] S4, the encoding network constructed in training step S3

[0102] The image coding network based on multimodal supervised contrastive analysis constructed in step S3 is trained to obtain coding networks for CT and PET, respectively. During training, the sum of intra-modal and inter-modal supervised contrastive loss functions is used as the objective function.

[0103] Intramodal supervised contrastive loss function for:

[0104]

[0105] Where N is the total number of samples, Let be the label of the sample, l(·) be the indicator function, which is 1 when the condition in (·) is met, otherwise it is 0, i is the current sample, j is the sample with the same label as i but different from i, z is the feature vector in the loss space, k is the sample among all samples that is different from i, and τ is the temperature parameter.

[0106] Intermodal supervised comparison loss function for:

[0107]

[0108] Where α is the marginal parameter, c represents the CT sample, p represents the PET sample, and s(c, p) is the similarity function between the CT sample and the PET sample, calculated as follows:

[0109]

[0110]

[0111]

[0112] Wherein, f(c; W) f θ φ )and The joint embedding space mappings for CT and PET are φ(c; θ). φ )and The feature representations extracted from CT and PET are θ, respectively. φ and W represents network parameters. f and W g As weight;

[0113] The final total loss function is:

[0114]

[0115] Wherein, λ1 and λ2 are the proportions of intramodal and intermodal losses, respectively;

[0116] During training, stochastic gradient descent is used for optimization; first, the partial derivative of the loss function J(θ) with respect to θ is calculated. The parameter θ is updated in the direction of the negative gradient. θ′ represents the updated network parameters, θ j These are the network parameters before the update, where σ is the learning rate. h is the training data input to the network. θ (x i y represents the weights of the training set. i The labels are the labels corresponding to the training set, and m is the number of samples input for each training iteration. A set of samples is randomly selected from the training set and updated according to the gradient descent rule after each training iteration.

[0117] S5, View Pooling Blend Image Encoding

[0118] The image data is encoded using the encoding network trained in step S4. This specifically includes the following steps:

[0119] S501. Extract features from the 12 different views of the CT and PET modal of each sample obtained in step S2 using the encoder trained in step S4. After the CT or PET modal is extracted by the corresponding encoder, 2048×1-dimensional feature vectors of the 12 views are obtained respectively.

[0120] S502. Construct a view pooling layer. Perform view max pooling fusion on the 2048×1-dimensional features of the 12 views of the obtained CT or PET modality. That is, for each element of the 2048×1 dimension, take the maximum value in the 12 views at the corresponding position to obtain the 2048×1-dimensional features after max pooling fusion.

[0121] The fusion method is represented as follows:

[0122]

[0123] in, Let m represent the i-th eigenvalue of the eigenvector of the j-th view. i Let be the i-th feature value of the fused feature vector. The final fused feature m is:

[0124] m = [m1, m2, ..., m 2048 ]

[0125] S6. Encode the clinical data using an attribute-based hybrid coding strategy.

[0126] Each patient sample's clinical data was encoded and used as input to the classifier. The clinical data included six types: age, primary site (including larynx, nasopharynx, oropharynx, and hypopharynx), HPV status (including positive, negative, and unknown status), treatment method (including radiotherapy and chemotherapy), T stage (including T1, T2, T3, and T4), and N stage (including N0, N1, N2, and N3). Age was a numerical variable, and its encoding was directly normalized by dividing by 100. Primary site, HPV status, and treatment method were categorical variables, encoded using one-hot encoding. T stage and N stage were ordinal variables; their original order was first converted to numerical order using integer encoding, then normalized to a one-bit code by dividing by the maximum value of the corresponding feature. The final result was a 12-dimensional feature vector. Specific encoding examples are shown in Table 1.

[0127] Table 1. Coding instructions for the six clinical variables.

[0128]

[0129]

[0130] S7. Construct a classifier network based on multimodal decoupling bilinear pooling fusion.

[0131] A multimodal decoupled bilinear pooling fusion classifier network is constructed, fusing the feature vectors of the three modalities obtained in steps S5 and S6 to obtain the final classification result. Bilinear pooling models local pairwise feature interactions in a translation-invariant manner. Compared to simple concatenation, it uses second-order statistical information, allowing for classification in a higher dimension when first-order information is inseparable. Specifically, the following steps are included:

[0132] S701. Construct a multimodal decoupled bilinear pooling fusion (MFB) module for CT and PET image coding. Use the 2048×1-dimensional features of the CT and PET modal obtained in step S5 as the input of MFB to obtain the fused 1000×1-dimensional features. After dimensionality reduction by two layers of multilayer perceptron (MLP), a 12×1-dimensional feature vector after image modality fusion is obtained.

[0133] S702. Construct a multimodal decoupled bilinear pooling fusion module for image coding and clinical coding. Take the 12-dimensional image feature vector and the 12-dimensional clinical coding obtained in step S5 as inputs, and pass through two layers of MLP to obtain the final input logical value.

[0134] Please see Figure 4The specific implementation methods of steps S701 and S702 are the same, only the input data and parameter settings are different. The operation of constructing the multimodal decoupled bilinear pooling fusion module described in steps S701 and S702 is as follows:

[0135] The module is divided into a feature expansion stage and a contraction stage. It is assumed that the multimodal decoupled bilinear pooling module fuses the feature vectors of two modes into an o-dimensional vector. In the feature expansion stage, bilinear pooling leads to high feature dimensionality and computational complexity. Therefore, based on the projection matrix of the original bilinear pooling method, it needs to be decoupled into two low-rank matrices, calculated as follows:

[0136]

[0137] in, This is the output of the bilinear model, where x and y are the input feature vectors of the two modes, respectively. and The learnable weight matrix is ​​for decoupling, where k is the dimension. For Hadama accumulation, It is a vector of all 1s, in order to obtain the final output vector The corresponding third-order vector needs to be learned. and Simply adjust the dimensions of U and V to form a 2D matrix. and get:

[0138]

[0139] in, To obtain the intermediate features after the Hadamard product, a dropout layer was added after obtaining z′ to avoid overfitting.

[0140] During the feature shrinking phase, pooling operations are performed, as shown in the following formula:

[0141]

[0142] Wherein, SumPooling(x, k) represents sum pooling on the input x using a one-dimensional non-overlapping window of size k;

[0143] Because of the added element-wise Hadamard layers, the values ​​at the final output neurons may vary significantly, potentially leading to local optima. Therefore, a power-regularization layer (z←sign(z)|z|) is added after the SumPooling layer. 0.5 And the L2 regularization layer (z←z / ||z||);

[0144] In step S701, k = 5 and o = 1000; in step S702, k = 3 and o = 10.

[0145] S8. Train the classifier network constructed in step S7.

[0146] During the training of the classifier network, the cross-entropy loss function is used. Defined as:

[0147]

[0148] Where y is the label and p is the predicted probability of the classifier network.

[0149] Evaluation Model

[0150] One positive sample and one negative sample are selected sequentially as the validation set, and the remaining four positive and negative samples are used for training, for a total of 5 training iterations. During each training iteration, the model with the highest AUC value on the validation set is saved as the optimal model. The overall prediction probability p = [p1, p2, p3, p4, p5] of the five validation sets is obtained from the corresponding optimal model, which is the prediction probability of the entire dataset. The final AUC value is calculated using p to evaluate the prediction model.

[0151] This invention adds a multimodal structure to the original contrastive learning method to jointly optimize the contrast loss within and between modalities. It also incorporates label information and uses decoupled bilinear pooling to fuse the extracted image features, as well as image and clinical features, thereby improving the accuracy of the head and neck cancer recurrence signal acquisition results.

Claims

1. A method for acquiring local recurrence information in head and neck cancer based on multimodal supervised contrastive learning, characterized in that, Specifically, the following steps are included: S1. Divide the sample set of head and neck cancer patient data and preprocess CT and PET images; S2. Perform multi-view data enhancement on the images processed in step S1, specifically including: rotating the 3D CT and PET images and the corresponding tumor contour labels along the Z-axis to obtain multiple 3D image views and corresponding contour label views at different angles; projecting the 3D image views and the corresponding contour label views onto 2D images along the horizontal axis; and cropping the 2D images to a uniform size based on the contour label views. S3. Construct an image coding network based on multimodal supervised contrastive learning, specifically including: for each two-dimensional image view obtained in S2, generate an enhanced image through random data augmentation; input the enhanced image into an encoder based on a residual network to obtain feature vectors; wherein, the multimodal supervised contrastive learning utilizes the recurrence label information of samples to treat all images of the same category as positive samples and images of different categories as negative samples in the same batch; S4. Train the image coding network based on multimodal supervised contrastive analysis constructed in step S3 to obtain the coding networks for CT and PET respectively; during the training process, the sum of intra-modal and inter-modal supervised contrastive loss functions is used as the objective function; Intramodal supervised contrastive loss function for: in, The total number of samples, For the label of the sample, For indicator functions, when The value is 1 when the condition is met, and 0 otherwise. For the current sample, To and The same and different labels The sample, For the feature vectors in the loss space, For all samples that are different The sample, For temperature parameters; Intermodal supervised comparison loss function for: in, For edge parameters, Indicates CT sample, Indicates PET sample, The similarity function between CT and PET samples is calculated as follows: in, and These are the joint embedding space mappings for CT and PET, respectively. and These are the feature representations extracted from CT and PET, respectively. and Represents network parameters, and As weight; The final total loss function is: in, and These represent the proportions of intramodal and intermodal losses, respectively. S5, View Pooling Fusion Image Encoding, specifically includes: extracting features from multiple different views of each sample obtained in S2 using the encoder trained in S4, and performing max pooling fusion on the feature vectors of multiple views of the same modality. S6. Encode the clinical data according to an attribute-based hybrid coding strategy; S7. Construct a classifier network based on multimodal decoupled bilinear pooling fusion, specifically including: First, construct a multimodal decoupled bilinear pooling fusion (MFB) module for CT and PET image encoding, taking the fused features of CT and PET modalities obtained in S5 as input to obtain the feature vector after image modal fusion; then, construct a multimodal decoupled bilinear pooling fusion module for image encoding and clinical encoding, taking the feature vector after image modal fusion and the clinical encoding obtained in S6 as input; S8. Train the classifier network constructed in step S7.

2. The method for acquiring local recurrence information in head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, The specific method of step S1 is as follows: using Matlab software, the positive samples of head and neck cancer patients with local recurrence and the negative samples of head and neck cancer patients without local recurrence are randomly and equally divided into training set and validation set, and then cross-validated multiple times. The grayscale values ​​of CT and PET data are converted into CT values ​​and SUV values ​​respectively, and then normalized. The normalization formula is as follows: in, This represents a normalized CT or PET image. This represents the CT or PET image before normalization. These represent operations to take the maximum and minimum values, respectively; the image is resampled using linear interpolation to obtain the same resolution.

3. The method for acquiring local recurrence information in head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, The specific method for step S2 is as follows: Data augmentation is performed on the images processed in step S1 to expand the training data, specifically including the following steps: S201. Rotate the 3D CT and PET images and contour labels along the Z-axis (vertical axis) to obtain 3D image views of CT and PET at different angles and corresponding contour label views. S202. Average the corresponding pixel values ​​of all slices along the horizontal axis for the obtained three-dimensional image view and the corresponding outline label view, and project all three-dimensional image views and outline label views into two-dimensional images. S203. Finally, each sample obtains a two-dimensional image view and a contour label view. Based on the contour label view, the image view is cropped to the size of the input sample.

4. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, The specific method for step S3 is as follows: The input image after S2 processing is used to construct an encoding network using a class-based multimodal contrastive learning method, specifically including the following steps: S301. Construct an input processing module for multimodal contrast learning. For the input two-dimensional image view of CT or PET, generate an enhanced image through one or more combinations of random data augmentation methods, including cropping, rotation, flipping, and grayscale changes. S302. Construct an encoder module for multimodal contrast learning. The "dual tower" structure of simCLR is used in both CT and PET modalities. Taking the contrast learning network in the CT modal as an example, the enhanced images generated in step S301 are input into the encoder based on the residual network. After passing through the global average pooling layer, two feature vectors are obtained. The encoding process of the PET modal is the same as that of the CT modal. S303. Construct a feature dimensionality reduction module containing a fully connected layer to reduce the dimensionality of the two features obtained from CT and PET modalities respectively, so as to calculate the loss in the low-dimensional space. Dimensionality reduction is performed on the two features obtained from CT and PET modalities respectively to calculate the intra-modal and inter-modal contrast loss. In the intra-modal case, an image is used as an anchor point to find all images of the same class in the same batch and treat them as positive samples. All images of different classes are treated as negative samples, so that positive samples are closer to each other and negative samples are further apart. In the inter-modal case, all images of different classes in the same batch are found, so that the distance between images of different classes is greater than the distance between images of the same class.

5. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, In step S4 of the training process, stochastic gradient descent is used for optimization; firstly, a loss function is used... right Find the partial derivative. ,parameter Update in the direction of the negative gradient. , For the updated network parameters, For the network parameters before the update, For learning rate, For the training data input to the network, For the weights of the training set, The labels corresponding to the training set. The number of samples input for each training iteration is [number]. A set of samples is randomly selected from the training set and updated according to the gradient descent rule after each training iteration.

6. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, Step S5 encodes the image data using the encoding network trained in step S4, specifically including the following steps: S501. The different views of the CT and PET modal of each sample obtained in step S2 are used to extract features through the encoder trained in step S4. After the CT or PET modal is extracted by the corresponding encoder, the feature vectors of CT and PET are obtained respectively. S502. Construct a view pooling layer and perform view max pooling fusion on the features of the obtained CT or PET modality. That is, for each element of the feature vector, take the maximum value in the 12 views at the corresponding position to obtain the feature after max pooling fusion. The fusion method is represented as follows: in, Indicates the first The feature vector of the i-th view 1 eigenvalue, The first feature vector after fusion The final fused feature is obtained from the feature values. for: 。 7. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, The specific method for step S6 is as follows: Each patient sample's clinical data is encoded and used as input to the classifier. The clinical data includes age, primary site (larynx, nasopharynx, oropharynx, hypopharynx), HPV status (positive, negative, unknown), treatment method (radiotherapy, chemotherapy), T stage (T1, T2, T3, T4), and N stage (N0, N1, N2, N3). Age is a numerical variable, and its encoding is directly divided by 100 to normalize it. Primary site, HPV status, and treatment method are categorical variables, using one-hot encoding. T stage and N stage are ordinal variables. First, the original order is converted to numerical order using integer encoding, and then divided by the maximum value of the corresponding feature to normalize it into a one-bit code.

8. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, The specific method of step S7 is as follows: the feature vectors of the three modalities obtained in steps S5 and S6 are fused to obtain the final classification result; the bilinear pooling method models the local pairwise feature interaction in a translation-invariant manner, uses second-order statistical information, and performs classification in a higher dimension when the first-order information is inseparable, specifically including the following steps: S701. Construct a multimodal decoupled bilinear pooling fusion (MFB) module for CT and PET image coding. Use the features of CT and PET modalities obtained in step S5 as input to the MFB to obtain the fused features. After dimensionality reduction by a multilayer perceptron (MLP), the feature vector after image modality fusion is obtained. S702. Construct a multimodal decoupled bilinear pooling fusion module for image coding and clinical coding. The feature vector after image modality fusion and the clinical coding obtained in step S6 are used as inputs. After passing through two layers of MLP, the final input logical value is obtained.

9. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 8, characterized in that, The steps S701 and S702 for constructing the multimodal decoupled bilinear pooling fusion module are as follows: The module is divided into a feature expansion stage and a contraction stage. It is assumed that the multimodal decoupled bilinear pooling module fuses the feature vectors of two modes into an o-dimensional vector. In the feature expansion stage, bilinear pooling leads to high feature dimensionality and computational complexity. Therefore, based on the projection matrix of the original bilinear pooling method, it needs to be decoupled into two low-rank matrices, calculated as follows: in, It is the output of the bilinear model. and These are the input feature vectors for the two modalities, respectively. and For decoupled learnable weight matrices, As a dimension, For Hadama accumulation, It is a vector of all 1s, in order to obtain the final output vector It is necessary to learn the corresponding third-order vector. and ,Will and Simply adjust the dimensions to a 2D matrix and ,get: in, To obtain the intermediate features after the Hadamard product; after obtaining Then, to avoid overfitting, a dropout layer was added; During the feature shrinking phase, pooling operations are performed, as shown in the following formula: in, This indicates that a one-dimensional size is Non-overlapping windows in the input area Pooling is performed on top of the pool; A power regularization layer was added after the SumPooling layer. ) and the L2 regularization layer ( ).

10. The method for obtaining local recurrence information of head and neck cancer based on multimodal supervised contrastive learning according to claim 1, characterized in that, The specific method for step S8 is as follows: During the training of the classifier network, the cross-entropy loss function is used. Defined as: in, For tags, This represents the predicted probability of the classifier network.

Images