Rectal cancer neoadjuvant therapy efficacy prediction method and system based on multi-modal knowledge decomposition

Abstract

The present application belongs to the technical field of medical data processing, and discloses a rectal cancer neoadjuvant therapy efficacy prediction method and system based on multi-modal knowledge decomposition. The present application provides a multi-modal data fusion algorithm based on multi-modal knowledge decomposition, which decomposes multi-modal data into multiple independent modal knowledge, reduces redundant information, and does not lose important inter-modal complementary information. The present application simultaneously uses unsupervised clustering and supervised CNN algorithm. Compared with the traditional method of only using unsupervised random selection or image selection method based on information entropy, the present application can reduce the interference of noise data on picture screening, so as to select representative colonoscopy pictures and CT pictures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, in particular to a rectal cancer neoadjuvant therapy efficacy prediction method and system based on multi-modal knowledge decomposition. BACKGROUND

[0002] Rectal cancer is a common malignant tumor of the digestive tract, and its incidence is increasing globally year by year, especially in China. The incidence and mortality of rectal cancer remain high. The treatment strategy for rectal cancer usually includes surgery, radiotherapy and chemotherapy, and other comprehensive application of multiple ways. For locally advanced rectal cancer (such as stage II or III), the standard treatment regimen is preoperative neoadjuvant radiotherapy and chemotherapy. It can reduce the tumor and improve the local control rate by applying radiotherapy and chemotherapy before surgery, thereby improving the survival rate of patients. However, the efficacy of neoadjuvant radiotherapy and chemotherapy varies greatly among different patients. Some patients can achieve pathological complete remission, that is, the tumor completely disappears after radiotherapy and chemotherapy, while some patients have poor efficacy or even no effect. Therefore, how to effectively predict the efficacy of neoadjuvant therapy has become an important research topic in clinical research. Predicting the response of patients to neoadjuvant radiotherapy and chemotherapy before surgery not only optimizes the treatment regimen and reduces the economic burden of patients, but also avoids unnecessary treatment side effects.

[0003] Imaging plays an important role in predicting the efficacy of neoadjuvant therapy for rectal cancer. MRI (magnetic resonance imaging) and CT (computed tomography) are widely used to assess tumor size, infiltration, and changes in lymph nodes, especially MRI can provide more accurate imaging features for tumor staging and tumor regression. High-resolution MRI, colonoscopy, and other methods have become standard evaluation methods. In recent years, imageomics has gradually become a hot research direction. It extracts high-dimensional features from image data, combines with clinical data of patients, and builds a efficacy prediction model, which has shown good application prospects in many international researches.

[0004] However, the current rectal cancer neoadjuvant therapy efficacy prediction method has the following shortcomings. First, the current multi-modal method often adopts a simple fusion strategy, such as splicing, vector inner product, etc., without fully considering the heterogeneity between modalities, making it difficult to extract complementary information between modalities. Second, since rectal cancer patients often have multiple CT images and colonoscopy images, in order to reduce the computational pressure, most current methods often use unsupervised sampling methods such as random sampling, which makes it difficult to ensure that the extracted images are representative. SUMMARY

[0005] The present application aims to solve at least one of the above technical problems, and provides a rectal cancer neoadjuvant therapy efficacy prediction method and system based on multi-modal knowledge decomposition.

[0006] To achieve the above object, the application provides a rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition in the first aspect, comprising:

[0007] Obtain multi-modal medical data of a plurality of rectal cancer patients, and the multi-modal medical data of each patient comprises structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures, medical text data, and whether the rectal cancer after neoadjuvant therapy shows pathological complete remission;

[0008] Feature extraction is performed on the structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures, and medical text data respectively to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and cross-entropy loss of the clinical representation, colonoscopy representation, CT representation, and text representation is calculated;

[0009] Shared representation of the clinical representation, colonoscopy representation, CT representation, and text representation is extracted, and shared loss and shared representation average value are calculated according to the shared representation;

[0010] Modality-specific representation of the clinical representation, colonoscopy representation, CT representation, and text representation is extracted respectively to obtain clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation, and text modality-specific representation;

[0011] Orthogonal loss of the clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation, text modality-specific representation, and shared representation average value is calculated;

[0012] Cross-modal representation of the clinical representation, colonoscopy representation, CT representation, and text representation is extracted;

[0013] The shared representation average value, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modal representation are vector spliced to obtain multi-modal fusion representation;

[0014] Whether the patient shows pathological complete remission after rectal cancer neoadjuvant therapy is predicted according to the multi-modal fusion representation, and the neural network is updated according to the cross-entropy loss, shared loss, and orthogonal loss.

[0015] Preferably, the cross-entropy loss function L1 is:

[0016] L1=

[0017] crossEntopy(FC1(e c ),y)+crossEntopy(FC1(e CT ),y)+

[0018] crossEntopy(FC1(e m),y)+crossEntopy(FC1(e text ),y);

[0019] Among them, e c As a clinical manifestation, e CT For CT characterization, e m For colonoscopy characteristics, e text For text representation, FC1 is a fully connected network, and crossEntropy represents the binary classification cross-entropy loss function, whose expression is: in, 'a' represents the predicted value, and 'a' represents the true label.

[0020] Preferably, the shared loss function L2 is:

[0021] L2=(e share [0]-e share [1]) 2 +(e share [1]-e share [2]) 2 +(e share [2]-

[0022] e share [3]) 2 ;

[0023] Among them, e share [0] represents the shared representation corresponding to the clinical representation, e share [1] is the shared representation corresponding to the CT representation, e share [2] is the shared representation corresponding to the colonoscopy representation, e share [3] is the shared representation corresponding to the text representation.

[0024] Preferably, the orthogonal loss function L3 is:

[0025] L3 = e cp ⊙e ave +e CTp ⊙e ave +e mp ⊙e ave +e textp ⊙e ave ;

[0026] Among them, e cp For clinical modality-specific characterization, e CTp For CT modality-specific characterization, e mp For colonoscopy modality-specific characterization, e textp For text modality-specific characterization, e ave To represent the average value, ⊙ represents the vector dot product.

[0027] Preferably, the method for extracting the cross-modal representation of the clinical representation, the colonoscopy representation, the CT representation and the text representation is as follows:

[0028] Calculate the information entropy of the clinical representation, the colonoscopy representation, the CT representation and the text representation respectively.

[0029] Sort the clinical representation, the colonoscopy representation, the CT representation and the text representation from small to large according to the size of the information entropy, and fuse the intermediate representation obtained by fusing the latter representation with the former representation or the former several representations in turn.

[0030] Preferably, the calculation formula of the information entropy H is as follows:

[0031] Wherein, m is the length of the vector e, e i is the value of the vector e at position i.

[0032] Preferably, representative pictures in the colonoscopy pictures and the CT pictures are selected for feature extraction, and the selection method is as follows:

[0033] For each patient, cluster all the colonoscopy pictures or CT pictures to obtain K categories, and K is a hyperparameter.

[0034] Randomly select one colonoscopy picture or CT picture from each category to set a label, and obtain K labeled colonoscopy pictures or CT pictures for each patient, wherein the label is whether the rectal cancer after neoadjuvant therapy shows pathological complete remission.

[0035] Train the CNN neural network with all the labeled colonoscopy pictures or CT pictures of the patients.

[0036] Input the colonoscopy pictures or CT pictures of each patient into the above trained CNN neural network to obtain a prediction result.

[0037] Sort according to the size of the prediction result, and divide into K groups according to the sorting.

[0038] Randomly select one colonoscopy picture or CT picture from each group as a representative picture.

[0039] Preferably, the loss function L4 for training the CNN neural network is as follows:

[0040] Wherein, y m is the true label of the colonoscopy picture or CT picture, is the prediction value of the CNN neural network, which is a decimal number between 0 and 1.

[0041] Preferably, the method for predicting whether the patient shows pathological complete remission after rectal cancer neoadjuvant therapy according to the multi-modal fusion representation is as follows:

[0042] The multi-modal fusion representation is input into the updated neural network to obtain a prediction value, and the prediction value greater than 0.5 indicates that the patient shows pathological complete remission after neoadjuvant therapy of rectal cancer, and vice versa.

[0043] In a second aspect, the present application provides a rectal cancer neoadjuvant therapy efficacy prediction system based on multi-modal knowledge decomposition, comprising:

[0044] The acquisition module is configured to acquire multi-modal medical data of a plurality of rectal cancer patients, wherein the multi-modal medical data of each patient comprises structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures, medical text data, and whether the patient shows pathological complete remission after neoadjuvant therapy of rectal cancer;

[0045] The first feature extraction module is configured to extract features from the structured clinical features before neoadjuvant therapy, the colonoscopy pictures, the CT pictures, and the medical text data, respectively, to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and calculate cross-entropy loss of the clinical representation, the colonoscopy representation, the CT representation, and the text representation;

[0046] The second feature extraction module is configured to extract shared representation of the clinical representation, the colonoscopy representation, the CT representation, and the text representation, and calculate shared loss and shared representation average value according to the shared representation;

[0047] The third feature extraction module is configured to extract modality-specific representation of the clinical representation, the colonoscopy representation, the CT representation, and the text representation, respectively, to obtain clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation, and text modality-specific representation;

[0048] The calculation module is configured to calculate orthogonal loss of the clinical modality-specific representation, the colonoscopy modality-specific representation, the CT modality-specific representation, the text modality-specific representation, and the shared representation average value;

[0049] The fourth feature extraction module is configured to extract cross-modal representation of the clinical representation, the colonoscopy representation, the CT representation, and the text representation;

[0050] The fusion module is configured to vector splice the shared representation average value, the clinical modality-specific representation, the CT modality-specific representation, the colonoscopy modality-specific representation, the text modality-specific representation, and the cross-modal representation to obtain multi-modal fusion representation;

[0051] The prediction module is configured to predict whether the patient shows pathological complete remission after neoadjuvant therapy of rectal cancer according to the multi-modal fusion representation, and update the neural network according to the cross-entropy loss, the shared loss, and the orthogonal loss.

[0052] Compared with the prior art, the present application has the following beneficial effects:

[0053] The application provides a multi-modal data fusion algorithm based on multi-modal knowledge decomposition, which decomposes multi-modal data into multiple independent modal knowledge, reduces redundant information, and does not miss important inter-modal complementary information.

[0054] The application simultaneously uses unsupervised clustering and supervised CNN algorithm, compared with the traditional unsupervised random selection or information entropy-based image selection method, the application can reduce the interference of noise data on picture screening, so that representative colonoscopy pictures and CT pictures can be selected. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flowchart of a rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition provided by an embodiment of the application is shown.

[0056] Figure 2 A multi-modal data fusion method framework provided by an embodiment of the application is shown.

[0057] Figure 3 A structure diagram of a rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition provided by an embodiment of the application is shown.

[0058] Figure 4 A schematic block diagram of an example electronic device of an embodiment of the application is shown. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0060] The first embodiment of the application provides a rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition, Figure 1 A flowchart of a rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition provided by an embodiment of the application is shown.

[0061] As shown in Figure 1 The rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition specifically includes the following steps:

[0062] S101, obtain multi-modal medical data of a plurality of rectal cancer patients, the multi-modal medical data of each patient including structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures, medical text data, and whether the rectal cancer after neoadjuvant therapy shows pathological complete remission.

[0063] The patients can be all chronic disease patients in a certain region, a certain city, a certain province, a certain country, etc., and the larger the number of patients, the more accurate the final prediction result. The multi-modal medical data of the patients can be obtained from an electronic health record. The multi-modal medical data of each patient is represented by the following parts: structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures, medical text data, and whether the rectal cancer after neoadjuvant therapy shows pathological complete remission. Understandably, the structured clinical features before neoadjuvant therapy, the colonoscopy pictures, the CT pictures, and the medical text data are used as model inputs, and whether the rectal cancer after neoadjuvant therapy shows pathological complete remission is used as a label.

[0064] Specifically, x c The structured clinical features of the patient before neoadjuvant therapy generally include demographic information, preoperative laboratory test indicators, and other information of the patient. r The colonoscopy pictures of the patient, in an actual medical scenario, a patient i will have multiple colonoscopy pictures, denoted as x CT The CT image features of the patient, in an actual medical scenario, a patient i will have multiple CT images, denoted as x text The medical text features of the patient, such as medical orders, colonoscopy findings, etc. y is whether the rectal cancer patient after neoadjuvant therapy shows pathological complete remission, also known as a label; if y = 1, it means that the patient after neoadjuvant radiotherapy and chemotherapy shows pathological complete remission; if y = 0, it means that the patient after neoadjuvant radiotherapy and chemotherapy does not show pathological complete remission. The symbol N represents the number of collected rectal cancer patients, and the entire data set D is represented as:

[0065] S102, feature extraction is performed on the structured clinical features before neoadjuvant therapy, the colonoscopy pictures, the CT pictures, and the medical text data, respectively, to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and the cross-entropy loss of the clinical representation, the colonoscopy representation, the CT representation, and the text representation is calculated.

[0066] This step is to perform single-modal feature extraction and cross-entropy loss calculation. The method for single-modal feature extraction is a commonly used extraction method in the art, and those skilled in the art can select it according to the actual scene. The cross-entropy loss is the key of this step.

[0067] For single-modal feature extraction, for example, a fully connected network can be used to extract structured clinical features before neoadjuvant therapy, to obtain clinical representation, denoted as e c CT image features can be extracted using version transformer, and aggregated using GRU, to obtain CT representation, denoted as e CT Endoscopy picture features can be extracted using version transformer, and aggregated using GRU, to obtain endoscopy representation, denoted as e m Medical text features can be extracted using transformer, and aggregated using GRU, to obtain text representation, denoted as e text .

[0068] Before performing multi-modal feature fusion, it is necessary to remove redundant features and noise in each single-modal in advance to obtain better fusion representation, so the cross-entropy loss of the single-modal is calculated.

[0069] After obtaining the clinical representation, endoscopy representation, CT representation and text representation, the cross-entropy loss is calculated, and the specific method is as follows:

[0070] The clinical representation e c , the endoscopy representation e m , the CT representation e CT and the text representation e text are input into the fully connected network FC1, and the cross-entropy loss function L1 is calculated:

[0071] L1=

[0072] crossEntopy(FC1(e c ),y)+crossEntopy(FC1(e CT ),y)+

[0073] crossEntopy(FC1(e m ),y)+crossEntopy(FC1(e text ),y);

[0074] Wherein, FC1 is a fully connected network, crossEntropy represents a binary cross-entropy loss function, and its expression is Wherein, represents the predicted value, and a represents the true label.

[0075] L1 has four parts of cross-entropy loss function, which are used for removing redundant features and noise in clinical data, endoscopy data, CT data and text data, respectively.

[0076] In some preferred embodiments, in order to ensure that the CT images and colonoscopy images used in the single-modal feature extraction are representative images, representative images need to be selected. The method for selecting representative images is the same for CT images and colonoscopy images, and is referred to as the method for selecting colonoscopy images or the method for selecting CT images below, and specifically includes the following steps:

[0077] S102-1, for each patient, clustering all colonoscopy images or CT images to obtain K categories, K being a hyperparameter.

[0078] Common clustering methods can be used for clustering, such as spectral clustering, k-means clustering, k-means++ clustering algorithm, etc. K is set by the user, and it is foreseeable that the larger the value of K, the more images are selected, and the performance may be better, but the computational load will also be larger. The clustering result is denoted as Clu i , the length of which is , i.e., finally K categories are obtained.

[0079] S102-2, randomly selecting one colonoscopy image or CT image from each category to set a label, obtaining K labeled colonoscopy images or CT images for each patient, wherein the label is whether the rectal cancer after neoadjuvant therapy shows pathologic complete remission.

[0080] Understandably, the label of each patient is actually the same.

[0081] S102-3, training a CNN neural network using all labeled colonoscopy images or CT images of the patients.

[0082] This step is to train a CNN neural network based on the K colonoscopy images or CT images selected from all patients and their labels. The loss function in this training process is: where y m is the true label of the colonoscopy image or CT image, is the predicted value of the CNN neural network, which is a decimal number between 0 and 1.

[0083] S102-4, inputting each patient's colonoscopy image or CT image into the trained CNN neural network to obtain a prediction result.

[0084] It should be noted that here, all original colonoscopy images or CT images of each patient are input into the trained CNN neural network for prediction.

[0085] S102-5, sorting according to the size of the prediction result, and dividing into K groups according to the sorting.

[0086] Specifically, the prediction results are arranged in descending or ascending order according to their sizes, and then the pictures are grouped into K groups according to the order, and finally K groups are obtained.

[0087] S102-6, randomly select one picture from each of the K groups as a representative picture.

[0088] S103, extract shared representations of the clinical representation, the colonoscopy representation, the CT representation and the text representation, and calculate a shared loss and a shared representation average value according to the shared representations.

[0089] The extraction method of the shared representation in this step is a commonly used extraction method in the art, and a person skilled in the art can select it according to the actual scene. The shared loss is the key of this step.

[0090] For the extraction of shared features, the same shared encoder is used to extract shared representations before the multi-modal medical data. The shared encoder can be implemented using a fully connected network or other neural networks. The shared representation is denoted by symbol e share In the embodiments of the present application, the multi-modal medical data has four modalities, i.e., structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures and medical text data, and each modality representation is input into the shared encoder. Therefore, it is easy to know that e share is composed of four parts.

[0091] According to e share , a shared loss function L3 is calculated, which is used to ensure that the shared representations extracted by the shared encoder are close to each other. The formula of the loss function L2 is:

[0092] L2=(e share [0]-e share [1]) 2 +(e share [1]-e share [2]) 2 +(e share [2]-

[0093] e share [3]) 2 ;

[0094] Wherein, e share [0] is the shared representation corresponding to the clinical representation, i.e., the shared representation obtained by inputting the clinical representation into the shared encoder, e share [1] is the shared representation corresponding to the CT representation, i.e., the shared representation obtained by inputting the CT representation into the shared encoder, e share [2] is the shared representation corresponding to the colonoscopy representation, i.e., the shared representation obtained by inputting the colonoscopy representation into the shared encoder, and e​share [3] is the shared representation corresponding to the text representation, i.e., the shared representation obtained by inputting the text representation into the shared encoder.

[0095] The shared representation average value e ave The calculation formula is as follows:

[0096]

[0097] S104, modal-specific representations of the clinical representation, the colonoscopy representation, the CT representation, and the text representation are extracted respectively to obtain the clinical modal-specific representation, the colonoscopy modal-specific representation, the CT modal-specific representation, and the text modal-specific representation.

[0098] This step is to obtain the specific representation of each modal representation, and the extraction method is a commonly used extraction method in the art, which can be selected by those skilled in the art according to the actual scene. For example, a fully connected network or other neural network can be used.

[0099] S105, the orthogonal loss of the clinical modal-specific representation, the colonoscopy modal-specific representation, the CT modal-specific representation, the text modal-specific representation, and the shared representation average value is calculated.

[0100] Specifically, the orthogonal loss function L3 is as follows:

[0101] L3 = e cp ⊙e ave +e CTp ⊙e ave +e mp ⊙e ave +e textp ⊙e ave , where ⊙ represents

[0102] vector dot product.

[0103] The smaller the vector dot product is, the more orthogonal the two vectors are, i.e., the less correlation between the two vectors. The loss function L4 makes the clinical modal-specific representation, the colonoscopy modal-specific representation, the CT modal-specific representation, and the text modal-specific representation orthogonal to the shared representation average value, ensuring that the four specific representations are not related to the modal shared representation, thereby ensuring that the specific representation is extracted without causing loss of shared information.

[0104] S106, the cross-modal representation of the clinical representation, the colonoscopy representation, the CT representation, and the text representation is extracted.

[0105] The embodiment of the present application first proposes cross-modal representation, which is obtained by inputting clinical representation, colonoscopy representation, CT representation and text representation into a cross-modal encoder. Unlike traditional one-time fusion, the embodiment of the present application sequentially fuses two-by-two between multi-modal representations, thereby reducing information loss in the fusion process as much as possible. The fusion order of the modes is determined by calculating the information entropy, and the last fusion of the representation with the largest information entropy will help to retain more data information.

[0106] The calculation method of the cross-modal encoder is as follows:

[0107] S106-1, calculate the information entropy of the clinical representation, colonoscopy representation, CT representation and text representation, and the information entropy H is calculated according to the following formula:

[0108] Wherein, m is the length of the vector, e i is the value of the vector e at position i, and the vector e is the clinical representation, colonoscopy representation, CT representation or text representation. It can be seen that four information entropies can be obtained in this step.

[0109] S106-2, sort the four input representation vectors from small to large according to the information entropy. According to the sorting, the intermediate representation after fusing the latter representation with the former representation or the former several representations is fused.

[0110] Specifically, the vector with small information entropy is fused first, and then the fused vector is continuously fused with the next vector. For example, if the sorting result is H(e c )>H(e CT )>H(e m ) and H(e text ), then e c and e CT are spliced into vectors, and then input into a fully connected network to obtain e tmp1 . Then e tmp1 and e m are fused in the same way to obtain e tmp2 . Finally, e tmp2 and e text are fused in the same way to obtain the final cross-modal representation.

[0111] S107, vector splicing is performed on the shared representation average value, clinical modality specific representation, CT modality specific representation, colonoscopy modality specific representation, text modality specific representation and cross-modal representation to obtain a multi-modal fusion representation.

[0112] Figure 2 The multi-modal data fusion method framework provided by the embodiment of the present application is shown in the figure, which can be combined with Figure 2Further understanding the method of multi-modal fusion representation. As can be seen from the above, the multi-modal data fusion method based on multi-modal knowledge decomposition used by the present application is actually applicable to multi-modal fusion in any scene, decomposes multi-modal data into multiple independent modal knowledge for fusion, and in the fusion process, calculates a multi-modal representation loss function, a shared loss function of shared representation, and an orthogonal loss function of multi-modal representation, and updates the neural network with these loss functions, ensuring that the fusion process reduces redundant information while not missing important complementary information between modal features.

[0113] S108, whether the patient shows pathological complete remission after neoadjuvant therapy of rectal cancer is predicted according to the multi-modal fusion representation, and the neural network is updated according to the cross-entropy loss, the shared loss and the orthogonal loss.

[0114] This step is to use the multi-modal fusion feature as input, whether the patient shows pathological complete remission after neoadjuvant therapy as label y, and use the neural network updated by the above cross-entropy loss, shared loss and orthogonal loss to make prediction. Specifically, the multi-modal fusion representation is input into the updated neural network to obtain a prediction value, and the prediction value greater than 0.5 indicates that the patient shows pathological complete remission after neoadjuvant therapy of rectal cancer, otherwise the patient does not show pathological complete remission.

[0115] In addition, in order to evaluate the difference between the probability distribution of the predicted neoadjuvant therapy effect of the model and the probability distribution of the actual label, the cross-entropy loss function L5 is also used to update the neural network together with the above losses.

[0116] In some embodiments, the total loss function is calculated as follows:

[0117] The multi-modal fusion representation is input into the fully connected network to obtain a neoadjuvant therapy effect prediction value

[0118] The cross-entropy loss function L5 is calculated, and its mathematical expression is:

[0119]

[0120] The neural network is updated based on the total loss function L total , and its expression is:

[0121] L total = L1+L2+L3+L5.

[0122] The second embodiment of the present application provides a rectal cancer neoadjuvant therapy effect prediction system based on multi-modal knowledge decomposition, Figure 3 The structure of the rectal cancer neoadjuvant therapy effect prediction system 300 based on multi-modal knowledge decomposition provided by the embodiment of the present application is shown in the structure diagram.

[0123] As shown in Figure 3 The rectal cancer neoadjuvant therapy efficacy prediction system 300 based on multi-modal knowledge decomposition includes an acquisition module 301, a first feature extraction module 302, a second feature extraction module 303, a third feature extraction module 304, a calculation module 305, a fourth feature extraction module 306, a fusion module 307, and a prediction module 308. The detailed functions of each module are as follows.

[0124] The acquisition module 301 is configured to acquire multi-modal medical data of a plurality of rectal cancer patients, wherein the multi-modal medical data of each patient includes structured clinical features before neoadjuvant therapy, colonoscopy pictures, CT pictures, medical text data, and whether the rectal cancer patient shows pathological complete remission after neoadjuvant therapy.

[0125] The first feature extraction module 302 is configured to extract features from the structured clinical features, colonoscopy pictures, CT pictures, and medical text data before neoadjuvant therapy to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and calculate cross-entropy loss of the clinical representation, colonoscopy representation, CT representation, and text representation.

[0126] The second feature extraction module 303 is configured to extract shared representation of the clinical representation, colonoscopy representation, CT representation, and text representation, and calculate shared loss and shared representation average value according to the shared representation.

[0127] The third feature extraction module 304 is configured to extract modality-specific representation of the clinical representation, colonoscopy representation, CT representation, and text representation to obtain clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation, and text modality-specific representation.

[0128] The calculation module 305 is configured to calculate orthogonal loss of the clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation, text modality-specific representation, and shared representation average value.

[0129] The fourth feature extraction module 306 is configured to extract cross-modal representation of the clinical representation, colonoscopy representation, CT representation, and text representation.

[0130] The fusion module 307 is configured to vector splice the shared representation average value, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modal representation to obtain multi-modal fusion representation.

[0131] The prediction module 308 is configured to predict whether the patient shows pathological complete remission after rectal cancer neoadjuvant therapy according to the multi-modal fusion representation, and update the neural network according to the cross-entropy loss, shared loss, and orthogonal loss.

[0132] Based on the above-mentioned embodiments, the present application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the steps in the rectal cancer neoadjuvant therapy efficacy prediction method based on multi-modal knowledge decomposition in the first embodiment when executing the computer program.

[0133] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0134] As shown in Figure 4 The electronic device 400 can include a computing unit 401 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0135] Various components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; the storage unit 408, such as a magnetic disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0136] The computing unit 401 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs various methods and processes described above, such as the omics data processing method or the model training method for omics data processing. For example, in some embodiments, the omics data processing method or the model training method for omics data processing can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded onto the RAM 403 and executed by the computing unit 401, one or more steps of the omics data processing method or the model training method for omics data processing described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the omics data processing method or the model training method for omics data processing by any other appropriate means, such as by means of firmware.

[0137] Based on the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method for predicting neoadjuvant therapy efficacy of rectal cancer based on multi-modal knowledge decomposition disclosed in the embodiments of the present application.

[0138] Based on the above embodiments, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the method for predicting neoadjuvant therapy efficacy of rectal cancer based on multi-modal knowledge decomposition disclosed in the embodiments of the present application.

[0139] The computer program includes computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB (Universal Serial Bus) flash disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a ROM (Read-Only Memory), a RAM (Random Access Memory), an electrical carrier signal, a telecommunications signal, and a software distribution medium, and the like. It should be noted that the computer-readable medium contains contents that can be appropriately added, deleted, or modified according to requirements of legislation and patent practices in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practices, the computer-readable medium does not include electrical carrier signals and telecommunications signals.

[0140] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0141] In several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices, or units, and can be electrical, mechanical, or in other forms.

[0142] The units described as separate components may or can not be physically separate, and the components shown as units may or can not be physical units, i.e., they can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0143] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0144] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application.

[0145] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, characterized in that, Include: Acquire multimodal medical data from multiple rectal cancer patients. The multimodal medical data for each patient includes structured clinical features before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer. Feature extraction was performed on the structured clinical features, colonoscopy images, CT images and medical text data before neoadjuvant therapy to obtain clinical representation, colonoscopy representation, CT representation and text representation, and the cross-entropy loss of clinical representation, colonoscopy representation, CT representation and text representation was calculated; Shared representations are extracted from clinical, colonoscopy, CT, and text representations. The shared loss and average shared representation are calculated based on these shared representations. The shared loss function L2 is: ; Shared representations corresponding to clinical representations Shared representations corresponding to CT representations This is a shared characteristic corresponding to colonoscopy characteristics. This refers to the shared representation corresponding to the text representation; Modality-specific representations of clinical representations, colonoscopy representations, CT representations, and text representations were extracted respectively to obtain clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, and text modality-specific representations. The orthogonal loss function L3 is used to calculate the average values ​​of clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, text modality-specific representations, and shared representations. ;e cp For clinical modality-specific characterization, e CTp For CT modality-specific characterization, e mp For colonoscopy modality-specific characterization, e textp For text modality-specific characterization, e ave To share the average value of the representation, It is the dot product of vectors; Cross-modal representations of clinical representations, colonoscopy representations, CT representations, and text representations are extracted. The extraction method includes: calculating the information entropy of clinical representations, colonoscopy representations, CT representations, and text representations respectively; sorting the clinical representations, colonoscopy representations, CT representations, and text representations from smallest to largest according to the information entropy; and fusing the next representation with the previous representation or the intermediate representation after fusing the first few representations according to the order. The shared representation average, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modality representation are vector-concatenated to obtain a multimodal fusion representation; The multimodal fusion representation is used to predict whether patients will achieve pathological complete remission after neoadjuvant therapy for rectal cancer, and the neural network is updated based on cross-entropy loss, sharing loss, and orthogonal loss.

2. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition as described in claim 1, characterized in that, The cross-entropy loss function L1 is: ； Among them, e c As a clinical manifestation, e CT For CT characterization, e m For colonoscopy characteristics, e text For text representation, FC1 is a fully connected network, and crossEntropy represents the binary classification cross-entropy loss function, whose expression is: ,in, Indicates the predicted value. This indicates the actual label.

3. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition as described in claim 1, characterized in that, The formula for calculating information entropy H is: Where m is the length of vector e, Let be the value of vector e at position i.

4. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition as described in claim 1, characterized in that, Representative images from colonoscopy and CT scans were selected for feature extraction. The selection method was as follows: For each patient, all colonoscopy or CT images are clustered to obtain K categories, where K is a hyperparameter. One colonoscopy or CT image is randomly selected from each category and labeled. This yields K labeled colonoscopy or CT images for each patient, where the label indicates whether the patient achieved complete pathological remission after neoadjuvant therapy for rectal cancer. Train a CNN neural network using labeled colonoscopy or CT images of all patients; Each patient's colonoscopy or CT image is input into a trained CNN neural network to obtain prediction results; The prediction results are sorted by size and then divided into K groups based on the sorting. One colonoscopy or CT image is randomly selected from each of the K groups as a representative image.

5. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition as described in claim 4, characterized in that, The loss function L4 for training a CNN neural network is: ,in, These are the actual labels for colonoscopy or CT images. It is the predicted value of the CNN neural network, a decimal between 0 and 1.

6. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition as described in claim 1, characterized in that, The method for predicting whether a patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer based on the aforementioned multimodal fusion characterization is as follows: The multimodal fusion representation is input into the updated neural network to obtain the predicted value. A predicted value greater than 0.5 indicates that the patient has achieved complete pathological remission after neoadjuvant therapy for rectal cancer, while a predicted value less than 0.5 indicates that the patient has not achieved complete pathological remission.

7. A system for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, characterized in that, Include: The acquisition module is used to acquire multimodal medical data from multiple rectal cancer patients. The multimodal medical data of each patient includes structured clinical features before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer. The first feature extraction module is used to extract features from the structured clinical features, colonoscopy images, CT images and medical text data before neoadjuvant therapy, respectively, to obtain clinical representation, colonoscopy representation, CT representation and text representation, and to calculate the cross-entropy loss of clinical representation, colonoscopy representation, CT representation and text representation; The second feature extraction module is used to extract shared features from clinical features, colonoscopy features, CT features, and text features, and to calculate the shared loss and the average value of the shared features based on the shared features. The shared loss function L2 is: ; Shared representations corresponding to clinical representations Shared representations corresponding to CT representations This is a shared characteristic corresponding to colonoscopy characteristics. This refers to the shared representation corresponding to the text representation; The third feature extraction module is used to extract modality-specific features of clinical features, colonoscopy features, CT features and text features respectively, to obtain clinical modality-specific features, colonoscopy modality-specific features, CT modality-specific features and text modality-specific features; The calculation module is used to calculate the orthogonal loss of the average values ​​of clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, text modality-specific representations, and shared representations. The orthogonal loss function L3 is: ;e cp For clinical modality-specific characterization, e CTp For CT modality-specific characterization, e mp For colonoscopy modality-specific characterization, e textp For text modality-specific characterization, e ave To share the average value of the representation, It is the dot product of vectors; The fourth feature extraction module is used to extract cross-modal representations of clinical representations, colonoscopy representations, CT representations and text representations. The extraction method includes: calculating the information entropy of clinical representations, colonoscopy representations, CT representations and text representations respectively; sorting the clinical representations, colonoscopy representations, CT representations and text representations from smallest to largest according to the size of the information entropy; and fusing the next representation with the previous representation or the intermediate representation after fusing the first few representations according to the order. The fusion module is used to concatenate shared representation averages, clinical modality-specific representations, CT modality-specific representations, colonoscopy modality-specific representations, text modality-specific representations, and cross-modal representations into vectors to obtain a multimodal fusion representation. The prediction module is used to predict whether a patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer based on the multimodal fusion representation, and to update the neural network based on cross-entropy loss, sharing loss and orthogonal loss.

Images