Multi-modal knowledge decomposition-based rectal cancer neoadjuvant therapy curative effect prediction method and system

Abstract

The invention belongs to the technical field of medical data processing, and discloses a rectal cancer neoadjuvant therapy curative effect prediction method and system based on multi-modal knowledge decomposition. The invention provides a multi-modal data fusion algorithm based on multi-modal knowledge decomposition, which decomposes multi-modal data into a plurality of independent modal knowledge, and does not lose important inter-modal complementary information while reducing redundant information. According to the method, the unsupervised clustering algorithm and the supervised CNN algorithm are used at the same time, and compared with a traditional image selection method only using unsupervised random selection or based on information entropy, the interference of noise data on image screening can be reduced, so that representative enteroscope images and CT images can be selected.

Description

Technical Field

[0001] This invention relates to the field of medical data processing technology, and in particular to a method and system for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition. Background Technology

[0002] Rectal cancer is a common malignant tumor of the digestive tract, and its incidence is rising globally, especially in my country, where the incidence and mortality rates remain high. Treatment strategies for rectal cancer typically involve a combination of surgery, radiotherapy, and chemotherapy. For locally advanced rectal cancer (such as stage II or III), the standard treatment is neoadjuvant chemoradiotherapy. This approach aims to shrink the tumor and improve local control rates by administering chemoradiotherapy before surgery, thereby increasing patient survival. However, the efficacy of neoadjuvant chemoradiotherapy varies significantly among patients. Some patients achieve complete pathological remission, meaning the tumor disappears completely after chemoradiotherapy, while others experience poor or no response. Therefore, effectively predicting the efficacy of neoadjuvant therapy has become an important topic in clinical research. Predicting a patient's response to neoadjuvant chemoradiotherapy before surgery can not only optimize treatment plans and reduce the financial burden on patients but also avoid unnecessary treatment side effects.

[0003] Imaging plays a crucial role in predicting the efficacy of neoadjuvant therapy for rectal cancer. MRI (Magnetic Resonance Imaging) and CT (Computed Tomography) are widely used to assess tumor size, invasiveness, and lymph node changes. MRI, in particular, provides more accurate imaging features of tumor staging and regression. High-resolution MRI and colonoscopy have become standard assessment methods. In recent years, radiomics has gradually become a hot research area. By extracting high-dimensional features from imaging data and combining them with patients' clinical data, it constructs efficacy prediction models and has shown promising application prospects in several international studies.

[0004] However, current methods for predicting the efficacy of neoadjuvant therapy for rectal cancer have the following shortcomings. First, current multimodal methods often employ simple fusion strategies, such as splicing and vector inner products, without fully considering the heterogeneity between modalities and making it difficult to extract complementary information between them. Second, rectal cancer patients are often recorded with multiple CT and colonoscopy images; however, to reduce computational burden, most current methods often use unsupervised sampling methods such as random sampling, making it difficult to guarantee that the sampled images are representative. Summary of the Invention

[0005] The purpose of this invention is to solve at least one of the above-mentioned technical problems, and to provide a method and system for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition.

[0006] To achieve the above objectives, the first aspect of this invention provides a method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, comprising:

[0007] Acquire multimodal medical data from multiple rectal cancer patients. The multimodal medical data for each patient includes structured clinical features before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer.

[0008] Feature extraction was performed on the structured clinical features, colonoscopy images, CT images, and medical text data before neoadjuvant therapy to obtain clinical representation, colonoscopy representation, CT representation, and text representation. The cross-entropy loss of the clinical representation, colonoscopy representation, CT representation, and text representation was calculated.

[0009] Extract shared representations from clinical representations, colonoscopy representations, CT representations, and text representations, and calculate the shared loss and the average value of the shared representations based on the shared representations;

[0010] Modality-specific representations of clinical representations, colonoscopy representations, CT representations, and text representations were extracted respectively to obtain clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, and text modality-specific representations.

[0011] Calculate the orthogonal loss of the average values ​​of clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, text modality-specific representations, and shared representations;

[0012] Extract cross-modal representations from clinical, colonoscopy, CT, and textual representations;

[0013] The shared representation average, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modality representation are vector-concatenated to obtain a multimodal fusion representation;

[0014] The multimodal fusion representation is used to predict whether a patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer, and the neural network is updated based on cross-entropy loss, sharing loss, and orthogonal loss.

[0015] Preferably, the cross-entropy loss function L1 is:

[0016] L1 =

[0017] crossEntopy(FC1(e c ),y)+crossEntopy(FC1(e CT ),y)+

[0018] crossEntopy(FC1(e m),y)+crossEntopy(FC1(e text ),y);

[0019] Among them, e c As a clinical manifestation, e CT For CT characterization, e m For colonoscopy characteristics, e text For text representation, FC1 is a fully connected network, and crossEntropy represents the binary classification cross-entropy loss function, whose expression is: in, 'a' represents the predicted value, and 'a' represents the true label.

[0020] Preferably, the shared loss function L2 is:

[0021] L2=(e share [0]-e share [1]) 2 +(e share [1]-e share [2]) 2 +(e share [2]-

[0022] e share [3]) 2 ;

[0023] Among them, e share [0] represents the shared representation corresponding to the clinical representation, e share [1] is the shared representation corresponding to the CT representation, e share [2] is the shared representation corresponding to the colonoscopy representation, e share [3] is the shared representation corresponding to the text representation.

[0024] Preferably, the orthogonal loss function L3 is:

[0025] L3 = e cp ⊙e ave +e CTp ⊙e ave +e mp ⊙e ave +e textp ⊙e ave ;

[0026] Among them, e cp For clinical modality-specific characterization, e CTp For CT modality-specific characterization, e mp For colonoscopy modality-specific characterization, e textp For text modality-specific characterization, e ave To represent the average value, ⊙ represents the vector dot product.

[0027] The preferred method for extracting cross-modal representations of clinical, colonoscopy, CT, and textual representations is as follows:

[0028] Calculate the information entropy of clinical representation, colonoscopy representation, CT representation, and text representation respectively;

[0029] The clinical representation, colonoscopy representation, CT representation, and text representation are sorted from smallest to largest according to the size of information entropy. Then, the next representation is fused with the previous representation or the intermediate representation formed by fusing the first few representations in turn according to the order.

[0030] Preferably, the formula for calculating information entropy H is:

[0031] Where m is the length of vector e, e i Let be the value of vector e at position i.

[0032] Preferably, representative images from colonoscopy and CT scans are selected for feature extraction. The selection method is as follows:

[0033] For each patient, all colonoscopy or CT images are clustered to obtain K categories, where K is a hyperparameter.

[0034] One colonoscopy or CT image is randomly selected from each category and labeled. K labeled colonoscopy or CT images are obtained for each patient, where the label indicates whether the patient achieved complete pathological remission after neoadjuvant therapy for rectal cancer.

[0035] Train a CNN neural network using labeled colonoscopy or CT images of all patients;

[0036] Each patient's colonoscopy or CT image is input into the trained CNN neural network to obtain the prediction result;

[0037] The prediction results are sorted by size and then divided into K groups based on the sorting.

[0038] One colonoscopy or CT image is randomly selected from each of the K groups as a representative image.

[0039] Preferably, the loss function L4 for training a CNN neural network is:

[0040] Among them, y m These are the actual labels for colonoscopy or CT images. It is the predicted value of the CNN neural network, a decimal between 0 and 1.

[0041] Preferably, the method for predicting whether a patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer based on the multimodal fusion characterization is as follows:

[0042] The multimodal fusion representation is input into the updated neural network to obtain the predicted value. A predicted value greater than 0.5 indicates that the patient has achieved complete pathological remission after neoadjuvant therapy for rectal cancer, while a predicted value less than 0.5 indicates that the patient has not achieved complete pathological remission.

[0043] Secondly, this invention provides a system for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, comprising:

[0044] The acquisition module is used to acquire multimodal medical data from multiple rectal cancer patients. The multimodal medical data of each patient includes structured clinical features before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer.

[0045] The first feature extraction module is used to extract features from the structured clinical features, colonoscopy images, CT images, and medical text data before neoadjuvant therapy, respectively, to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and to calculate the cross-entropy loss of the clinical representation, colonoscopy representation, CT representation, and text representation.

[0046] The second feature extraction module is used to extract shared representations from clinical representations, colonoscopy representations, CT representations, and text representations, and to calculate the shared loss and the average value of the shared representations based on the shared representations.

[0047] The third feature extraction module is used to extract modality-specific features of clinical features, colonoscopy features, CT features and text features respectively, to obtain clinical modality-specific features, colonoscopy modality-specific features, CT modality-specific features and text modality-specific features;

[0048] The calculation module is used to calculate the orthogonal loss of the average values ​​of clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, text modality-specific representations, and shared representations.

[0049] The fourth feature extraction module is used to extract cross-modal representations of clinical, colonoscopy, CT, and text representations.

[0050] The fusion module is used to concatenate shared representation averages, clinical modality-specific representations, CT modality-specific representations, colonoscopy modality-specific representations, text modality-specific representations, and cross-modal representations into vectors to obtain a multimodal fusion representation.

[0051] The prediction module is used to predict whether a patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer based on the multimodal fusion representation, and to update the neural network based on cross-entropy loss, sharing loss, and orthogonal loss.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] This invention provides a multimodal data fusion algorithm based on multimodal knowledge decomposition, which decomposes multimodal data into multiple independent modal knowledge, reducing redundant information without losing important intermodal complementary information.

[0054] This invention uses both unsupervised clustering and supervised CNN algorithms. Compared with traditional image selection methods that only use unsupervised random selection or information entropy-based methods, this invention can reduce the interference of noisy data on image selection, thereby enabling the selection of representative colonoscopy and CT images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart illustrating the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, provided in an embodiment of the present invention.

[0056] Figure 2 A framework diagram of a multimodal data fusion method provided in a specific embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the structure of the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic block diagram of an example electronic device according to an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The first embodiment of this invention provides a method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition. Figure 1 This is a flowchart illustrating the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, as provided in an embodiment of the present invention.

[0061] like Figure 1 As shown, the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition specifically includes the following steps:

[0062] S101, acquire multimodal medical data from multiple rectal cancer patients. The multimodal medical data for each patient includes structured clinical features before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer.

[0063] Patients can be any chronic disease patient from a specific region, city, province, or country. The larger the number of patients, the more accurate the final prediction. Patient multimodal medical data can be obtained from electronic health records. Each patient's multimodal medical data is represented by the following components: structured clinical characteristics before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer. Understandably, the structured clinical characteristics before neoadjuvant therapy, colonoscopy images, CT images, and medical text data serve as model inputs, while whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer serves as a label.

[0064] Specifically, x c This refers to the structured clinical characteristics of a patient prior to neoadjuvant therapy, which generally include the patient's demographics, preoperative laboratory test results, and other information. r These are colonoscopy images of a patient. In actual medical settings, a patient i may have multiple colonoscopy images, denoted as . x CT For the characteristics of a patient's CT images, in a real medical scenario, a patient i will have multiple CT images, denoted as x text The data represents the patient's medical text features, such as medical orders and colonoscopy findings. y represents whether the rectal cancer patient achieved pathological complete remission after neoadjuvant therapy, also known as the label; if y = 1, it indicates that the patient achieved pathological complete remission after neoadjuvant chemoradiotherapy; if y = 0, it indicates that the patient did not achieve pathological complete remission after neoadjuvant chemoradiotherapy. The symbol N represents the number of rectal cancer patients collected, and the entire dataset D is represented as:

[0065] S102 extracts features from the structured clinical features, colonoscopy images, CT images, and medical text data before neoadjuvant therapy to obtain clinical representations, colonoscopy representations, CT representations, and text representations, and calculates the cross-entropy loss of the clinical representations, colonoscopy representations, CT representations, and text representations.

[0066] This step involves extracting features from a single modality and calculating the cross-entropy loss. The methods used for single-modality feature extraction are commonly used in this field, and those skilled in the art can choose the appropriate method based on the specific scenario. The key aspect of this step is the formulation of the cross-entropy loss.

[0067] For unimodal feature extraction, for example, a fully connected network can be used to extract structured clinical features prior to neoadjuvant therapy, resulting in a clinical characterization, denoted as e. c CT image features can be extracted using a version transformer and then clustered using a GRU to obtain the CT characterization, denoted as e. CT The colonoscopy image features were extracted using a version transformer and then clustered using a GRU to obtain the colonoscopy characterization, denoted as e. m We use a transformer to extract features from medical text and then use GRU to perform clustering to obtain the text representation, denoted as e. text .

[0068] Before performing multimodal feature fusion, redundant features and noise in each single modality need to be removed in advance to obtain a better fusion representation. Therefore, the cross-entropy loss of single modality is proposed.

[0069] After obtaining the clinical, colonoscopy, CT, and text representations, their cross-entropy loss is calculated using the following method:

[0070] Clinical manifestations e c Colonoscopy characteristics e m CT characterization e CT and text representation e text The input is fed into the fully connected network FC1, and the cross-entropy loss function L1 is calculated:

[0071] L1 =

[0072] crossEntopy(FC1(e c ),y)+crossEntopy(FC1(e CT ),y)+

[0073] crossEntopy(FC1(e m ),y)+crossEntopy(FC1(e text ),y);

[0074] Where FC1 is a fully connected network, and crossEntropy represents the binary classification cross-entropy loss function, its expression is: in, 'a' represents the predicted value, and 'a' represents the true label.

[0075] L1 has a four-part cross-entropy loss function, which is used to remove redundant features and noise from clinical data, colonoscopy data, CT data, and text data, respectively.

[0076] In some preferred embodiments, to ensure that the CT and colonoscopy images used during single-modal feature extraction are representative, it is necessary to select representative images. The method for selecting representative images for CT and colonoscopy is the same; hereinafter, "colonoscopy image or CT image" refers to the method for selecting colonoscopy images or CT images, which specifically includes the following steps:

[0077] S102-1, for each patient, cluster all colonoscopy or CT images to obtain K categories, where K is a hyperparameter.

[0078] Common clustering methods can be used for clustering, such as spectral clustering, k-means clustering, and k-means++ clustering algorithms. K is set by the user; predictably, a larger value for K will result in more images being selected, potentially leading to better performance, but also increasing computational cost. The clustering result is denoted as Clu. i Its length is That is, we eventually get K categories.

[0079] S102-2, randomly select one colonoscopy or CT image from each category and set a label to obtain K labeled colonoscopy or CT images for each patient, where the label indicates whether the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer.

[0080] Understandably, the labels for each patient are actually the same.

[0081] S102-3, train a CNN neural network using labeled colonoscopy or CT images of all patients.

[0082] This step involves training a CNN neural network based on K colonoscopy or CT images selected from all patients and their labels. The loss function during training is: Among them, y m These are the actual labels for colonoscopy or CT images. It is the predicted value of the CNN neural network, a decimal between 0 and 1.

[0083] S102-4, Input each patient's colonoscopy or CT image into the trained CNN neural network to obtain the prediction result.

[0084] It should be noted that here, all the original, unfiltered colonoscopy or CT images of each patient are input into a trained CNN neural network for prediction.

[0085] S102-5, sort the prediction results by size, and divide them into K groups according to the sorting.

[0086] Specifically, first arrange the predicted results in descending or ascending order based on their magnitude, then... The data is divided into groups of 1 and 2, resulting in K groups.

[0087] S102-6, Randomly select one colonoscopy or CT image from each group in group K as a representative image.

[0088] S103 extracts shared representations from clinical representations, colonoscopy representations, CT representations, and text representations, and calculates the shared loss and the average value of the shared representations based on the shared representations.

[0089] The method used to extract the shared representation in this step is a commonly used method in the field, and those skilled in the art can choose according to the actual scenario. The formulation of the shared loss is the key to this step.

[0090] For the extraction of shared features, a single shared encoder is used to extract shared representations prior to the extraction of multimodal medical data. The shared encoder can be implemented using a fully connected network or other neural networks. The notation e is used. share This is used to represent the shared representation. In this embodiment, the multimodal medical data comprises four modalities: structured clinical features prior to neoadjuvant therapy, colonoscopy images, CT images, and medical text data. Each modal representation is input into the shared encoder. Therefore, it is easy to see that e share It consists of four parts.

[0091] According to e share Calculate the shared loss function L3, which ensures that the four shared representations extracted by the shared encoder are close to each other. The formula for the loss function L2 is:

[0092] L2=(e share [0]-e share [1]) 2 +(e share [1]-e share [2]) 2 +(e share [2]-

[0093] e share [3]) 2 ;

[0094] Among them, e share [0] represents the shared representation corresponding to the clinical representation, i.e., the shared representation obtained by inputting the clinical representation into the shared encoder, e share [1] is the shared representation corresponding to the CT representation, that is, the shared representation obtained by inputting the CT representation into the shared encoder, e share [2] is the shared representation corresponding to the colonoscopy representation, that is, the shared representation obtained by inputting the colonoscopy representation into the shared encoder, eshare [3] is the shared representation corresponding to the text representation, that is, the shared representation obtained by inputting the text representation into the shared encoder.

[0095] Shared representation average e ave The calculation formula is:

[0096]

[0097] S104 extracts modality-specific representations of clinical representation, colonoscopy representation, CT representation and text representation respectively, to obtain clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation and text modality-specific representation.

[0098] This step aims to obtain specific representations for each modality. The extraction method used is a commonly used method in this field, and those skilled in the art can choose the appropriate method based on the specific scenario. For example, a fully connected network or other neural networks can be used.

[0099] S105 calculates the orthogonal loss of the average values ​​of clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, text modality-specific representations, and shared representations.

[0100] Specifically, the orthogonal loss function L3 is:

[0101] L3 = e cp ⊙e ave +e CTp ⊙e ave +e mp ⊙e ave +e textp ⊙e ave , where ⊙ represents

[0102] Vector dot product.

[0103] The smaller the vector dot product, the more orthogonal the two vectors are, indicating that there is no correlation between them. The loss function L4 ensures that the average value of the four specific representations (clinical modality specific representation, colonoscopy modality specific representation, CT modality specific representation, and text modality specific representation) is orthogonal to the shared representation, guaranteeing that these four specific representations are uncorrelated with the modality shared representation. This ensures that specific representations are extracted without causing the loss of shared information.

[0104] S106 extracts cross-modal representations of clinical, colonoscopy, CT, and textual representations.

[0105] This invention proposes a cross-modal representation for the first time, obtained by inputting clinical, colonoscopy, CT, and text representations into a cross-modal encoder. Unlike traditional one-time fusion, this invention fuses multimodal representations pairwise sequentially, thereby minimizing information loss during the fusion process. The fusion order of modalities is determined by calculating information entropy; the representation with the highest information entropy is fused last, which helps retain more data information.

[0106] The calculation method for this cross-modal encoder is as follows:

[0107] S106-1, calculate the information entropy of clinical representation, colonoscopy representation, CT representation, and text representation. The formula for calculating information entropy H is as follows:

[0108] Where m is the length of the vector, e i Let be the value of vector e at position i, where vector e represents the clinical representation, colonoscopy representation, CT representation, or text representation. It can be seen that this step yields four information entropies.

[0109] S106-2: Sort the four input representation vectors in ascending order according to their information entropy. Then, according to the sorting, fuse each subsequent representation with the previous representation or an intermediate representation obtained by fusing the first few representations.

[0110] Specifically, the vectors with lower information entropy are first fused, and then the fused vector is fused with the next vector. For example, if the sorting result is H(e... c )>H(e CT )>H(e m ) and H(e text ), then first put e c and e CT After concatenating the vectors, the result is fed into a fully connected network to obtain e. tmp1 Then, follow the same method with e. tmp1 With e m Fusion, resulting in e tmp2 Finally, e tmp2 and e text By fusing them in the same manner, the final cross-modal representation is obtained.

[0111] S107 involves vector concatenating the shared representation average, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modality representation to obtain a multimodal fusion representation.

[0112] Figure 2 This is a framework diagram of a multimodal data fusion method provided in a specific embodiment of the present invention, which can be combined with... Figure 2This further clarifies the method for obtaining multimodal fusion representations. As can be seen from the above, the multimodal data fusion method based on multimodal knowledge decomposition used in this invention is applicable to multimodal fusion in any scenario. It decomposes multimodal data into multiple independent modal knowledge for fusion, and during the fusion process, it calculates the multimodal representation loss function, the shared loss function of shared representations, and the orthogonal loss function of multimodal representations, and uses these loss functions to jointly update the neural network. This ensures that the fusion process reduces redundant information while not losing important complementary information between modal features.

[0113] S108, predict whether the patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer based on the multimodal fusion representation, and update the neural network based on cross-entropy loss, sharing loss, and orthogonal loss.

[0114] This step involves using the multimodal fusion features as input and whether the patient achieved pathological complete remission after neoadjuvant therapy as the label y. A neural network updated with the aforementioned cross-entropy loss, sharing loss, and orthogonal loss is then used for prediction. Specifically, the multimodal fusion representation is input into the updated neural network to obtain a predicted value. A predicted value greater than 0.5 indicates that the patient achieved pathological complete remission after neoadjuvant therapy for rectal cancer; conversely, a value less than 0.5 indicates that the patient did not achieve pathological complete remission.

[0115] In addition, to evaluate the difference between the probability distribution of the neoadjuvant therapy efficacy predicted by the model and the probability distribution of the actual labels, the cross-entropy loss function L5 was used together with the above loss to update the neural network.

[0116] In some specific implementations, the total loss function is calculated as follows:

[0117] The multimodal fusion representation is input into a fully connected network to obtain the predicted value of neoadjuvant therapy efficacy.

[0118] The cross-entropy loss function L5 is calculated as follows:

[0119]

[0120] Based on the total loss function L total To update the neural network, the expression is:

[0121] L total =L1+L2+L3+L5.

[0122] The second embodiment of the present invention provides a system for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition. Figure 3 This is a schematic diagram of the structure of the neoadjuvant therapy efficacy prediction system 300 for rectal cancer based on multimodal knowledge decomposition provided in an embodiment of the present invention.

[0123] like Figure 3 As shown, the neoadjuvant therapy efficacy prediction system 300 for rectal cancer based on multimodal knowledge decomposition includes: an acquisition module 301, a first feature extraction module 302, a second feature extraction module 303, a third feature extraction module 304, a calculation module 305, a fourth feature extraction module 306, a fusion module 307, and a prediction module 308. The detailed functions of each module are as follows.

[0124] The acquisition module 301 is used to acquire multimodal medical data of multiple rectal cancer patients. The multimodal medical data of each patient includes structured clinical features before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the rectal cancer showed complete pathological remission after neoadjuvant therapy.

[0125] The first feature extraction module 302 is used to extract features from the structured clinical features, colonoscopy images, CT images, and medical text data before neoadjuvant therapy, respectively, to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and to calculate the cross-entropy loss of the clinical representation, colonoscopy representation, CT representation, and text representation.

[0126] The second feature extraction module 303 is used to extract shared representations from clinical representations, colonoscopy representations, CT representations and text representations, and to calculate the shared loss and the average value of the shared representations based on the shared representations.

[0127] The third feature extraction module 304 is used to extract modality-specific features of clinical features, colonoscopy features, CT features and text features respectively, to obtain clinical modality-specific features, colonoscopy modality-specific features, CT modality-specific features and text modality-specific features;

[0128] The calculation module 305 is used to calculate the orthogonal loss of the average values ​​of clinical modality-specific representations, colonoscopy modality-specific representations, CT modality-specific representations, text modality-specific representations, and shared representations.

[0129] The fourth feature extraction module 306 is used to extract cross-modal representations of clinical representations, colonoscopy representations, CT representations and text representations;

[0130] The fusion module 307 is used to perform vector concatenation of the shared representation average value, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation and cross-modality representation to obtain a multimodal fusion representation;

[0131] The prediction module 308 is used to predict whether a patient will achieve pathological complete remission after neoadjuvant therapy for rectal cancer based on the multimodal fusion representation, and to update the neural network based on cross-entropy loss, sharing loss, and orthogonal loss.

[0132] Based on the above embodiments, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition in the first embodiment.

[0133] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0134] like Figure 4 As shown, the electronic device 400 may include a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0135] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as omics data processing methods or model training methods for omics data processing. For example, in some embodiments, the omics data processing method or the model training method for omics data processing may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the omics data processing method or the model training method for omics data processing described above may be performed. Alternatively, in other embodiments, computing unit 401 may be configured by any other suitable means (e.g., by means of firmware) to perform omics data processing methods or model training methods for omics data processing.

[0137] Based on the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition disclosed in the embodiments of the present invention.

[0138] Based on the above embodiments, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition disclosed in the embodiments of the present invention.

[0139] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0145] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition, characterized in that: Include: Obtain multimodal medical data of multiple rectal cancer patients. The multimodal medical data of each patient includes structured clinical characteristics before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the rectal cancer has pathological complete remission after neoadjuvant therapy; Feature extraction was performed on structured clinical features, colonoscopy images, CT images, and medical text data before neoadjuvant therapy to obtain clinical representation, colonoscopy representation, CT representation, and text representation, and the cross entropy loss of clinical representation, colonoscopy representation, CT representation, and text representation was calculated; Extract shared representations of clinical representation, colonoscopy representation, CT representation, and text representation, and calculate shared loss and shared representation average based on the shared representation; The modality-specific representations of clinical representation, colonoscopy representation, CT representation and text representation are extracted respectively to obtain clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation and text modality-specific representation; Calculate the orthogonal loss of the average of clinical modality-specific representation, colonoscopy modality-specific representation, CT modality-specific representation, textual modality-specific representation, and shared representation; Extract cross-modal representations of clinical representations, colonoscopy representations, CT representations, and textual representations; The shared representation average, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modality representation are vector-concatenated to obtain a multimodal fusion representation. Whether the patient shows pathological complete remission after neoadjuvant treatment for rectal cancer is predicted based on the multimodal fusion representation, and the neural network is updated based on cross entropy loss, shared loss, and orthogonal loss.

2. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 1, characterized in that: The cross entropy loss function L1 is: L1＝ crossEntopy(FC1(e c ),y)+crossEntopy(FC1(e CT ),y)+ crossEntopy(FC1(e m ),y)+crossEntopy(FC1(e text ),y); Among them, e c For clinical manifestations, e CT For CT characterization, e m For colonoscopy, e text is the text representation, FC1 is the fully connected network, crossEntropy represents the binary cross entropy loss function, and its expression is in, represents the predicted value and a represents the true label.

3. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 1, characterized in that: The shared loss function L2 is: L2=(e share [0]-e share [1]) 2 +(e share [1]-e share [2]) 2 +(e share [2]-e share [3]) 2 Among them, e share [0] is the shared representation corresponding to the clinical representation, e share [1] is the shared representation corresponding to the CT representation, e share [2] is the shared representation corresponding to the colonoscopy representation, e share [3] is the shared representation corresponding to the text representation.

4. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 1, characterized in that: The orthogonal loss function L3 is: L3=e cp ☉and ave +and CTp ☉and ave +and mp ☉and ave +and text ☉and ave ; Among them, e cp For clinical modality-specific characterization, e CTp is the CT modality-specific characterization, e mp is the modality-specific characterization of colonoscopy, e textp is the text modality-specific representation, e ave is the shared representation average, and ⊙ is the vector dot product.

5. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 1, characterized in that: The method for extracting cross-modal representations of clinical representation, colonoscopy representation, CT representation and text representation is: The information entropy of clinical representation, colonoscopy representation, CT representation and text representation is calculated respectively; Clinical representation, colonoscopy representation, CT representation and text representation are sorted from small to large according to the size of information entropy, and the latter representation is fused with the former representation or the intermediate representation after the fusion of the first several representations in turn according to the sorting.

6. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 5, characterized in that: The calculation formula of information entropy H is: Where m is the length of vector e, e i is the value of vector e at position i.

7. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 5, characterized in that: Select representative images from colonoscopy and CT images for feature extraction. The selection method is as follows: For each patient, all colonoscopy images or CT images are clustered to obtain K categories, where K is a hyperparameter; Randomly select a colonoscopy or CT image from each category and set a label to obtain K labeled colonoscopy or CT images for each patient, where the label is whether the rectal cancer has shown pathological complete remission after neoadjuvant treatment; Train the CNN neural network using all the labeled colonoscopy or CT images of the patients; Input each patient's colonoscopy image or CT image into the above-mentioned trained CNN neural network to obtain the prediction result; Sort the prediction results by size and divide them into K groups according to the sorting; One colonoscopy image or CT image was randomly selected from each group K as a representative image.

8. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 7, characterized in that: The loss function L4 when training the CNN neural network is: Among them, y m is the real label of the colonoscopy or CT image. It is the predicted value of the CNN neural network, a decimal between 0 and 1.

9. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer based on multimodal knowledge decomposition according to claim 1, characterized in that: The method for predicting whether a patient exhibits pathological complete remission after neoadjuvant treatment for rectal cancer according to the multimodal fusion representation is: The multimodal fusion representation is input into the updated neural network to obtain a predicted value. A predicted value greater than 0.5 indicates that the patient showed pathological complete remission after neoadjuvant treatment for rectal cancer, otherwise the patient did not show pathological complete remission.

10. A rectal cancer neoadjuvant therapy efficacy prediction system based on multimodal knowledge decomposition, characterized in that: Include: The acquisition module is used to acquire multimodal medical data of multiple rectal cancer patients. The multimodal medical data of each patient includes structured clinical characteristics before neoadjuvant therapy, colonoscopy images, CT images, medical text data, and whether the rectal cancer has pathological complete remission after neoadjuvant therapy; The first feature extraction module is used to extract features from structured clinical features, colonoscopy images, CT images, and medical text data before neoadjuvant therapy, obtain clinical representations, colonoscopy representations, CT representations, and text representations, and calculate the cross entropy loss of the clinical representations, colonoscopy representations, CT representations, and text representations; The second feature extraction module is used to extract the shared representation of the clinical representation, the colonoscopy representation, the CT representation and the text representation, and calculate the shared loss and the shared representation average according to the shared representation; The third feature extraction module is used to extract the modality-specific representations of the clinical representation, the colonoscopy representation, the CT representation and the text representation, respectively, to obtain the clinical modality-specific representation, the colonoscopy modality-specific representation, the CT modality-specific representation and the text modality-specific representation; A calculation module, used to calculate the orthogonal loss of the clinical modality-specific representation, the colonoscopy modality-specific representation, the CT modality-specific representation, the text modality-specific representation, and the average value of the shared representation; The fourth feature extraction module is used to extract cross-modal representations of clinical representation, colonoscopy representation, CT representation and text representation; A fusion module is used to perform vector concatenation of the shared representation average, clinical modality-specific representation, CT modality-specific representation, colonoscopy modality-specific representation, text modality-specific representation, and cross-modality representation to obtain a multimodal fusion representation; A prediction module is used to predict whether the patient shows pathological complete remission after neoadjuvant therapy for rectal cancer based on the multimodal fusion representation, and to update the neural network based on cross entropy loss, shared loss, and orthogonal loss.

Images