A method and system for constructing a multi-modal interpretable post-colorectal cancer prediction model
Patent Information
- Application Number
- CN202510291461.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-12
- Publication Date
- 2026-09-15
Smart Images
Figure CN122762005A_ABST
Abstract
Description
Technical Field
[0001] This application proposes a method and system for constructing a multimodal interpretable colorectal cancer prediction model, which belongs to the field of colorectal cancer prediction technology. Background Technology
[0002] In biomedical research, gene expression data and pathological images are crucial for understanding disease mechanisms and for diagnosis and treatment. However, traditional methods typically analyze gene expression data and image data separately, failing to fully uncover the correlation between the two. While deep learning has achieved some success in processing such data, it faces challenges in interpretability, particularly in research on complex diseases such as colorectal cancer, where it is difficult to determine the basis and key factors for model decisions. XAI methods still have limitations in integrating multimodal data, failing to effectively utilize global and local knowledge to improve the accuracy and interpretability of the analysis.
[0003] Approximate Implementation Schemes: Some studies have attempted to perform deep learning analysis on gene expression data or pathological images independently, and to some extent, use simple visualization techniques or rule-based methods for interpretation. However, effective means are lacking in multimodal fusion and comprehensive interpretation. Some methods use traditional machine learning algorithms combined with gene expression data for classification in gene analysis, but do not consider the fusion with image information. In image analysis, although there are some deep learning-based image classification models, they have significant shortcomings in their association with gene expression data and model interpretability.
[0004] Current technologies lack effective multimodal fusion methods, failing to fully integrate information from gene expression data and digital pathological images, resulting in incomplete analysis results. Model interpretability is poor; neither gene analysis models nor image classification models clearly reveal the intrinsic link between the model's decision-making process and the prognosis and treatment response of colorectal cancer, hindering medical research and clinical applications. Classification accuracy is limited, failing to accurately identify genes closely related to colorectal cancer, impacting disease diagnosis and treatment efficacy assessment.
[0005] In colorectal cancer research, existing analytical methods are difficult to effectively integrate gene expression data and digital pathological image information, and lack interpretability of the model, resulting in insufficient accuracy in the analysis of colorectal cancer prognosis and treatment response, and difficulty in accurately identifying lesions and key genes. Summary of the Invention
[0006] This application provides a method and system for constructing a multimodal interpretable predictive model for colorectal cancer. By constructing a multimodal interpretable artificial intelligence (XAI) model, it integrates RNA-seq gene expression data related to tertiary lymphoid structures with digital pathological images to improve classification accuracy and identify genes that are significantly related to the prognosis and treatment response of colorectal cancer.
[0007] To solve the above-mentioned technical problems, the technical solution adopted in this application is as follows: In a first aspect, this application provides a method for constructing a multimodal interpretable predictive model for colorectal cancer, including: Acquire gene expression data and digital pathology images of colorectal cancer; The class association embedding method is used to process digital pathology images and extract the class association codes of the images; The class association coding of the image is fused with the coding of gene expression data to obtain the fused coding; The multimodal model is trained based on the fusion encoding, and classification prediction is performed after training. The classification prediction results are then interpreted using interpretability analysis methods to obtain the target multimodal interpretable colorectal cancer prediction model.
[0008] As a further improvement to this application, after obtaining colorectal cancer gene expression data and digital pathological images, the method further includes: Preprocessing of gene expression data includes: For RNA-seq gene expression data of colorectal cancer, gene expression data that meet the requirements are screened and retained, and TMM is used to standardize the gene expression data. After standardization, gene expression data are initially screened based on the correlation between gene expression data and colorectal cancer to obtain gene expression data with high disease correlation. Digital pathology images are preprocessed, including cropping and normalization.
[0009] As a further improvement of this application, the class association embedding method adopts a class association embedding framework in the form of a recurrent generative adversarial network, including an encoder, a decoder and a multi-class discriminator. The encoder decomposes the digital pathology image into class association code and individual code, wherein the class association code is used to encode image features related to the colorectal cancer classification task, and the individual code is used to encode image sample features that are not related to the class. By using the building block consistency feature extraction method, class association features in individual code subspaces are identified and weakened by exchanging and combining the codes of different class samples, and low-dimensional representations of class association features are learned and feature space is separated. A loss function is constructed that includes reconstruction loss, adversarial loss, and classification loss. The encoder and decoder are trained and updated using the loss function, enabling the class association embedding framework to extract the class association encoding of images.
[0010] As a further improvement to this application, the method of fusing the class association coding of the image with the coding of gene expression data to obtain the fused coding includes: A combination of gene embedding and expression embedding is used to generate gene codes based on the correlation between genes and their expression levels; The gene code is fused with the class association code extracted from digital pathology images, so that the fused code contains comprehensive information from both the gene and the image.
[0011] As a further improvement to this application, the step of training the multimodal model based on the fusion encoding, performing classification prediction after training, and interpreting the classification prediction results using interpretability analysis methods to obtain a target multimodal interpretable colorectal cancer prediction model includes: The fused encoded data is divided into training set, validation set, and test set; The multimodal model is trained using the fused encoded data, and the parameters of the multimodal model are adjusted to minimize the loss function. The performance of the multimodal model is then validated using the validation set to obtain a well-trained multimodal interpretable colorectal cancer prediction model. The pre-trained multimodal interpretable colorectal cancer prediction model was used to classify and predict the test set, determine whether the samples have tertiary lymphoid structures, and further predict the prognosis and treatment response of rectal cancer, thus obtaining the target multimodal interpretable colorectal cancer prediction model.
[0012] As a further improvement to this application, the prediction of rectal cancer prognosis and treatment response includes: The class association embedding method is used to perform interpretability analysis on classification results. Specifically, it involves drawing the class transformation path of samples, generating synthetic samples and comparing them to obtain a saliency map, which explains the basis of the model's decision-making. By combining gene coding information, genes that are significantly associated with the prognosis and treatment response of colorectal cancer were identified.
[0013] As a further improvement to this application, the training of the multimodal model based on the fusion encoding also includes: An optimization algorithm was used to optimize the training process and convergence speed of the initial colorectal cancer prediction model.
[0014] Secondly, this application provides a system for constructing a multimodal interpretable colorectal cancer prediction model, comprising: The acquisition module is used to acquire colorectal cancer gene expression data and digital pathology images; The extraction module is used to process digital pathology images using class association embedding methods to extract the class association codes of the images; The fusion module is used to fuse the class association coding of images with the coding of gene expression data to obtain fused coding; The training module is used to train the multimodal model based on the fusion encoding, perform classification prediction after training, and interpret the classification prediction results using interpretability analysis methods to obtain the target multimodal interpretable colorectal cancer prediction model.
[0015] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a multimodal interpretable colorectal cancer prediction model.
[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for constructing a multimodal interpretable colorectal cancer prediction model.
[0017] Fifthly, this application provides a computer program product comprising computer instructions that instruct a computer to execute the method for constructing a multimodal interpretable colorectal cancer post-prediction model.
[0018] The advantages of this application over the prior art are as follows: This application constructs a multimodal interpretable artificial intelligence model that integrates RNA-seq gene expression data with digital pathological images. First, class association embedding (CAE) is used to process the digital pathological images, extracting the class association codes from the images. These CAE codes are then fused with gene codes. Next, the fused codes are trained, and finally, a binary classification experiment of tertiary lymphoid structures is performed to determine whether they are positive or negative. Interpretability analysis methods are used to interpret the results, aiming to improve classification accuracy and identify key genes. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description of the relevant technical solutions in the embodiments of this application or the prior art is provided with accompanying drawings. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0020] Figure 1 This application provides a flowchart of a method for constructing a multimodal interpretable colorectal cancer prediction model. Figure 2 This is a diagram of the multimodal model architecture. Figure 3 This is a flowchart of gene embedding. Figure 4 Flowchart of the LRP interpretability method; Figure 5 This is a structural diagram of a multimodal interpretable colorectal cancer prediction model construction system according to this application. Detailed Implementation
[0021] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0022] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.
[0023] like Figure 1 As shown, this application overcomes the shortcomings of the prior art and provides a method for constructing a multimodal interpretable predictive model for colorectal cancer, including the following steps: S1, acquire colorectal cancer gene expression data and digital pathology images; S2, using class association embedding method to process digital pathological images and extract class association codes from the images; S3, the class association coding of the image is fused with the coding of gene expression data to obtain the fused coding; S4. The multimodal model is trained based on the fusion encoding. After training, classification prediction is performed, and the classification prediction results are interpreted using the interpretability analysis method to obtain the target multimodal interpretable colorectal cancer prediction model.
[0024] Based on the multimodal XAI model, and combined with class association embedding interpretable frameworks and gene expression data and digital pathological images, the effective fusion of multimodal information is achieved, improving classification accuracy and enhancing model interpretability. This enables accurate identification of genes significantly associated with the prognosis and treatment response of colorectal cancer, providing strong support for the research and treatment of colorectal cancer.
[0025] Furthermore, a multimodal interpretable artificial intelligence model was constructed by fusing RNA-seq gene expression data with digital pathological images. Class association embedding (CAE) was used to process the digital pathological images, extracting the class association codes from the images, and then fusing them with gene codes. The fused codes were then trained and optimized. Finally, a binary classification experiment of tertiary lymphoid structures was performed to determine whether they were positive or negative. Interpretability analysis was used to interpret the results, aiming to improve classification accuracy and identify key genes.
[0026] This application discloses a method for constructing a multimodal interpretable predictive model for colorectal cancer, the specific steps of which are as follows: Step 1: Data preprocessing; After obtaining colorectal cancer gene expression data and digital pathology images, the following is also included: Preprocessing of gene expression data includes: For RNA-seq gene expression data of colorectal cancer, gene expression data that meet the requirements are screened and retained, and TMM is used to standardize the gene expression data. After standardization, gene expression data are initially screened based on the correlation between gene expression data and colorectal cancer to obtain gene expression data with high disease correlation. For example, for RNA-seq gene expression data of tertiary lymphoid structures, genes with a CPM greater than the 50th percentile in at least 60% of the samples were retained. Subsequently, TMM was used for standardization to remove outliers and noise, ensuring data quality. Simultaneously, preliminary screening was performed based on the correlation between genes and colorectal cancer, retaining expression data of genes closely related to the disease.
[0027] Digital pathology images are cropped and normalized to ensure their size and pixel values meet the model input requirements.
[0028] CPM (Counts Per Million) is a standardization method commonly used in RNA sequencing (RNA-seq) data analysis to eliminate the impact of differences in sequencing depth between different samples on the comparison of gene expression levels.
[0029] Step 2: Class Association Embedding (CAE) Image Processing. The digital pathology image is processed using the class association embedding method to extract the class association codes of the image.
[0030] like Figure 2 The diagram shows the architecture of the multimodal model. The class association embedding method in this application employs a recurrent generative adversarial network (GAN) framework, including an encoder E (composed of encoding class association style and encoding individual style), a decoder G, and a multi-class discriminator D. The encoder decomposes digital pathology images into class association codes and individual codes. The class association codes encode image features relevant to the colorectal cancer classification task, while the individual codes encode image sample features independent of the class.
[0031] Figure 2 In the pre-training phase, the gene2vec method is first used to treat co-expressed genes as context, training to obtain an initial low-dimensional embedding vector for each gene. Further, the gene expression level information is converted into embedding vectors and fused with the initial gene coding vectors.
[0032] In the supervised fine-tuning phase, labeled RNA-seq data and digital pathology image data are used to fine-tune the entire model to achieve accurate prediction of the presence or absence of tertiary lymphoid structures. The specific process is as follows: A deep convolutional neural network (CNN) is used as the encoder to extract high-level features from the digital pathology images. This encoder maps the original image to a high-dimensional feature space, generating image embedding vectors. These image embedding vectors are decomposed into class-association encoding and individual-style encoding. Class-association encoding captures image features associated with specific categories (such as tertiary lymphoid structures), while individual-style encoding captures the differences between samples. Multimodal feature fusion: A multi-head attention mechanism is employed to fuse gene encoding and image encoding. This attention mechanism adaptively learns the correlations between different gene and image features, thus achieving more efficient feature fusion. The fused feature vector undergoes a non-linear transformation via a feedforward neural network to further extract deeper feature representations. The fused feature vector is then input into a classifier to predict whether a sample possesses a tertiary lymphoid structure. Interpretability analysis: By reconstructing gene expression profiles, this study analyzes which genes play a key role in the model's decision-making process. A layer-by-layer correlation propagation method is employed to backpropagate the model's predictions to the input layer, calculating the contribution of each gene to the predictions and identifying genes significantly associated with colorectal cancer prognosis and treatment response. Through progressive backpropagation of correlation scores, the genes with the greatest impact on the prediction results are ultimately determined.
[0033] Furthermore, by utilizing the Building Block Consistency Feature Extraction (BBCFE) method, class association features in individual code subspaces are identified and weakened by exchanging and combining the codes of different class samples, while encouraging the appearance of class association features in the class association code space, so as to efficiently learn the low-dimensional representation of class association features and separate the feature space.
[0034] Furthermore, a loss function is designed that includes reconstruction loss, adversarial loss, and classification loss. By combining these loss functions to train and update the encoder and decoder, the model is ensured to accurately extract the class association encoding of the image.
[0035] This application proposes a multimodal fusion method that effectively integrates RNA-seq gene expression data with digital pathology images through class association encoding and gene encoding to achieve information complementarity. An improved and applied class association embedding (CAE) method is used to process the digital pathology images, including its specific network architecture, BBCFE feature extraction method, and loss function design, to ensure accurate extraction of class association features from the images.
[0036] Step 3: Gene coding and fusion, specifically, the class association coding of the image and the coding of gene expression data are fused to obtain fused coding.
[0037] Figure 3 This is a flowchart of the gene embedding process. Borrowing from the gene2vec gene embedding method, the screened gene expression data is encoded. For example, a combination of gene embedding and expression embedding is used to generate gene coding vectors based on inter-gene correlations and expression levels. Then, the gene codes are fused with class association codes extracted from digital pathology images, so that the fused code contains comprehensive information from both genes and images.
[0038] Figure 3 In the input layer, words (such as "cat") are mapped to word vectors (dimensional V) through one-hot encoding (0 / 1). These vectors are then weighted by a weight matrix W (N×V, where N is the vocabulary size) to generate embedding vectors E. Multiple word vectors are aggregated into a hidden layer state h (dimensional V) through weighted summation (Σ). The hidden layer is multiplied by the weight matrix W (V×N) to calculate the unnormalized score. The Softmax layer converts the score into a probability distribution (e.g., given "cat", "like" has the highest probability, reaching 0.47). The output layer presents the probability results intuitively through numerical values and color depth. The core logic of matrix operations is: the product of the input word vector and the weight matrix extracts semantic features; the hidden layer aggregates contextual information; Softmax normalization achieves probability modeling; and finally, end-to-end learning from the vocabulary to probabilities is completed through dimension matching (N×V→V×N).
[0039] We further draw on the gene-encoding approach of gene2vec and integrate it with image encoding, as well as the training and optimization strategies used in the fusion model, to improve the model's classification accuracy and generalization ability.
[0040] Step 4: Model training and optimization. The multimodal model is trained and optimized based on the fusion encoding.
[0041] The fused encoded data is divided into training, validation, and test sets. The model is trained using the training set, and the loss function is minimized by adjusting model parameters, such as the weights and biases of the neural network.
[0042] As a preferred approach, the performance of the model is monitored using a validation set during training to prevent overfitting.
[0043] As a further preferred option, an optimization algorithm is also employed to accelerate the model training process and improve convergence speed.
[0044] Step 5: Classification and Interpretability Analysis. After training, classification prediction is performed, and the classification prediction results are interpreted using interpretability analysis methods. Figure 4 This is a flowchart of the LRP interpretability method.
[0045] Figure 4In the process, data starts from the left input layer and extracts features layer by layer through forward propagation matrix operations. The input features are used to generate intermediate features (feature maps) through weight matrix calculations, and the final output layer obtains the prediction results. Then, the correlation propagation is entered, and the output results are decomposed into correlation scores of each layer. Through layer-by-layer back projection, a heatmap is finally generated in the input layer, with red nodes highlighting the contribution of key features to the prediction.
[0046] As a preferred approach, a well-trained multimodal interpretable colorectal cancer prediction model is used to classify and predict the test set data, determine whether samples have grade 3 lymphoid structures, and further predict their prognosis and treatment response. Specifically, this includes: Based on local interpretation generation methods in CAE, interpretability analysis of classification results is performed on class-association manifolds. By plotting class transition paths of samples, synthetic samples are generated and compared to obtain a saliency map, which explains the basis of model decisions. Simultaneously, by combining gene coding information, the study analyzes which genes play key roles in the model decision-making process, thereby identifying genes significantly associated with colorectal cancer prognosis and treatment response.
[0047] Furthermore, interpretability analysis methods based on multimodal fusion can simultaneously explain the roles of genes and images in the prognosis and treatment response analysis of colorectal cancer, and identify key genes.
[0048] Through validation, the method presented in this application, compared to previous methods that processed gene expression data and image data separately, demonstrates that multimodal fusion can more comprehensively analyze colorectal cancer-related information, improving the understanding and analysis capabilities of the disease. Regarding classification accuracy, by fusing comprehensive information from genes and images, this application achieved higher accuracy in experiments, showing a significant improvement compared to methods based solely on single-modal data.
[0049] Regarding model interpretability, this application not only explains the decision-making basis for image classification but also, by combining genetic information, clearly reveals the relationship between genes and disease prognosis and treatment response, a significant shortcoming of existing technologies. Through interpretability analysis, key genes and lesions in images can be accurately identified, providing more valuable information for medical research and clinical applications, and helping doctors develop more precise treatment plans.
[0050] In handling complex data, this application utilizes CAE feature extraction and fusion methods, as well as effective training optimization strategies, to better cope with the complexity and diversity of gene expression data and pathological images, and improve the stability and reliability of the model. Existing technologies often perform poorly when handling such complex multimodal data.
[0051] This application has been proven feasible through experiments, and the results are shown in Table 1 below:
[0052] Compared with traditional models such as support vector machines, Gaussian Naive Bayes, random forests, stochastic gradient descent, multilayer perceptrons, and extreme gradient boosting, the multimodal interpretable colorectal cancer prediction model of this application improves the accuracy of binary classification experiments on the presence or absence of tertiary lymph nodes by 5%.
[0053] Through interpretability analysis, the multimodal interpretable colorectal cancer prediction model of this application successfully identified several known key genes closely related to the formation of tertiary lymphoid structures and discovered some potential new gene markers, which showed significant correlation with the disease in subsequent biological validation.
[0054] like Figure 4 As shown, the second objective of this application is to provide a system for constructing a multimodal interpretable predictive model for colorectal cancer. Based on the above-mentioned method for constructing a multimodal interpretable predictive model for colorectal cancer, the system includes: Acquisition module 100 is used to acquire colorectal cancer gene expression data and digital pathology images; The extraction module 200 is used to process digital pathological images using the class association embedding method to extract the class association codes of the images; The fusion module 300 is used to fuse the class association code of the image with the code of the gene expression data to obtain the fused code; The training module 400 is used to train the multimodal model based on the fusion encoding, perform classification prediction after training, and interpret the classification prediction results using the interpretability analysis method to obtain the target multimodal interpretable colorectal cancer prediction model.
[0055] A third objective of this application is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for constructing a multimodal interpretable colorectal cancer prediction model. The device also includes a communication interface and a bus.
[0056] A fourth objective of this application is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for constructing a multimodal interpretable colorectal cancer prediction model.
[0057] A fifth objective of this application is to provide a computer program product comprising computer instructions that instruct a computer to execute the above-described method for constructing a multimodal interpretable colorectal cancer prediction model.
[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0060] This application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, readable storage media, optical storage, etc.) containing computer-usable program code.
[0061] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0062] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort should fall within the scope of protection of this application.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.
Claims
1. A method for constructing a multimodal interpretable predictive model for colorectal cancer, characterized in that, include: Acquire gene expression data and digital pathology images of colorectal cancer; The class association embedding method is used to process digital pathology images and extract the class association codes of the images; The class association coding of the image is fused with the coding of gene expression data to obtain the fused coding; The multimodal model is trained based on the fusion encoding, and classification prediction is performed after training. The classification prediction results are then interpreted using interpretability analysis methods to obtain the target multimodal interpretable colorectal cancer prediction model.
2. The method of claim 1, wherein the method is characterized by, After acquiring colorectal cancer gene expression data and digital pathological images, the process also includes: Preprocessing of gene expression data includes: For RNA-seq gene expression data of colorectal cancer, gene expression data that meet the requirements are screened and retained, and TMM is used to standardize the gene expression data. After standardization, gene expression data are initially screened based on the correlation between gene expression data and colorectal cancer to obtain gene expression data with high disease correlation. Digital pathology images are preprocessed, including cropping and normalization.
3. The method of claim 1, wherein the method is characterized by, The class association embedding method adopts a class association embedding framework in the form of a recurrent generative adversarial network, including an encoder, a decoder and a multi-class discriminator. The encoder decomposes digital pathology images into class association codes and individual codes, wherein the class association codes are used to encode image features related to the colorectal cancer classification task, and the individual codes are used to encode image sample features that are not related to the class. By using the building block consistency feature extraction method, class association features in individual code subspaces are identified and weakened by exchanging and combining the codes of different class samples, and low-dimensional representations of class association features are learned and feature space is separated. A loss function is constructed that includes reconstruction loss, adversarial loss, and classification loss. The encoder and decoder are trained and updated using the loss function, enabling the class association embedding framework to extract the class association encoding of images.
4. The method of claim 1, wherein the method is characterized by, The process of fusing the class association coding of images with the coding of gene expression data to obtain fused coding includes: A combination of gene embedding and expression embedding is used to generate gene codes based on the correlation between genes and their expression levels; The gene code is fused with the class association code extracted from digital pathology images, so that the fused code contains comprehensive information from both the gene and the image.
5. The method of claim 1, wherein the method is characterized by, The process involves training a multimodal model based on the fusion encoding, performing classification predictions after training, and interpreting the classification prediction results using interpretability analysis methods to obtain a target multimodal interpretable colorectal cancer prediction model, including: The fused encoded data is divided into training set, validation set, and test set; The multimodal model is trained using the fused encoded data, and the parameters of the multimodal model are adjusted to minimize the loss function. The performance of the multimodal model is then validated using the validation set to obtain a well-trained multimodal interpretable colorectal cancer prediction model. The pre-trained multimodal interpretable colorectal cancer prediction model is used to classify and predict the test set, determine whether the samples in the test set have tertiary lymphoid structures to determine whether they are positive or negative, and further predict the prognosis and treatment response of rectal cancer, thus obtaining the target multimodal interpretable colorectal cancer prediction model.
6. The method for constructing a multimodal interpretable colorectal cancer prediction model according to claim 1, characterized in that, The prediction of rectal cancer prognosis and treatment response includes: The classification results are interpreted based on the interpretability analysis method. Specifically, the class transformation paths of the samples are plotted, synthetic samples are generated and compared, and a saliency map is obtained to explain the basis of the model's decision. By combining gene coding information, genes that are significantly associated with the prognosis and treatment response of colorectal cancer were identified.
7. The method for constructing a multimodal interpretable colorectal cancer prediction model according to claim 1, characterized in that, The training of the multimodal model based on the fusion encoding also includes: An optimization algorithm was used to optimize the training process and convergence speed of the initial colorectal cancer prediction model.
8. A system for constructing a multimodal interpretable predictive model for colorectal cancer, characterized in that, include: The acquisition module is used to acquire colorectal cancer gene expression data and digital pathology images; The extraction module is used to process digital pathology images using class association embedding methods to extract the class association codes of the images; The fusion module is used to fuse the class association coding of images with the coding of gene expression data to obtain fused coding; The training module is used to train the multimodal model based on the fusion encoding, perform classification prediction after training, and interpret the classification prediction results using interpretability analysis methods to obtain the target multimodal interpretable colorectal cancer prediction model.
9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a multimodal interpretable colorectal cancer prediction model as described in claim 8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for constructing a multimodal interpretable colorectal cancer prediction model as described in claim 8.