Disease prediction system, method, device and storage medium based on multi-modal omics

Abstract

The application provides a disease prediction system, method, device and storage medium based on multi-modal omics. The system comprises: a high-dimensional feature vector acquisition module for acquiring a plurality of high-dimensional feature vectors corresponding to pathological section images; a pathological feature vector determination module for determining a pathological feature vector corresponding to the high-dimensional feature vector based on an attention mechanism; a clinical feature vector determination module fused with the pathological feature determination module for multi-modal fusion of the pathological feature vector and the clinical feature vector to obtain a fused pathological feature; and a pathological prediction result output module for inputting the fused pathological feature into a multi-task classification network to obtain a pathological prediction result. The system effectively fuses the pathological feature vectors and the clinical feature vectors corresponding to different pathological categories, makes up for the deficiency of pathological prediction based on a single data source in the prior art, provides multi-dimensional cross-verification pathological information for pathological prediction, and improves the accuracy of the pathological prediction result.

Description

Technical Field

[0001] This application belongs to the field of medical artificial intelligence technology, and relates to a disease prediction system, method, device and storage medium based on multimodal omics. Background Technology

[0002] Lung cancer, the leading cause of cancer-related morbidity and mortality worldwide, has entered an era of precision medicine based on molecular subtyping in its treatment. Accurate identification of mutations in key driver genes such as EGFR, KRAS, and ALK is a core prerequisite for guiding targeted and immunotherapies. However, current gold standard next-generation high-throughput sequencing technology faces significant bottlenecks in clinical applications, including high testing costs, long turnaround times, and stringent requirements for biopsy sample quality. These factors collectively limit its accessibility in resource-constrained environments.

[0003] Against this backdrop, the application of artificial intelligence (AI) technology in lung cancer diagnosis and treatment is evolving from a single-modal analysis paradigm to a multi-modal information fusion paradigm. Nevertheless, existing AI models still have significant limitations, exhibiting low accuracy in disease prediction. Therefore, improving the accuracy of disease prediction has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a disease prediction system, method, device, and storage medium based on multimodal omics, for reducing the risk of disease.

[0005] While measuring costs, improve the accuracy of disease prediction.

[0006] In a first aspect, this application provides a disease prediction system based on multimodal omics. The system includes: a high-dimensional feature vector acquisition module for acquiring multiple high-dimensional feature vectors corresponding to pathological slide images; a pathological feature vector determination module connected to the high-dimensional feature vector acquisition module for determining pathological feature vectors corresponding to the high-dimensional feature vectors based on an attention mechanism; a clinical feature vector determination module for determining clinical feature vectors corresponding to the pathological slide images; a fused pathological feature determination module connected to both the pathological feature vector determination module and the clinical feature vector determination module for multimodal fusion of the pathological feature vectors and the clinical feature vectors to obtain fused pathological features; and a pathological prediction result output module connected to the fused pathological feature determination module for inputting the fused pathological features into a multi-task classification network to obtain pathological prediction results.

[0007] In this application, pathological feature vectors and clinical feature vectors are effectively fused to provide multi-dimensional cross-validated pathological information for pathological prediction, thereby improving the accuracy of the prediction results. Furthermore, the multimodal fusion in this application's embodiments overcomes the shortcomings of existing technologies that rely on a single data source for pathological prediction, achieving both high efficiency and high accuracy in mutation classification and recurrence prediction.

[0008] In one implementation of the first aspect, the high-dimensional feature vector acquisition module includes: a pathological slide image acquisition unit, a tissue region determination unit, a segmentation unit, and a high-dimensional feature vector extraction unit; the pathological slide image acquisition unit is used to acquire the pathological slide image; the tissue region determination unit is connected to the pathological slide image acquisition unit and is used to determine the tissue region corresponding to the pathological slide image; the segmentation unit is connected to the tissue region determination unit and is used to segment the tissue region to obtain multiple segmented sub-images; the high-dimensional feature vector extraction unit is connected to the segmentation unit and is used to perform feature extraction operations on each segmented sub-image to obtain a high-dimensional feature vector corresponding to each segmented sub-image.

[0009] In one implementation of the first aspect, the system further includes a preprocessing module; the preprocessing module is connected to the high-dimensional feature vector acquisition module and is used to perform preprocessing operations on the pathological slide image to obtain a preprocessed pathological slide image; wherein, the preprocessing operation includes a standardization process to eliminate noise in the pathological slide image and enhance the key features of the pathological slide image.

[0010] In one implementation of the first aspect, the pathological feature vector determination module includes an attention branch generation unit, an independent classifier construction unit, an attention score set output unit, and a pathological feature vector determination unit; the attention branch generation unit is used to split the attention module into n attention branches, wherein the attention mechanism includes the attention module; the independent classifier construction unit is connected to the attention branch generation unit and is used to construct an independent classifier in each attention branch; the attention score set output unit is connected to the independent classifier construction unit and is used to output an attention score set corresponding to the high-dimensional feature vector based on each independent classifier, wherein each attention score in the nth attention score set is used to determine the positive and negative features in the nth pathological category; the pathological feature vector determination unit is connected to the attention score set output unit and is used to determine the pathological feature vector corresponding to each pathological category based on the high-dimensional feature vector and each attention score set.

[0011] In one implementation of the first aspect, the fused pathological feature determination module includes a vector alignment unit, an attention feature vector determination unit, and a vector fusion unit; the vector alignment unit is used to perform an alignment operation on the pathological feature vector and the clinical feature vector; the attention feature vector determination unit, connected to the vector alignment unit, is used to perform fusion processing on the aligned pathological feature vector and the clinical feature vector based on single-head cross-attention to obtain an attention feature vector; the vector fusion unit, connected to both the attention feature vector determination unit and the clinical feature vector determination module, is used to perform multimodal fusion processing on the attention feature vector and the clinical feature vector to obtain fused pathological features.

[0012] In one implementation of the first aspect, the pathological prediction result output module includes a mutation classification branch unit and a recurrence prediction branch unit; the mutation classification branch unit is used to output pathological mutation prediction results; and the recurrence prediction branch unit is used to output pathological recurrence prediction results.

[0013] Secondly, this application provides a disease prediction method based on artificial intelligence, characterized in that the method includes:

[0014] Multiple high-dimensional feature vectors corresponding to pathological slide images are obtained; pathological feature vectors corresponding to the high-dimensional feature vectors are determined based on an attention mechanism; clinical feature vectors corresponding to the pathological slide images are determined; the pathological feature vectors and clinical feature vectors are fused in a multimodal manner to obtain fused pathological features; the fused pathological features are input into a multi-task classification network to obtain pathological prediction results.

[0015] In one implementation of the second aspect, the expression for determining the attention score set corresponding to the high-dimensional feature vector based on the attention mechanism is as follows:

[0016]

[0017] in, Indicates the first One attention branch, This represents the vector of the first shared attention encoding layer. This represents the vector of the second shared attention encoding layer. This represents the vector output by the first fully connected layer. This represents the first vector in the output of the first fully connected layer. Vector elements.

[0018] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the disease prediction method based on multimodal omics as described in any one of the second aspects of embodiments of this application.

[0019] Fourthly, embodiments of this application provide an electronic device, the electronic device comprising: a memory storing a computer program; and a processor communicatively connected to the memory, which, when the computer program is invoked, executes the disease prediction method based on multimodal omics as described in any one of the second aspects of embodiments of this application.

[0020] As described above, the disease prediction system, method, device, and storage medium based on multimodal omics described in this application have the following beneficial effects:

[0021] 1) This application effectively fuses pathological feature vectors and clinical feature vectors, providing multi-dimensional cross-validated pathological information for pathological prediction and improving the accuracy of pathological prediction results. Furthermore, the multimodal fusion in this application's embodiments overcomes the shortcomings of existing technologies that rely on a single data source for pathological prediction, achieving both high efficiency and high accuracy in mutation classification and recurrence prediction.

[0022] 2) In this application, by determining the tissue region corresponding to the pathological slide image, background information interference in the pathological slide image can be eliminated. This avoids the need to segment the background information in the pathological slide image subsequently. After eliminating background information interference, the focus can be on the tissue region, improving the accuracy and reliability of subsequent tasks such as cell segmentation, feature extraction, and pathological prediction. Simultaneously, it reduces the interference of noise and redundant data on the artificial intelligence model. The tissue region after eliminating background information is segmented to obtain multiple segmented sub-images. Feature extraction is then performed on each segmented sub-image to obtain a high-precision, high-dimensional feature vector corresponding to each segmented sub-image, providing an accurate vector foundation for subsequent pathological prediction. Through instance-level clustering loss and pseudo-label generation mechanisms, the slide-level supervision signal is extended to the patch level, forcing the model to learn a category-specific feature space. Combined with the noise robustness of the smoothed SVM loss function, the generalization ability in small sample scenarios is significantly improved, reducing the dependence on large-scale labeled data.

[0023] 3) This application determines the attention score set for each pathological category and selects feature vectors that are close to the pathological category from the high-dimensional feature vectors based on the attention score set for each pathological category. This achieves accurate feature selection of the pathological feature vectors corresponding to the pathological category and dynamically generates a customized pathological feature vector for each pathological category. This pathological feature vector eliminates the interference of irrelevant information and retains only the features most relevant to the current pathological category, which significantly improves the accuracy of the pathological feature vector and provides an accurate pathological feature basis for subsequent disease prediction.

[0024] 4) By aligning the pathological feature vector and the clinical feature vector, a unified-dimensional vector basis is provided for subsequent feature fusion. In this embodiment, the pathological feature vector and the clinical feature vector are fused to obtain the attention feature vector, which solves the problem of inaccurate prediction when predicting diseases based on a single data source in traditional technology. This greatly improves the richness of predicted pathological features and provides a rich and accurate vector basis for subsequent prediction of disease types. The attention-based dynamic fusion module performs cross-modal interaction between the pathological image features extracted by CLAM and clinical data (TNM staging, vascular invasion, pleural invasion). Through self-attention mechanism, it adaptively weights multi-source information, greatly improving the model's generalization ability. Attached Figure Description

[0025] Figure 1A The diagram shows the application scenario of the disease prediction system based on multimodal omics provided in the embodiments of this application.

[0026] Figure 1B The diagram shown is a structural diagram of a disease prediction system based on multimodal omics provided in an embodiment of this application.

[0027] Figure 1C The diagram shown is a structural diagram of another disease prediction system based on multimodal omics provided in an embodiment of this application.

[0028] Figure 2 The diagram shown is a structural diagram of the high-dimensional feature vector acquisition module provided in an embodiment of this application.

[0029] Figure 3 The diagram shown is a structural diagram of the pathological feature vector determination module provided in an embodiment of this application.

[0030] Figure 4 The diagram shown is a structural diagram of the fusion pathological feature determination module provided in an embodiment of this application.

[0031] Figure 5 The diagram shown is a structural diagram of the pathological prediction result output module provided in the embodiments of this application.

[0032] Figure 6The diagram shown is an overall structural diagram of an artificial intelligence-based disease prediction system provided in an embodiment of this application.

[0033] Figure 7 The diagram shown is a schematic of the training process for feature extraction based on the CLAM framework provided in an embodiment of this application.

[0034] Figure 8 The flowchart shown is a disease prediction method based on artificial intelligence provided in an embodiment of this application.

[0035] Figure 9 The diagram shown is a structural diagram of an electronic device provided in an embodiment of this application.

[0036] Component designation explanation

[0037] 100 Disease prediction system based on multimodal omics 1402 Attention Feature Vector Determination Unit 110 High-dimensional feature vector acquisition module 1403 Vector Fusion Unit 1101 Pathological slide image acquisition unit 150 Pathological prediction result output module 1102 Organizational area determination unit 1501 Mutant taxonomic branching unit 1103 Segmentation unit 1502 Recurrence prediction branch unit 1104 High-dimensional feature vector extraction unit 160 Preprocessing module 120 Pathological Feature Vector Determination Module 90 electronic devices 1201 Attention Branch Generation Unit 91 processor 1202 Independent classifier building blocks 92 Non-volatile storage media 1203 Attention Score Set Output Unit 93 System bus 1204 Pathological feature vector determination unit 94 Internal memory 130 Clinical Feature Vector Determination Module 95 Network interface 140 Integrated pathological feature determination module 1401 Vector alignment unit Detailed Implementation

[0038] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0039] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0040] The following embodiments of this application provide a disease prediction system, method, device, and storage medium based on multimodal omics, including but not limited to the hardware application scenarios listed in these embodiments.

[0041] like Figure 1A As shown in the diagram, this embodiment provides a hardware application scenario diagram corresponding to a multimodal omics-based disease prediction method, including: a medical image scanning device and an electronic device. The medical image scanning device and the electronic device are communicatively connected.

[0042] Medical image scanning equipment is used to acquire pathological slide images of users to be tested and send the pathological slide images to electronic devices. The electronic devices are used to perform a series of image and vector processing on the pathological slide images to obtain the pathological prediction results corresponding to the pathological slides.

[0043] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0044] like Figure 1B As shown in the diagram, this application provides a structural diagram of a disease prediction system based on multimodal omics. Figure 1B As shown, the disease prediction system 100 based on multimodal omics provided in this application includes a high-dimensional feature vector acquisition module 110, a pathological feature vector determination module 120, a clinical feature vector determination module 130, a fused pathological feature determination module 140, and a pathological prediction result output module 150.

[0045] The high-dimensional feature vector acquisition module 110 is used to acquire multiple high-dimensional feature vectors corresponding to the pathological slide image.

[0046] Specifically, the high-dimensional feature vector represents the vector representation corresponding to the pathological slide image.

[0047] The pathological feature vector determination module 120 is connected to the high-dimensional feature vector acquisition module and is used to determine the pathological feature vector corresponding to the high-dimensional feature vector based on an attention mechanism.

[0048] The clinical feature vector determination module 130 is used to determine the clinical feature vector corresponding to the pathological slide image.

[0049] Specifically, clinical data corresponding to the pathological slide images is acquired. This clinical data includes TNM staging, vascular invasion status, and pleural invasion status, corresponding cell invasion information / staging information / data. The clinical data is encoded into clinical feature vectors; for example, stages I and II are labeled as 0, and stages III and IV are labeled as 1, with 0 representing no invasion and 1 representing invasion.

[0050] It should be noted that all clinical feature vectors can be independently labeled with tumor boundaries and mutation types by certified pathologists, and verified through expert consensus to ensure the accuracy of the labeling.

[0051] The pathological feature determination module 140 is connected to the pathological feature vector determination module and the clinical feature vector determination module, respectively, and is used to perform multimodal fusion of the pathological feature vector and the clinical feature vector to obtain fused pathological features.

[0052] The pathological prediction result output module 150 is connected to the fused pathological feature determination module and is used to input the fused pathological features into the multi-task classification network to obtain the pathological prediction result.

[0053] The pathological prediction results include EGFR 19del / L858R mutation, other EGFR mutations, KRAS G12C mutation, other KRAS mutations, ALK fusion mutations, and other mutations, as well as the pathological prediction results corresponding to the recurrence risk prediction.

[0054] This application provides a disease prediction system based on multimodal omics. In this system, a high-dimensional feature vector acquisition module 110 acquires multiple high-dimensional feature vectors corresponding to pathological slide images; a pathological feature vector determination module 120 determines the pathological feature vectors corresponding to the high-dimensional feature vectors based on an attention mechanism; a pathological slide image acquisition unit 130 acquires the clinical feature vectors corresponding to the pathological slide images; a pathological feature fusion determination module 140 performs multimodal fusion of the pathological feature vectors and the clinical feature vectors to obtain fused pathological features; and a pathological prediction result output module 150 inputs the fused pathological features into a multi-task classification network to obtain pathological prediction results. This system effectively fuses pathological feature vectors and clinical feature vectors, providing multi-dimensional cross-validated pathological information for pathological prediction and improving the accuracy of pathological prediction results. Furthermore, the multimodal fusion in this application overcomes the shortcomings of existing technologies that rely on a single data source for pathological prediction, and this application combines high efficiency and high accuracy in mutation classification and recurrence prediction.

[0055] like Figure 1C As shown in the figure, this application provides a structural diagram of another disease prediction system based on multimodal omics, as follows: Figure 1C As shown in the diagram, another structural diagram of a disease prediction system based on multimodal omics provided in this application embodiment also includes a preprocessing module 160.

[0056] The preprocessing module 160 is connected to the high-dimensional feature vector acquisition module 110 and is used to perform preprocessing operations on the pathological slide image to obtain the preprocessed pathological slide image; wherein, the preprocessing operation includes a standardization process to eliminate noise in the pathological slide image and enhance the key features of the pathological slide image.

[0057] For example, the contrast-limited adaptive histogram equalization technique can be used to improve the image contrast of pathological slide images. Its core is to limit the contrast by cropping the local histogram to avoid over-enhancement. Then, adaptive Gamma correction is used to optimize brightness consistency, and morphological opening operation (using circular structuring elements with a radius of 5 pixels) is used to remove artifacts (such as dust and scratches) in the pathological slide images.

[0058] It should be noted that the specific steps for preprocessing pathological slide images listed in the above examples are merely illustrative. In practical applications, other suitable preprocessing steps can be performed on pathological slide images based on specific application requirements, and this application does not impose any restrictions on this.

[0059] Specifically, after preprocessing the pathological slide images, multiple high-dimensional feature vectors corresponding to the pathological slide images are determined to improve the accuracy of the high-dimensional feature vectors.

[0060] like Figure 2 As shown in the figure, this application embodiment provides a structural diagram of a high-dimensional feature vector acquisition module, as follows: Figure 2 As shown, the high-dimensional feature vector acquisition module 110 provided in this application embodiment includes a pathological slide image acquisition unit 1101, a tissue region determination unit 1102, a segmentation unit 1103, and a high-dimensional feature vector extraction unit 1104.

[0061] The pathological slide image acquisition unit 1101 is used to acquire the pathological slide image.

[0062] Specifically, the pathological slide images include lung cancer pathological slide images, gastric cancer pathological slide images, or pathological slide images corresponding to other suitable diseases, which will not be elaborated upon in this application.

[0063] The tissue region determination unit 1102 is connected to the pathological slide image acquisition unit 1101 and is used to determine the tissue region corresponding to the pathological slide image.

[0064] Specifically, the whole slide image (WSI) is first segmented using the HSV color space threshold to exclude the background from the whole slide image and retain only the tissue area.

[0065] The segmentation unit 1103 is connected to the tissue region determination unit 1102 and is used to segment the tissue region to obtain multiple segmented sub-images.

[0066] For example, the tissue region is divided into 256×256 pixel sub-patterns (patch) to lay the foundation for subsequent feature extraction.

[0067] The high-dimensional feature vector extraction unit 1104 is connected to the segmentation unit 1103 and is used to perform feature extraction operations on each segmented sub-image to obtain a high-dimensional feature vector corresponding to each segmented sub-image.

[0068] Specifically, by performing feature extraction on each segmented sub-image, a high-dimensional feature vector corresponding to each segmented sub-image is obtained, which can transform gigapixel-level pathological images into vector forms that are easy to compute, providing a vector basis for subsequent multimodal fusion and pathological prediction.

[0069] This application provides a structural diagram of a high-dimensional feature vector acquisition module, including: a pathological slide image acquisition unit 1101, a tissue region determination unit 1102, a segmentation unit 1103, and a high-dimensional feature vector extraction unit 1104. The pathological slide image acquisition unit is used to acquire the pathological slide image, and the tissue region determination unit is used to determine the tissue region corresponding to the pathological slide image. The segmentation unit is used to segment the tissue region to obtain multiple segmented sub-images. The high-dimensional feature vector extraction unit is used to perform feature extraction operations on each segmented sub-image to obtain a high-dimensional feature vector corresponding to each segmented sub-image. By determining the tissue region corresponding to the pathological slide image, background information interference in the pathological slide image can be eliminated, and the subsequent segmentation of background information in the pathological slide image can be avoided. After eliminating background information interference, the focus can be on the tissue region, improving the accuracy and reliability of subsequent tasks such as cell segmentation, feature extraction, and pathological prediction, while reducing the interference of noise and redundant data on the artificial intelligence model. Segmenting the tissue region after eliminating background information to obtain multiple segmented sub-images and performing feature extraction operations on each segmented sub-image can obtain a high-precision high-dimensional feature vector corresponding to each segmented sub-image, providing an accurate vector basis for subsequent pathological prediction.

[0070] By using instance-level clustering loss and pseudo-label generation mechanism, the slice-level supervision signal is extended to the patch level, forcing the model to learn a category-specific feature space. Combined with the noise robustness of the smooth SVM loss function, the generalization ability in small sample scenarios is significantly improved, and the dependence on large-scale labeled data is reduced.

[0071] like Figure 3 As shown in the figure, this application embodiment provides a structural diagram of a pathological feature vector determination module, as follows: Figure 3 As shown, the pathological feature vector determination module 120 provided in this application embodiment includes an attention branch generation unit 1201, an independent classifier construction unit 1202, an attention score set output unit 1203, and a pathological feature vector determination unit 1204.

[0072] The attention branch generation unit 1201 is used to split the attention module into n attention branches.

[0073] The attention mechanism includes the attention module.

[0074] For example, the n attention branches are parallel, and the n attention branches are represented as follows: .

[0075] The independent classifier construction unit 1202 is connected to the attention branch generation unit 1201 and is used to construct an independent classifier in each attention branch.

[0076] Specifically, the number of attention branches and independent classifiers is the same, and the n independent classifiers are in parallel.

[0077] Specifically, n independent classifiers can be represented as: .

[0078] The attention score set output unit 1203 is connected to the independent classifier construction unit 1202 and is used to output the attention score set corresponding to the high-dimensional feature vector based on each independent classifier.

[0079] Each attention score in the nth attention score set is used to determine the positive and negative features in the nth pathological category.

[0080] For example, the expression corresponding to the attention score set output by each of the independent classifiers for the high-dimensional feature vector is:

[0081]

[0082] in, Indicates the first One attention branch, This represents the vector of the first shared attention encoding layer. This represents the vector of the second shared attention encoding layer. This represents the vector output by the first fully connected layer. This represents the first vector in the output of the first fully connected layer. Vector elements.

[0083] The pathological feature vector determination unit 1204 is connected to the attention score set output unit 1203 and is used to determine the pathological feature vector corresponding to each pathological category based on the high-dimensional feature vector and each attention score set.

[0084] Specifically, the expression for determining the nth pathological category based on the high-dimensional feature vector and the nth attention score set is as follows:

[0085]

[0086] in, Represents a high-dimensional feature vector. This represents the nth attention score.

[0087] This application provides a structural diagram of a pathological feature vector determination module. In this structure, the attention score set corresponding to the high-dimensional feature vector is determined, and each pathological category has a specific attention score set. Each pathological feature vector can be obtained from the high-dimensional feature vector and the attention score set corresponding to each pathological category. This method determines the attention score set of each pathological category and, based on the attention score set of each pathological category, selects feature vectors that are close to the pathological category from the high-dimensional feature vectors. This achieves accurate feature selection of the pathological feature vectors corresponding to the pathological category, dynamically generating a customized pathological feature vector for each pathological category. This pathological feature vector eliminates the interference of irrelevant information and retains only the features most relevant to the current pathological category, significantly improving the accuracy of the pathological feature vector and providing an accurate pathological feature basis for subsequent disease prediction.

[0088] like Figure 4 As shown in the diagram, this application provides a structural diagram of a module for determining fused pathological features. Figure 4 As shown, the fusion pathological feature determination module 140 provided in this application embodiment includes: a vector alignment unit 1401, an attention feature vector determination unit 1402, and a vector fusion unit 1403.

[0089] The vector alignment unit 1401 is used to perform alignment operations on the pathological feature vector and the clinical feature vector.

[0090] The input port of the vector alignment unit 1401 is connected to the clinical feature vector determination module.

[0091] For example, pathological feature vectors and clinical feature vector (Dimensions are determined by the number of clinical variables) Aligned to a uniform dimension via a fully connected layer. .

[0092] It should be noted that the above embodiments illustrate the use of pathological feature vectors. and clinical feature vector The specific alignment dimensions are provided for illustrative purposes only. In practical applications, other suitable alignment dimensions can be selected based on specific application requirements, and this application does not impose any restrictions on this.

[0093] The attention feature vector determination unit 1402 is connected to the vector alignment unit 1401 and is used to perform fusion processing on the pathological feature vector and the clinical feature vector after the alignment operation based on single-head cross attention to obtain the attention feature vector.

[0094] Specifically, when using single-head cross-attention for fusion, the pathological feature vector serves as the Query, and the clinical feature vector serves as the Key and Value. The expression for calculating the attention score is as follows:

[0095]

[0096] in , , , Represents the attention feature vector ( Represents the learnable parameter matrix, Indicates the scaling factor.

[0097] The vector fusion unit 1403 is connected to the attention feature vector determination unit 1402 and the clinical feature vector determination module 130 respectively, and is used to perform multimodal fusion processing on the attention feature vector and the clinical feature vector to obtain fused pathological features.

[0098] Specifically, the output of the attention fusion module After layer normalization and residual connection processing, the expression corresponding to the fused pathological features is obtained as follows:

[0099]

[0100] This application provides a structural diagram of a pathological feature determination module. In this diagram, a vector alignment unit aligns the pathological feature vector and the clinical feature vector, providing a unified-dimensional vector foundation for subsequent feature fusion. The attention feature vector determination unit fuses the aligned pathological feature vector and the clinical feature vector based on single-head cross-attention to obtain an attention feature vector. The vector fusion unit performs multimodal fusion processing on the attention feature vector and the clinical feature vector to obtain fused pathological features. This solves the problem of inaccurate prediction when predicting diseases based on a single data source in traditional technologies. In this application embodiment, the pathological feature vector and the clinical feature vector are fused to obtain the attention feature vector, which greatly improves the richness of predicted pathological features and provides a rich and accurate vector foundation for subsequent disease type prediction. The attention-based dynamic fusion module performs cross-modal interaction between the pathological image features extracted by CLAM and clinical data (TNM staging, vascular invasion, pleural invasion). Through self-attention mechanism, it adaptively weights multi-source information, greatly improving the model's generalization ability.

[0101] like Figure 5 As shown in the figure, this application embodiment provides a structural diagram of a pathological prediction result output module, as follows: Figure 5 As shown, the pathological prediction result output module 150 provided in this application embodiment includes: a mutation classification branch unit 1501 and a recurrence prediction branch unit 1502;

[0102] The fusion pathological feature determination module 140 inputs the fusion pathological features into the mutation classification branch and the recurrence prediction branch in the pathological prediction result output module 150, respectively. The mutation classification branch unit 1501 is used to output the pathological mutation prediction result; the recurrence prediction branch unit 1502 is used to output the pathological recurrence prediction result.

[0103] For example, the mutation classification branch can output six pathological prediction results, and the recurrence prediction branch outputs two pathological recurrence results. The six pathological prediction results are: Category 1: EGFR 19del / L858R mutation, Category 2: other EGFR mutations, Category 3: KRAS G12C mutation, Category 4: other KRAS mutations, Category 5: ALK fusion mutation, and Category 6: other mutations and recurrence risk prediction. The two pathological recurrence results include recurrence or no recurrence.

[0104] Specifically, the training process for the output pathological prediction results and pathological recurrence results also includes determining the loss functions for the pathological prediction results and pathological recurrence results. The loss function corresponding to the pathological prediction results is the weighted cross-entropy. ( For the first Category weights (to address data imbalance) Indicates the first The mutation probability of a category Indicates the first Category labels, This represents the weighted cross-entropy corresponding to the pathological prediction results.

[0105] The recurrence prediction branch is handled similarly to the mutation branch, except that a fully connected layer (512→256→2) is used to output the probabilities of two types of recurrence risks. This design enables dynamic weight allocation of multimodal information, highlighting key diagnostic features.

[0106] Regarding the multi-task classifier training strategy, this application adopts a total loss function of... (Trading hyperparameters) =1.0, =0.8), where, This represents the weighted cross-entropy corresponding to the pathological recurrence result. During training, the Adam optimizer (initial learning rate 1e-4, weight decay 2e-4) is used, along with linear warm-up and cosine annealing to schedule the learning rate. Five-fold cross-validation is used during model training to ensure robustness.

[0107] It should be noted that the six pathological prediction results output by the mutation classification branches listed in the above embodiments are merely illustrative examples. In actual applications, other suitable pathological prediction results can be output by the mutation classification branches based on specific application requirements, and this application does not impose any restrictions on this.

[0108] like Figure 6 As shown in the figure, this application provides an overall structural diagram of disease prediction based on artificial intelligence. Figure 6 The various structures and steps described above Figures 1B to 5 The details have already been explained in the document, and this application does not impose any restrictions on them.

[0109] like Figure 7 As shown in the diagram, this application provides a schematic diagram of the training process for feature extraction based on the CLAM framework. Figure 7 As can be seen, under this framework, CLAM treats each WSI (called a "packet") as being composed of a large number of small regions or patches (called "instances").

[0110] Traditional MIL algorithms are limited to binary classification problems and employ rigid, non-trainable aggregation functions (such as max pooling). They only use the patch with the highest prediction probability for the positive class for slice-level prediction, making them unsuitable for multi-class classification or binary classification problems where no inherent positive / negative assumptions can be made.

[0111] Based on the technical problems existing in the prior art, the applicant proposes a CLAM architecture. CLAM solves multi-class classification problems by constructing multiple trainable attention-based pooling functions. In CLAM's multi-class attention pooling design, multiple parallel attention networks obtain n different attention score sets, corresponding to the n classes in the multi-class classification problem. This enables the model to clearly distinguish positive features and negative evidence in the morphological features of each class, and to summarize numerous sub-graph features into n slice-level feature representations.

[0112] Attention pooling is the core innovation of the CLAM method. It replaces aggregation functions such as max pooling in traditional MIL, achieving flexible and trainable aggregation from patch-level representation to slice-level representation. Specifically, after the WSI is segmented into sub-tiles, each sub-tile undergoes initial feature extraction using a ResNet-50 convolutional neural network pre-trained on ImageNet data, outputting a high-dimensional feature vector of dimension 1024, forming the feature vector. (k is the number of valid patch blocks). In CLAM, the first fully connected layer Represent each fixed 1024-dimensional patch-level feature Further compressed into a 512-dimensional vector Before feeding the high-dimensional feature vector into the attention module, two layers of shared attention encoding are used. Further processing is performed on the high-dimensional feature vectors.

[0113] The attention module is then split into n parallel attention branches. Simultaneously, n parallel independent classifiers are constructed to score the slice-level representation specific to each class. Based on this, the attention score of the k-th patch for the n-th class is... The slice-level representation of the nth class, obtained by aggregating attention score distributions, is given by formula (1). Given by formula (2):

[0114] (1)

[0115] (2)

[0116] Corresponding unnormalized slice-level scores Through the classifier layer Give, that is For reasoning, prediction scores are calculated at the slice level. The softmax function is used to calculate the predicted probability distribution for each category.

[0117] To further encourage the learning of category-specific features, an additional binary clustering objective is introduced during the training of the weakly supervised feature extractor. For each of the n categories, in the first fully connected layer... Then, a clustering layer with 512 hidden units is placed. If the weights of the clustering network corresponding to the nth class are expressed as... Then, it is the cluster assignment score predicted for the k-th subgraph. Given by formula (3):

[0118] (3)

[0119] Since patch-level labels are unavailable, the output of an attention network is used to generate pseudo-labels for each slice in each iteration of training to supervise clustering. Instead of clustering all patches in a slice, the objective is optimized only for subsets of patches that the model strongly focuses on or almost completely ignores. Intuitively, because each attention branch is supervised by slice-level labels during training, B patches with high attention scores (and thus positive clusters) are expected to be strong positive evidence for class Y, while B patches with low attention scores (and thus negative clusters) are expected to be strong negative evidence for class Y. Therefore, the clustering task can be intuitively interpreted as constraining the patch-level feature space. This makes the strong feature evidence and its negative evidence linearly separable for each category. For mutation type typing problems, it is generally reasonable to assume that the existence of representative tissue patches from different categories is a mutually exclusive event, thus imposing additional clustering supervision on the n-1 out-of-class attention branches as well.

[0120] For subgraph-level clustering tasks, we choose to use a smoothed top-1 SVM loss, which is based on the mature multi-class SVM loss. The smoothed variant (Equation 4) adds temperature scaling to the multi-class SVM loss. This gives it the properties of being infinitely differentiable and having non-sparse gradients, making it suitable for optimizing deep neural networks when the algorithm is implemented efficiently. Given a predicted score vector s and a true label y, the smooth SVM loss is defined as:

[0121] (4)

[0122] in, Indicates parameters, This represents the model's predicted score for the j-th category. The model represents the true category.

[0123] The predicted score of y.

[0124] During the training of the weakly supervised feature extractor, randomly sampled slices with a batch size of 1 are used. The multinomial sampling probability of each slice is inversely proportional to the frequency of its true class to alleviate class imbalance in the training set. The total loss for a given slice... It is slice-level classification loss Subgraph-level clustering loss The sum can be obtained through scalars. and Optional scaling:

[0125] (5)

[0126] calculate When using standard cross-entropy loss, Compare with true slice-level labels; calculate At that time, the instance-level clustering prediction score for each sampling patch is calculated using a binary smoothing SVM loss. The corresponding pseudo-clustering labels are compared. All experiments use the Adam optimizer to update parameters, with an L2 weight decay parameter of 1e-5 and a learning rate of 2e-4. After training, a feature extractor capable of extracting comprehensive pathological features is obtained. In subsequent multimodal multi-task classification tasks, we only use the attention pooling module within the attention module, without using the clustering molecular module. That is, the feature extractor extracts n feature representations representing n categories for each WSI dataset for subsequent training.

[0127] The scope of protection of the disease prediction system based on multimodal omics described in this application is not limited to the order of steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0128] This application also provides an artificial intelligence-based disease prediction method. The artificial intelligence-based disease prediction method can realize the multimodal omics-based disease prediction system described in this application. However, the implementation steps of the multimodal omics-based disease prediction method described in this application include, but are not limited to, the steps of the artificial intelligence-based disease prediction method listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0129] like Figure 8 As shown, in one embodiment, the disease prediction method based on artificial intelligence of this application includes the following steps S81 to S85.

[0130] S81, obtain multiple high-dimensional feature vectors corresponding to the pathological slide image.

[0131] In some embodiments, the expression for determining the attention score set corresponding to the high-dimensional feature vector based on the attention mechanism is:

[0132]

[0133] in, Indicates the first One attention branch, This represents the vector of the first shared attention encoding layer. This represents the vector of the second shared attention encoding layer. This represents the vector output by the first fully connected layer. This represents the first vector in the output of the first fully connected layer. Vector elements.

[0134] S82, determine the pathological feature vector corresponding to the high-dimensional feature vector based on the attention mechanism.

[0135] S83, determine the clinical feature vector corresponding to the pathological slide image.

[0136] S84, perform multimodal fusion of the pathological feature vector and the clinical feature vector to obtain fused pathological features.

[0137] S85, the fused pathological features are input into the multi-task classification network to obtain the pathological prediction results.

[0138] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0139] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0140] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] This application also provides an electronic device. Figure 9 The diagram shown is a structural schematic of an electronic device 90 in one embodiment of this application. The disease prediction system based on multimodal omics provided in this embodiment can be applied to... Figure 9 The electronic device shown is 90, but it is not limited to this. For example... Figure 9 As shown, the electronic device 90 includes a processor 91, a memory, a system bus 93, and a network interface 95. The memory may include a non-volatile storage medium 92 and internal memory 94.

[0142] The non-volatile storage medium 92 can store an operating system and a computer program. The computer program includes program instructions that, when executed, cause a processor to perform any of the multimodal omics-based disease prediction systems provided in the embodiments of this application.

[0143] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0144] The internal memory 94 provides an environment for the execution of a computer program in a non-volatile storage medium. When the computer program is executed by the processor, it enables the processor to execute any of the multimodal omics-based disease prediction systems provided in the embodiments of this application.

[0145] This network interface 95 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] It should be understood that processor 91 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.

[0147] The electronic device 90 in this application embodiment may include terminal devices such as tablet computers, laptop computers, mobile phones, supercomputers, and smart wearable devices. It can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0148] For example, electronic devices can be stations (STAION, ST) in WLANs, cellular phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, computers, laptops, handheld communication devices, handheld computing devices, and / or other devices for communicating over wireless systems, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0149] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0150] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0151] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0152] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0153] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A disease prediction system based on multimodal omics, characterized in that, The system includes: The high-dimensional feature vector acquisition module is used to acquire multiple high-dimensional feature vectors corresponding to pathological slide images; The pathological feature vector determination module is connected to the high-dimensional feature vector acquisition module and is used to determine the pathological feature vector corresponding to the high-dimensional feature vector based on an attention mechanism. A clinical feature vector determination module is used to determine the clinical feature vector corresponding to the pathological slide image; A pathological feature determination module is connected to both the pathological feature vector determination module and the clinical feature vector determination module, and is used to perform multimodal fusion of the pathological feature vector and the clinical feature vector to obtain fused pathological features. The pathological prediction result output module is connected to the fused pathological feature determination module and is used to input the fused pathological features into the multi-task classification network to obtain the pathological prediction result.

2. The system according to claim 1, characterized in that, The high-dimensional feature vector acquisition module includes: a pathological slide image acquisition unit, a tissue region determination unit, a segmentation unit, and a high-dimensional feature vector extraction unit; The pathological slide image acquisition unit is used to acquire the pathological slide image; The tissue region determination unit is connected to the pathological slide image acquisition unit and is used to determine the tissue region corresponding to the pathological slide image. The segmentation unit is connected to the tissue region determination unit and is used to segment the tissue region to obtain multiple segmented sub-images; The high-dimensional feature vector extraction unit is connected to the segmentation unit and is used to perform feature extraction operations on each segmented sub-image to obtain a high-dimensional feature vector corresponding to each segmented sub-image.

3. The system according to claim 1, characterized in that, The system also includes a preprocessing module; The preprocessing module is connected to the high-dimensional feature vector acquisition module and is used to perform preprocessing operations on the pathological slide image to obtain the preprocessed pathological slide image; wherein, the preprocessing operation includes a standardization process to eliminate noise in the pathological slide image and enhance the key features of the pathological slide image.

4. The system according to claim 1, characterized in that, The pathological feature vector determination module includes an attention branch generation unit, an independent classifier construction unit, an attention score set output unit, and a pathological feature vector determination unit. The attention branch generation unit is used to split the attention module into n attention branches, wherein the attention mechanism includes the attention module; The independent classifier construction unit is connected to the attention branch generation unit and is used to construct an independent classifier in each attention branch. The attention score set output unit is connected to the independent classifier construction unit and is used to output the attention score set corresponding to the high-dimensional feature vector based on each independent classifier, wherein each attention score in the nth attention score set is used to determine the positive and negative features in the nth pathological category. The pathological feature vector determination unit is connected to the attention score set output unit and is used to determine the pathological feature vector corresponding to each pathological category based on the high-dimensional feature vector and each attention score set.

5. The system according to claim 1, characterized in that, The fusion pathological feature determination module includes a vector alignment unit, an attention feature vector determination unit, and a vector fusion unit; The vector alignment unit is used to align the pathological feature vector and the clinical feature vector; The attention feature vector determination unit is connected to the vector alignment unit and is used to perform fusion processing on the pathological feature vector and the clinical feature vector after the alignment operation based on single-head cross attention to obtain the attention feature vector. The vector fusion unit is connected to the attention feature vector determination unit and the clinical feature vector determination module, respectively, and is used to perform multimodal fusion processing on the attention feature vector and the clinical feature vector to obtain fused pathological features.

6. The system according to claim 1, characterized in that, The pathological prediction result output module includes a mutation classification branch unit and a recurrence prediction branch unit; The mutation classification branch unit is used to output pathological mutation prediction results; The recurrence prediction branch unit is used to output pathological recurrence prediction results.

7. A disease prediction method based on artificial intelligence, characterized in that, The method includes: Obtain multiple high-dimensional feature vectors corresponding to pathological slide images; The pathological feature vector corresponding to the high-dimensional feature vector is determined based on the attention mechanism. Determine the clinical feature vector corresponding to the pathological slide image; The pathological feature vector and the clinical feature vector are fused using a multimodal method to obtain fused pathological features; The fused pathological features are input into a multi-task classification network to obtain pathological prediction results.

8. The method according to claim 7, characterized in that, The expression for determining the attention score set corresponding to the high-dimensional feature vector based on the attention mechanism is: in, Indicates the first One attention branch, This represents the vector of the first shared attention encoding layer. This represents the vector of the second shared attention encoding layer. This represents the vector output by the first fully connected layer. This represents the first vector in the output of the first fully connected layer. Vector elements.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 7 to 8.

10. An electronic device, characterized in that, The electronic device includes: A memory that stores a computer program; The processor, which is communicatively connected to the memory, executes the method of any one of claims 7 to 8 when the computer program is invoked.

Images