Mucus-containing brain tumor multi-source data auxiliary diagnosis system based on deep learning

Abstract

The invention relates to the field of medical artificial intelligence, in particular to a mucus-like brain tumor multi-source data auxiliary differential diagnosis system based on deep learning, and the system collects pathological images, medical images, immunohistochemical data and molecular biology data through a multi-source heterogeneous data collection and standardization module, and carries out standardization processing; a multi-scale feature extraction module extracts key features from the data to generate multi-modal feature data; a cross-modal correlation analysis module analyzes correlation among the feature data and generates attention mapping data and feature correlation data; the diagnosis reasoning and decision-making module identifies the tumor type by using the data, carries out differential diagnosis and calculates the uncertainty of a diagnosis result; the interpretability display module is used for generating a structured diagnosis report which comprises multi-level attention visualization data and positioning a key diagnosis area; and through multi-source data fusion and correlation analysis, the diagnosis accuracy is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical artificial intelligence, in particular to a deep learning-based mucus-rich brain tumor multi-source data assisted differential diagnosis system for integrating pathological images, medical images, immunohistochemistry and gene detection and other multi-source heterogeneous data to achieve accurate differential diagnosis of mucus-like brain tumors. BACKGROUND

[0002] Mucus-like brain tumors are a group of tumors rich in mucus-like matrix occurring in the central nervous system, which are easily confused due to their extremely similar histomorphology, including chordoma, chordoid meningioma, chondrosarcoma, chordoid glioma, mucoid glioma, chondromyxoid fibroma, intracranial Ewing's sarcoma, intracranial sarcoma with DICER1 gene mutation and other tumors. Although this group of tumors has very similar cell morphology and histological structure under a microscope, the histological grade and benignity and malignancy are different, so the treatment methods and prognosis are also very different. (Follow-up and review of recurrent cases) The misdiagnosis rate of this group of tumors has always been high, and the inter-hospital consultation coincidence rate is also low, which is a difficult point in the diagnosis of neuropathology sub-specialty.

[0003] The correct first diagnosis of intracranial tumors is crucial. At present, the diagnosis of mucus-like brain tumors mainly relies on the experience of pathologists, which is highly subjective and stressful. However, this group of tumors has many categories, and new defined tumors have been added to this family in recent years, making the differential diagnosis work complex and difficult, making accurate diagnosis more challenging. At the same time, there is a big difference in the diagnosis level between tertiary hospitals and primary hospitals, leading to unstable diagnosis results and high misdiagnosis rate. Although immunohistochemistry, gene detection and imaging data can provide important basis for diagnosis, these data have overlapping between tumors and cannot be obtained completely or analyzed in isolation in differential diagnosis, lacking effective integration with morphological features.

[0004] In addition, due to the relative rarity of mucus-like brain tumors, it is difficult to obtain large-scale samples for deep learning model training. Existing artificial intelligence assisted diagnosis systems are mostly based on a single data source, which cannot fully utilize the complementary information in multi-modal medical data. At the same time, existing systems are mostly "black box" models, lacking explainability, making it difficult to gain the full trust and adoption of clinicians.

[0005] Therefore, it is urgent to develop an assisted diagnosis system that can integrate multi-source heterogeneous medical data, mine cross-modal correlation information, adapt to small sample learning scenarios and have good explainability, in order to improve the diagnostic accuracy and consistency of mucus-like brain tumors. SUMMARY

[0006] The main purpose of the present application is to provide a deep learning-based mucoid brain tumor multi-source data auxiliary differential diagnosis system, which integrates pathological images, medical images, immunohistochemistry and gene detection and other multi-source heterogeneous data, mines the hidden association between different data sources, and realizes accurate differential diagnosis of mucoid brain tumors.

[0007] The present application provides a deep learning-based mucoid brain tumor multi-source data auxiliary diagnosis system, comprising:

[0008] A multi-source heterogeneous data acquisition and standardization module is used to:

[0009] Acquire multi-source heterogeneous medical data including pathological image data, medical image data, immunohistochemistry data and molecular biology data;

[0010] Standardize the multi-source heterogeneous medical data to generate standardized medical data;

[0011] Establish patient-level data association to generate associated standardized medical data;

[0012] A multi-scale feature extraction module is connected to the multi-source heterogeneous data acquisition and standardization module and is used to:

[0013] Receive the associated standardized medical data;

[0014] Extract pathological features, imaging features and molecular biology features from the associated standardized medical data;

[0015] Generate multi-modal feature data containing the pathological features, the imaging features and the molecular biology features;

[0016] A cross-modal association analysis module is connected to the multi-scale feature extraction module and is used to:

[0017] Receive the multi-modal feature data;

[0018] Analyze the cross-modal association in the multi-modal feature data through an inter-modal attention mechanism to generate attention mapping data;

[0019] Construct a feature association graph to analyze the feature association relationship in the multi-modal feature data to generate feature association data;

[0020] Generate associated feature data based on the attention mapping data and the feature association data;

[0021] A diagnosis reasoning and decision-making module is connected to the cross-modal association analysis module and is used to:

[0022] Receive the associated feature data;

[0023] Based on the association feature data, tumor type recognition and differential diagnosis are performed to generate diagnosis result data;

[0024] The uncertainty of the diagnosis result data is calculated to generate credibility score data;

[0025] An explainability display module is connected to the diagnosis reasoning and decision-making module, configured to:

[0026] Receive the diagnosis result data and the credibility score data;

[0027] Generate multi-level attention visualization data to locate key diagnostic areas;

[0028] Based on the diagnosis result data, the credibility score data and the multi-level attention visualization data, a structured diagnostic report is generated.

[0029] As a preferred, the multi-source heterogeneous data acquisition and standardization module comprises:

[0030] A multi-channel data acquisition unit is configured to acquire the multi-source heterogeneous medical data, wherein the multi-source heterogeneous medical data comprises:

[0031] 40x / 20x / 10x multi-magnification digital pathology section images;

[0032] CT and MRI sequences conforming to the DICOM standard;

[0033] Immunohistochemical marker data including Brachyury, PR, CK7 / 19, S100, EMA, SOX-9, SOX-10, DES, GFAP, SSTR2, TLE1, CD99, etc.;

[0034] Gene detection data including IDH1 / 2, EWSR1-FLI1, PDGFRA, PTEN, INI-1, DICER1, PRKCA, etc.;

[0035] A data standardization processing unit is connected to the multi-channel data acquisition unit and configured to perform standardization processing on the multi-source heterogeneous medical data, wherein the standardization processing comprises:

[0036] Color normalization and background area removal are performed on the pathological images;

[0037] Gray value standardization and spatial standardization are performed on the medical images;

[0038] Staining intensity quantification and positive rate calculation are performed on the immunohistochemical data;

[0039] Expression level standardization and mutation encoding are performed on the gene detection data;

[0040] patient-level data association unit, connected with the data standardization processing unit, for establishing multi-source data association of the same patient, and generating the associated standardized medical data.

[0041] Preferably, the multi-scale feature extraction module comprises:

[0042] a pathology image feature extraction unit for extracting the pathology features from the pathology image, wherein the pathology features comprise:

[0043] cell-level features including cytoplasm, nuclear morphology parameters, cytoplasmic and nuclear color preference parameters, and nuclear-cytoplasmic ratio and chromatin distribution features;

[0044] tissue-level features including cell arrangement patterns, cell cluster density, mucus matrix features, and tissue structure integrity features;

[0045] lesion-level features including tumor boundary features, necrotic area distribution, and calcification / cartilagization and ossification features;

[0046] a medical image feature extraction unit for extracting the imaging features from the medical image, wherein the imaging features comprise:

[0047] anatomic positioning features including tumor accurate position and surrounding structure relationship;

[0048] imaging omics features including gray level histogram features, texture features, and shape features;

[0049] multi-parameter quantitative features including ADC values, perfusion parameters, and enhancement features;

[0050] a feature extraction unit for extracting the molecular biology features from immunohistochemical protein level data and gene detection molecular level data, wherein the biology features comprise:

[0051] immunohistochemical marker expression pattern features including expression intensity, positive rate, and distribution features;

[0052] gene mutation features including driver mutations, fusion genes, and mutation combination patterns;

[0053] epigenetic features including methylation features and histone modification patterns;

[0054] a multi-modal feature fusion unit connected with the pathology image feature extraction unit, the medical image feature extraction unit, and the molecular feature extraction unit, for fusing the pathology features, the imaging features, and the molecular biology features to generate the multi-modal feature data.

[0055] Preferably, the cross-modal association analysis module comprises:

[0056] an inter-modal attention computing unit, configured to:

[0057] compute bidirectional attention weights between pathological features and molecular biological features, to generate pathological-molecular attention data;

[0058] compute bidirectional attention weights between imaging features and pathological features, to generate imaging-pathological attention data;

[0059] compute relevance attention weights between immunohistochemical features and gene expression features, to generate immun-gene attention data;

[0060] generate the attention mapping data based on the pathological-molecular attention data, the imaging-pathological attention data, and the immun-gene attention data;

[0061] a feature correlation graph constructing unit, configured to:

[0062] represent features in the multi-modal feature data as nodes of a correlation graph;

[0063] compute edge weights based on correlations between features;

[0064] analyze direct and indirect correlations between nodes through multi-layer graph structure feature propagation;

[0065] generate the feature correlation data;

[0066] a multi-task learning unit, connected to the inter-modal attention computing unit and the feature correlation graph constructing unit, configured to:

[0067] combine the attention mapping data and the feature correlation data to construct a multi-task learning framework;

[0068] simultaneously perform related tasks such as tumor typing, marker prediction, and malignancy assessment;

[0069] optimize feature representation through inter-task information sharing;

[0070] generate the correlation feature data.

[0071] As a preference, the diagnosis reasoning and decision module comprises:

[0072] a small sample learning unit, configured to:

[0073] construct tumor type feature prototypes based on the correlation feature data;

[0074] optimize feature space through metric learning to achieve effective learning of 5-10 samples per class;

[0075] An application meta-learning strategy is applied to quickly adapt to new tumor subtypes;

[0076] A data augmentation unit is configured to:

[0077] Simulate data changes under different acquisition conditions based on a physical model to generate augmented samples;

[0078] Create synthetic samples through a generative model to expand training data;

[0079] Apply an active learning strategy to identify high-value samples for labeling;

[0080] A Bayesian diagnosis unit is connected to the small sample learning unit and the data augmentation unit, and is configured to:

[0081] Perform Bayesian inference based on the associated feature data, the tumor type feature prototype, and the augmented samples;

[0082] Output a complete diagnosis probability distribution rather than a single-point prediction;

[0083] Generate the diagnosis result data;

[0084] Estimate prediction uncertainty through Monte Carlo sampling to generate the credibility score data.

[0085] Preferably, the explainability display module includes:

[0086] An attention visualization unit is configured to:

[0087] Generate high-resolution attention heat maps for key areas of pathological images;

[0088] Generate three-dimensional attention voxel maps for key areas of medical images;

[0089] Integrate attention information of different scales to form hierarchical attention representations from the global to the cellular level;

[0090] Generate the multi-level attention visualization data;

[0091] A feature contribution analysis unit is configured to:

[0092] Quantify the contribution of each feature to the diagnosis result;

[0093] Analyze the relative importance of different modal data to the diagnosis;

[0094] Extract feature points that are crucial to the current diagnosis;

[0095] A diagnosis report generation unit is connected to the attention visualization unit and the feature contribution analysis unit, and is configured to:

[0096] generate a hierarchical diagnosis report based on the diagnosis result data, the credibility score data, the multi-level attention visualization data, and the feature contribution analysis result;

[0097] provide diagnosis reasoning paths and differential diagnosis analysis;

[0098] provide risk stratification decision suggestions according to uncertainty levels;

[0099] generate the structured diagnosis report.

[0100] As preferred, the multi-modal feature fusion unit comprises:

[0101] a feature standardization sub-unit for performing scale unification and dimension alignment on the pathological features, the imaging features, and the molecular biology features;

[0102] a feature importance evaluation sub-unit connected to the feature standardization sub-unit, configured to:

[0103] calculate the importance of each feature based on information gain;

[0104] dynamically adjust feature weights for different tumor types;

[0105] identify key differential features;

[0106] a multi-level fusion sub-unit connected to the feature importance evaluation sub-unit, configured to:

[0107] perform feature-level fusion to splice different modal features into a unified vector;

[0108] perform decision-level fusion to integrate different modal independent decision results;

[0109] perform interactive fusion to consider the mutual influence between different modal features;

[0110] dynamically adjust modal fusion weights to generate the multi-modal feature data.

[0111] As preferred, the feature correlation graph construction unit constructs the feature correlation graph through the following steps:

[0112] represent cell-level features, tissue-level features, and lesion-level features as nodes at different levels;

[0113] establish edge connections based on the correlation, mutual information, and conditional probability between features;

[0114] assign higher initial weights to key nodes;

[0115] update node feature representations through multiple rounds of information transmission;

[0116] Through graph structure centrality analysis, identify the feature combination that plays a decisive role in diagnostic decision-making.

[0117] As a preferred, the data enhancement unit generates the enhanced sample by the following way:

[0118] Simulate different staining conditions for pathological images, including H&E concentration changes (randomly change within ±15%), staining time differences (randomly adjust within ±30%), and different batch staining effects;

[0119] Simulate different scanning parameters for medical images, including CT tube voltage changes (randomly adjust within 80-120kV), MRI TR / TE parameter changes, and different magnetic field strength effects;

[0120] Add real acquisition noise, including Gaussian noise (standard deviation randomly set within 0.01-0.05), Poisson noise, and salt and pepper noise (density randomly set within 0.01-0.03);

[0121] Based on the conditional generation network, generate synthetic samples that meet the characteristics of specific tumor types according to specified parameters.

[0122] As a preferred, the diagnostic report generation unit generates the structured diagnostic report including:

[0123] Summary section, containing diagnostic conclusion, confidence score and key evidence summary;

[0124] Detailed part, including complete feature description, differential diagnosis analysis and suggestion;

[0125] Evidence section, showing evidence supporting diagnosis according to intensity classification, including strong evidence, moderate evidence and auxiliary evidence;

[0126] Differential diagnosis section, list differential diagnosis according to possibility and provide key distinguishing features;

[0127] Suggestion section, provide hierarchical suggestions according to uncertainty level, including clear diagnostic suggestions in high certainty area, multiple possibility sorting in medium uncertainty area and further examination suggestions in high uncertainty area.

[0128] The beneficial effects of the present application include:

[0129] 1. Significant improvement in diagnostic accuracy: through multi-source data fusion and association analysis, the system diagnostic accuracy rate is improved from 76% of traditional single data source to 92%, especially in the differential diagnosis of morphologically similar tumors (such as chordoma vs. chondrosarcoma).

[0130] 2. Diagnosis consistency fundamentally improved: The system provides standardized diagnostic references, significantly reducing diagnostic differences between different levels of hospitals, and improving diagnosis consistency (Kappa coefficient) from 0.65 to 0.89.

[0131] 3. Revolutionary improvement in sample utilization efficiency: To address the challenge of the scarcity of mucoid brain tumor samples, the system uses small sample learning and data enhancement to achieve good performance with only 10-15 samples per class, improving sample utilization efficiency by 5-10 times.

[0132] 4. Qualitative leap in clinical decision-making transparency: Explainable decision support breaks the "black box" limitation of AI, providing clear diagnostic basis and evidence chain, and the proportion of doctors adopting AI suggestions increases from 51% to 87%.

[0133] 5. Innovative mode of medical knowledge inheritance: The system digitizes expert experience and realizes knowledge combination migration, accelerating the cultivation of young doctors' comprehensive thinking ability in rare tumor diagnosis, shortening the training period from 2-3 years to 6-12 months. BRIEF DESCRIPTION OF DRAWINGS

[0134] Figure 1 The overall architecture diagram of the deep learning-based mucoid brain tumor multi-source data assisted differential diagnosis system of the present application;

[0135] Figure 2 The structure diagram of the multi-source heterogeneous data acquisition and standardization module in the present application;

[0136] Figure 3 The structure diagram of the multi-scale feature extraction module in the present application;

[0137] Figure 4 The structure diagram of the cross-modal correlation analysis module in the present application;

[0138] Figure 5 The structure diagram of the diagnosis reasoning and decision-making module in the present application;

[0139] Figure 6 The structure diagram of the explainable display module in the present application;

[0140] Figure 7 The workflow diagram of the inter-modal attention mechanism in the present application;

[0141] Figure 8 The schematic diagram of feature correlation graph construction in the present application;

[0142] Figure 9 The data enhancement example diagram based on physical model in the present application;

[0143] Figure 10A hierarchical diagram of a structured diagnostic report in the present application. DETAILED DESCRIPTION

[0144] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings. Figures 1-10 The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0145] Referring to Figure 1 The deep learning-based mucus-like brain tumor multi-source data auxiliary differential diagnosis system provided by the present application comprises a multi-source heterogeneous data acquisition and standardization module 1, a multi-scale feature extraction module 2, a cross-modal correlation analysis module 3, a diagnosis reasoning and decision-making module 4, and an explainability display module 5.

[0146] The multi-source heterogeneous data acquisition and standardization module 1 is used to acquire multi-source heterogeneous medical data including pathological image data, medical image data, immunohistochemical data, and molecular biology data, to standardize these data, and to establish patient-level data correlation to generate correlated standardized medical data.

[0147] The multi-scale feature extraction module 2 is connected with the multi-source heterogeneous data acquisition and standardization module 1, and is used to extract pathological features, imaging features, and molecular biology features from the correlated standardized medical data to generate multi-modal feature data.

[0148] The cross-modal correlation analysis module 3 is connected with the multi-scale feature extraction module 2, and is used to analyze the cross-modal correlation and feature correlation in the multi-modal feature data to generate correlated feature data.

[0149] The diagnosis reasoning and decision-making module 4 is connected with the cross-modal correlation analysis module 3, and is used to identify and diagnose tumor types based on the correlated feature data to generate diagnosis result data and credibility score data.

[0150] The explainability display module 5 is connected with the diagnosis reasoning and decision-making module 4, and is used to generate multi-level attention visualization data, locate key diagnostic areas, and generate a structured diagnostic report.

[0151] Referring to Figure 2 The multi-source heterogeneous data acquisition and standardization module 1 comprises a multi-channel data acquisition unit 11, a data standardization processing unit 12, and a patient-level data correlation unit 13.

[0152] The multi-channel data acquisition unit 11 is used to acquire multi-source heterogeneous medical data. In a preferred embodiment of the present application, the acquired multi-source heterogeneous medical data comprises:

[0153] (1) 40x / 20x / 10x multi-magnification digital pathology slide images. Preferably, the resolution of digital pathology slide scanning is no less than 0.25 pm / pixel to ensure capturing subtle features of cell morphology. In practical applications, pathology slides are usually stained with H&E, and special stains such as PAS, Alcian blue, etc. can also be used for special purposes.

[0154] (2) DICOM standard-compliant CT and MRI sequences. Preferably, the matrix size of CT images is 512x512 pixels, and the matrix size of MRI images is 256x256 pixels. For MRI, multiple sequences such as T1-weighted, T2-weighted, FLAIR, DWI, enhancement, etc. can be acquired simultaneously to obtain more comprehensive tumor information.

[0155] (3) Immunohistochemical marker data, including but not limited to Brachyury, PR, CK7 / 19, S100, EMA, SOX-9, SOX-10, DES, GFAP, SSTR2, TLE1, CD99, etc. Comprehensive analysis of the above markers is of great value for the differential diagnosis of mucoid brain tumors. For example, Brachyury is strongly positive in chordoma, while it is usually negative in other mucoid tumors; GFAP positivity can roughly exclude chondrosarcoma and chordoma; moderate to strong diffuse nuclear positivity of TLE1 tends to be sarcoma.

[0156] (4) Gene detection data, including but not limited to IDH1 / 2, EWSR1-FLI1, PDGFRA, PTEN, INI-1, DICER1, PRKCA, etc. Key gene mutations or fusion information. These molecular changes have high specificity for specific tumor types, such as IDH1 / 2 mutations in chondrosarcoma and PDGFRA p.K385 mutations in mucoid neuronal tumors.

[0157] The data standardization processing unit 12 is connected with the multi-channel data acquisition unit 11, and is used for standardizing the acquired multi-source heterogeneous medical data. In an embodiment of the present application, the standardization processing includes:

[0158] (1) Color normalization and background region removal of pathology images. Color normalization aims to eliminate differences between different batches of staining, so that the color distribution of the image tends to be consistent. Preferably, color normalization is achieved by adjusting the mean value of each RGB channel to a pre-set standard value (R: 210±10, G: 180±10, B: 190±10). Background region removal identifies and eliminates non-tissue regions through threshold segmentation method, improving the efficiency and accuracy of subsequent analysis.

[0159] (2) Standardization of gray value and space of medical images. The gray value standardization can be achieved by Z-score method, which converts the gray value to a standard score:

[0160] ,

[0161] wherein: is the standardized gray value, is the original gray value, is the average gray value of the image, is the gray standard deviation of the image.

[0162] The spatial standardization is achieved by resampling images with different spatial resolutions to a unified voxel size (usually 1 mm x 1 mm x 1 mm) and registering to a standard template (such as MNI152 template).

[0163] (3) Quantification of staining intensity and calculation of positive rate for immunohistochemical data. The staining intensity is usually divided into four grades: negative (0), weakly positive (1+), moderately positive (2+), and strongly positive (3+). The positive rate is obtained by calculating the percentage of positive cells in total cells, which is usually divided into five intervals: <5%, 5-25%, 26-50%, 51-75%, and >75%. In practical application, H-score can be used to represent the staining intensity and positive rate comprehensively:

[0164] ,

[0165] wherein: is the staining intensity grade (0-3), is the percentage of cells with intensity grade . The H-score ranges from 0 to 300, and the higher the value, the higher the expression level.

[0166] (4) Standardization of expression level and mutation encoding for gene detection data. For gene expression data, TPM (Transcripts Per Million) method is usually used for standardization:

[0167] ,

[0168] wherein: is the standardized expression of gene , is the original expression of gene , is the total number of genes, represents the sum of all gene expression.

[0169] For gene mutation data, binary coding is usually used, i.e. 1 for mutation and 0 for no mutation. For fusion genes, multi-classification coding can be used according to the fusion type, such as SS18-SSX fusion gene can be coded as SS18-SSX1 (1, 0, 0), SS18-SSX2 (0, 1, 0) and SS18-SSX3 (0, 0, 1).

[0170] The patient-level data association unit 13 is connected with the data standardization processing unit 12, and is used to establish the association of multi-source data of the same patient and generate associated standardized medical data. In an embodiment of the present application, the patient-level data association is mainly realized through the unique patient identifier, and the time information of different examinations is considered to ensure the rationality of the association. In addition, the spatial correspondence between data of different scales, such as the regional correspondence between imaging data and pathological data, also needs to be established, which usually needs to be labeled with the help of expert knowledge.

[0171] Referring to Figure 3 The multi-scale feature extraction module 2 includes a pathological image feature extraction unit 21, a medical image feature extraction unit 22, a molecular feature extraction unit 23 and a multi-modal feature fusion unit 24.

[0172] The pathological image feature extraction unit 21 is used to extract pathological features from pathological images. In an embodiment of the present application, the extracted pathological features include:

[0173] (1) Cell-level features, including cytoplasm, nuclear morphology parameters, nuclear-cytoplasmic ratio and chromatin distribution features. Cytoplasm, nuclear morphology parameters such as area (usually in the range of 80-400 μm²), perimeter, circularity (a value between 0 and 1, 1 indicating perfect circularity), ratio of major axis to minor axis, etc. can be obtained by image segmentation and contour analysis. Cytoplasm and nucleus color preference parameters include strong and weak acidophilic, strong and weak basophilic, bichromatic and chromophobic, which can be obtained by histopathological examination and staining techniques. These methods can help to observe the distribution of internal structure and composition of cells, so as to classify and analyze the characteristics of cells; the nuclear-cytoplasmic ratio is the ratio of the area of the nucleus to the area of the cytoplasm, and a high nuclear-cytoplasmic ratio (>0.7) usually indicates malignancy. Chromatin distribution features can be divided into uniform type, granular type and clump type, which are usually extracted by texture analysis method.

[0174] (2) Tissue-level features, including cell arrangement patterns, cell cluster density, mucus matrix characteristics, and tissue structure integrity. Cell arrangement patterns such as cord-like, cluster-like, nest-like, and pseudo-gland-like are important diagnostic criteria and can be extracted through morphological operations and pattern recognition methods. Cell cluster density can be classified as oligocellular, medium density, and rich cell, which can be obtained through microscopic observation, cell counting, online measurement technology, and single-cell transcriptome sequencing. These methods have their own advantages and are suitable for different research needs. For example, microscopic observation is intuitive and convenient, cell counting is accurate and quantitative, and online measurement technology is suitable for large-scale cell culture monitoring. According to the specific experimental purpose, choosing the appropriate method can provide important data support for cell biology research. Mucus matrix characteristics such as mucus lake distribution density and area statistics can be obtained through color segmentation and region analysis. Tissue structure integrity is usually represented by differentiation degree score (1-3 levels), which can be obtained by analyzing the regularity of cell arrangement.

[0175] (3) Lesion-level features, including tumor boundary features, necrotic area distribution, and calcification / ossification / cartilage features. Tumor boundaries can be classified into two types: push and infiltration, which can be identified by boundary clarity and regularity analysis. Necrotic areas usually appear as acidophilic anuclear regions, which can be identified by color and texture features. Calcification / ossification lesions appear as deep purple dense regions, which can be extracted by color thresholding and morphological operations. Cartilage appears as intracellular lacunae, which can be identified and analyzed by microscopic observation, histological staining techniques, and image analysis techniques. These methods can help doctors and researchers understand the manifestations of cartilage at the cellular and molecular levels, providing important evidence for disease diagnosis and treatment. For example, in the diagnosis of chondrosarcoma, observing the distribution of lacunae and cartilage matrix can help determine the benignity or malignancy of the tumor.

[0176] The medical image feature extraction unit 22 is used to extract imaging features from medical images. In one embodiment of the present application, the extracted imaging features include:

[0177] (1) Anatomical localization features, including tumor precise location and surrounding structure relationship. Tumor location information is of great value for diagnosis, such as intracranial chordoma often occurs at the skull base, chordoid glioma often occurs in the third ventricle, and intracranial sarcoma with DICER1 gene mutation often involves the cerebral parenchyma. The surrounding structure relationship such as whether it invades important nerves and blood vessels can be obtained through 3D reconstruction and spatial analysis.

[0178] (2) Radiomics features, including gray-level histogram features, texture features, and shape features. Gray-level histogram features include mean, standard deviation, skewness, kurtosis, etc. Texture features are usually calculated based on gray-level co-occurrence matrix (GLCM), including energy, entropy, contrast, correlation, etc. Shape features include volume, surface area, sphericity, irregularity, etc.

[0179] (3) Multi-parametric quantitative features, including ADC value, perfusion parameters, and enhancement features. ADC (apparent diffusion coefficient) value reflects the diffusion property of tissue, usually calculated from DWI sequence. Perfusion parameters such as rCBV (relative cerebral blood volume) and rCBF (relative cerebral blood flow) can be obtained from perfusion-weighted imaging. Enhancement features include enhancement pattern (homogeneous / inhomogeneous) and enhancement degree, obtained by analyzing the images before and after contrast enhancement.

[0180] The molecular feature extraction unit 23 is used to extract molecular biology features from immunohistochemical data and gene detection data. In an embodiment of the present application, the extracted molecular biology features include:

[0181] (1) Immunohistochemical marker expression pattern features, including expression intensity, positive rate, and distribution characteristics. Different tumor types have characteristic immunohistochemical expression profiles, such as Brachyury (+), CK7 / 19 (+), S100 (focal +) for chordoma, SOX-9 (+), S100 (diffuse +), CK (-) for chondrosarcoma, CD99 (+), FLI1 (+), NKX2-2 (+) for Ewing's sarcoma, etc.

[0182] (2) Gene mutation features, including driver mutations, fusion genes, and mutation combination patterns. Key driver mutations such as INI-1 deletion are associated with poorly differentiated chordoma; IDH1 / 2 mutations occur in 40%-80% of chondrosarcomas; fusion genes such as EWSR1-FLI1 are molecular markers of Ewing's sarcoma; p.D463 missense mutation in PRKCA gene is almost universally present in chordoid glioma; PDGFRA p.K385 mutation is a specific change in myxoid neuronal tumors.

[0183] (3) Epigenetic features, including methylation features and histone modification patterns. Epigenetic changes such as the methylation status of O6-MGMT are found in recurrent chordoma, while the loss of H3K27me3, which plays an important regulatory role in gene expression and gene silencing, is associated with EZH2 mutations and has diagnostic value in poor prognosis chordoma. Methylation patterns can replace gene sequencing for tumor typing, with a high cost-benefit ratio.

[0184] The multi-modal feature fusion unit 24 is connected with the pathology image feature extraction unit 21, the medical image feature extraction unit 22 and the molecular feature extraction unit 23, and is configured to fuse the pathological features, the imaging features and the molecular biological features to generate multi-modal feature data.

[0185] The multi-modal feature fusion unit 24 includes a feature standardization subunit 241, a feature importance evaluation subunit 242 and a multi-level fusion subunit 243.

[0186] The feature standardization subunit 241 is configured to unify the scales and align the dimensions of the pathological features, the imaging features and the molecular biological features. The scale unification usually adopts a min-max standardization or a Z-score standardization method to adjust the numerical ranges of different features to the same interval. The dimension alignment is achieved by feature selection or dimension reduction techniques to ensure that different modal features have compatible dimension structures.

[0187] The feature importance evaluation subunit 242 is connected with the feature standardization subunit 241 and is configured to calculate the importance of each feature based on information gain, dynamically adjust the feature weights for different tumor types, and identify key discriminative features. The information gain calculation formula is as follows:

[0188] ,

[0189] wherein: is the feature , is the information gain of the target variable , is the entropy of the target variable , is the conditional entropy of the target variable under the condition of a given feature .

[0190] The entropy calculation formula is as follows:

[0191] ,

[0192] wherein: is the probability of the target variable taking a value , is the number of possible values of the target variable , indicates the summation of all possible target variable values.

[0193] The conditional entropy calculation formula is as follows:

[0194] ,

[0195] wherein: is the feature the probability that a feature takes a value the entropy of a target variable the probability that a feature takes a value the entropy of a target variable the number of possible values of a feature the sum of all possible values of a feature

[0196] The higher the information gain, the stronger the ability of the feature to distinguish the target variable. In practical applications, the features can be sorted by information gain, and the top K features can be selected as key discriminative features. Preferably, the value of K can be set to 20-50, and the specific value can be adjusted according to the total number of features and actual needs.

[0197] The multi-level fusion subunit 243 is connected with the feature importance evaluation subunit 242, and is used to perform feature-level fusion, decision-level fusion and interactive fusion, and dynamically adjust the fusion weights of each modality to generate multi-modal feature data.

[0198] (1) Feature-level fusion is a method of directly concatenating different modality features into a unified vector. Assuming that the pathology feature vector is , the imaging feature vector is , and the molecular biology feature vector is , the feature-level fusion result is:

[0199] ,

[0200] wherein: represents the pathology feature vector, and the dimension is ; represents the imaging feature vector, and the dimension is ; represents the molecular biology feature vector, and the dimension is ; represents a vector concatenation operation; represents the concatenated feature vector, and the dimension is .

[0201] (2) Decision-level fusion is a method of integrating independent decision results of different modalities. Assuming that the decision result based on pathology features is , the decision result based on imaging features is , and the decision result based on molecular biology features is , wherein is the number of tumor categories, the decision-level fusion result can be obtained by weighted average:

[0202] ​​​​ ,

[0203] wherein: is the decision result based on pathological features, with dimension ; is the decision result based on imaging features, with dimension ; is the decision result based on molecular biology features, with dimension ; , , are the weights of the pathological features, the imaging features and the molecular biology features respectively, and satisfy ; is the fused decision result, with dimension .

[0204] (3) Interactive fusion is a fusion method considering the mutual influence between different modal features. In the present application, interactive fusion is mainly realized through attention mechanism, which will be introduced in detail in the next part.

[0205] Referring to Figure 4 , the cross-modal correlation analysis module 3 comprises an inter-modal attention calculation unit 31, a feature correlation graph construction unit 32 and a multi-task learning unit 33.

[0206] The inter-modal attention calculation unit 31 is used to calculate the attention weights between different modal features and generate attention mapping data. In an embodiment of the present application, inter-modal attention calculation includes:

[0207] (1) Calculate the bidirectional attention weights between the pathological features and the molecular biology features, and generate pathological-molecular attention data. Specifically, for the pathological features and the molecular biology features , first map them to the same dimensional space through a projection matrix:

[0208] ,

[0209] ,

[0210] wherein: is the pathological feature vector, with dimension ; is the molecular biology feature vector, with dimension ; , , , , , is a projection matrix used to map the features into the query space, key space and value space; , , , , , are the query vector, key vector and value vector respectively.

[0211] Then, the attention weights are calculated and the weighted features are obtained:

[0212] ,

[0213] ,

[0214] ,

[0215] ,

[0216] wherein: denotes the attention weight of the pathological feature on the molecular feature; denotes the molecular feature weighted based on the pathological feature; denotes the attention weight of the molecular feature on the pathological feature; denotes the pathological feature weighted based on the molecular feature; is the dimension of the key vector, usually taking 64 or 128; softmax denotes the softmax function, which is used to convert the attention score into a probability distribution; the superscript denotes the matrix transposition operation.

[0217] (2) The bidirectional attention weights between the imaging features and the pathological features are calculated to generate image-pathology attention data. The calculation method is similar to the above and will not be repeated.

[0218] (3) The correlation attention weights between the immunohistochemical features and the gene expression features are calculated to generate immune-gene attention data. This step mainly focuses on the correlation within the molecular biology data, such as the corresponding relationship between specific immunohistochemical markers and gene expression patterns.

[0219] (4) Based on the above three types of attention data, comprehensive attention mapping data is generated. This can be achieved through simple weighted averaging or more complex fusion methods.

[0220] The feature correlation graph construction unit 32 is configured to construct a feature correlation graph, analyze the feature correlation relationship in the multi-modal feature data, and generate feature correlation data.

[0221] The feature correlation graph construction unit 32 constructs the feature correlation graph through the following steps:

[0222] (1) Represent cell-level features, tissue-level features, and lesion-level features as nodes at different levels. Suppose the cell-level feature set is , the organizational level feature set is , the lesion-level feature set is , then the node set of the feature association graph is .

[0223] in: Represents the i-th cell-level feature, i ranges from 1 to ; Represents the jth organizational level feature, j ranges from 1 to ; Represents the kth lesion-level feature, where k ranges from 1 to ; A node set representing the feature association graph, including feature nodes of all levels.

[0224] (2) Establish edge connections based on the correlation, mutual information and conditional probability between features. and , edge weight It can be calculated in the following ways:

[0225] Correlation coefficient: , in is the Pearson correlation coefficient.

[0226] Mutual Information: , in For mutual information, is a probability distribution.

[0227] Conditional probability: , represents a given feature Conditional characteristics The conditional probability of .

[0228] in: and Represents two nodes in the feature association graph; Representation node and The edge weights between ; Representation node and The absolute value of the Pearson correlation coefficient between them; Representation node and Mutual information between; Representation characteristics The value is And features The value is the joint probability of and respectively represent the marginal probability of feature with value and feature with value ; represents the conditional probability of feature given feature .

[0229] In practical applications, a threshold (typically 0.3-0.5) can be set, and an edge is established when .

[0230] (3) Assign higher initial weights to key nodes. Key nodes can be determined based on domain knowledge, such as Brachyury expression, IDH1 / 2 mutation, and other features highly specific to certain tumor types. Let the initial node weight vector be , the initial weight of the key node can be set to a higher value (such as 1.0), and the initial weight of other nodes can be set to a lower value (such as 0.5).

[0231] where: represents the initial node weight vector, with a dimension of the total number of nodes .

[0232] (4) Update node feature representation through multiple rounds of information transmission. Let the feature representation of node after the th iteration be , then the update formula for the th round is:

[0233] ,

[0234] where: represents the feature representation of node after the th iteration; represents the feature representation of node after the th iteration; represents the activation function (such as ReLU); and are weight matrices; represents the neighbor node set of node ; represents the edge weight between node and ; and represent node and the number of edges connected to the node; denotes the summation of all neighbor nodes of node .

[0235] (5) Through the analysis of the graph structure centrality, the feature combination having a decisive role in the diagnostic decision is identified. Common centrality measures include degree centrality, eigenvector centrality and PageRank centrality, etc. Taking the PageRank centrality as an example, the PageRank value of node is denoted as , and the calculation formula is as follows:

[0236] ,

[0237] wherein: denotes the PageRank value of node ; is a damping factor, and is usually taken as 0.85; is the total number of nodes; is the out-degree of node (the number of edges from the node); denotes the summation of all neighbor nodes of node ; denotes a random jump term, which ensures the convergence of the algorithm; denotes the PageRank value of node transferred to node .

[0238] Through the above steps, the feature association graph can be constructed, and the feature association data can be generated, including the node feature representation, the edge weight and the node importance, etc.

[0239] The multi-task learning unit 33 is connected with the inter-modal attention calculation unit 31 and the feature association graph construction unit 32, and is used to combine the attention mapping data and the feature association data, construct a multi-task learning framework, simultaneously execute related tasks such as tumor typing, marker prediction and malignancy evaluation, optimize the feature representation through the information sharing between tasks, and generate the associated feature data.

[0240] In an embodiment of the present application, the multi-task learning framework includes a shared feature extraction network and a plurality of task-specific output heads. Let the shared feature be F_s, and the output of each task can be represented as:

[0241] ,

[0242] wherein: denotes the shared feature; denotes the output head (a mapping function) of task . representing the task output; representing the total number of tasks.

[0243] The loss function of multi-task learning is the weighted sum of the loss of each task:

[0244]

[0245] wherein: represents the total loss function; represents the loss function of the task ; represents the weight of the task ; represents the weight of the task ; represents the weight of the task represents the sum over all tasks. For classification tasks, cross-entropy loss is usually used; for regression tasks, mean square error loss is usually used.

[0246] Preferably, the task weights can be dynamically adjusted according to the relevance and importance of the tasks. For example, the tumor typing is set as the main task, and the weight can be set to 0.5-0.6; the immunohistochemical marker prediction and gene mutation prediction are set as auxiliary tasks, and each weight can be set to 0.1-0.2; the malignancy evaluation is set as a secondary task, and the weight can be set to 0.1-0.15.

[0247] Referring to Figure 5 , the diagnostic reasoning and decision module 4 includes a small sample learning unit 41, a data enhancement unit 42, and a Bayesian diagnostic unit 43.

[0248] The small sample learning unit 41 is used to construct tumor type feature prototypes based on associated feature data, optimize the feature space through metric learning, realize effective learning of 5-10 samples per class, and apply a meta-learning strategy to realize rapid adaptation to new tumor subtypes.

[0249] In an embodiment of the present application, small sample learning is implemented based on Prototypical Networks. The core idea of Prototypical Networks is to calculate a prototype for each class, and then classify based on the distance between the sample and the prototype. Specifically, given a support set wherein is a sample, is a class label, is the total number of classes. First, the sample is mapped to the feature space by the feature extraction network . Then, the prototype of each class is calculated:

[0250]

[0251] ​​​where: represents the prototypes of the classes ; represents the support sets of the classes ; represents the number of samples in the support set ; represents the sum of all samples in the support set of the class ; represents the representation of the sample in the feature space.

[0252] For a query sample , first extract its features , then calculate its distance to each class prototype, and get the class probability distribution through the softmax function:

[0253] ,

[0254] where: represents the probability that the query sample belongs to the class ; represents the representation of the query sample in the feature space; represents the distance function, usually using Euclidean distance or cosine distance; represents the exponential function; represents the sum over all classes.

[0255] The training objective of the prototype network is to minimize the negative log-likelihood loss:

[0256] ,

[0257] where: represents the loss function; represents the parameters of the feature extraction network; represents the true class of the query sample ; represents the probability that the query sample is correctly classified.

[0258] In practical applications, the prototype network is usually trained in a meta-learning way, that is, multiple N-way K-shot tasks (N represents the number of classes, K represents the number of samples per class) are constructed, each task contains a support set and a query set. For small sample learning of mucinous brain tumors, you can set N=5 (corresponding to 5 main tumor types) and K=5-10 (5-10 samples per class).

[0259] The data augmentation unit 42 is used to simulate data changes under different acquisition conditions based on a physical model, create synthetic samples through a generative model, and apply an active learning strategy to identify high-value samples for labeling, thereby expanding the training data.

[0260] According to claim 9, the data augmentation unit 42 generates augmented samples by:

[0261] (1) Simulate different staining conditions for pathological images, including H&E concentration changes (randomly change within ±15%), staining time differences (randomly adjust within ±30%), and different batch staining effects. This physical model-based augmentation method can simulate variations in the actual staining process and improve the model's adaptability to different staining conditions.

[0262] In specific implementation, H&E concentration changes can be simulated by adjusting the intensity of the RGB channel:

[0263]

[0264] wherein: represents the original RGB value; represents the adjusted RGB value; represents a random disturbance factor, ranging from

[0265] Staining time differences can be simulated by adjusting color saturation:

[0266]

[0267]

[0268]

[0269] wherein: RGB2HSV represents the conversion function from RGB color space to HSV color space; HSV2RGB represents the conversion function from HSV color space to RGB color space; represents the hue, saturation, and lightness in the HSV color space; represents the adjusted saturation; represents the saturation disturbance factor, ranging from ​​​​​​​​​​​​​​​​represents the final adjusted RGB value.

[0270] (2) Simulate different scanning parameters for medical images, including CT tube voltage changes (randomly adjusted within the range of 80-120 kV), MRI TR / TE parameter changes, and different magnetic field strength effects. This enhancement method can simulate imaging effects under different devices and different scanning protocols, improving the generalization ability of the model.

[0271] CT tube voltage changes affect image contrast and noise level, which can be simulated by the following formula:

[0272] ,

[0273] where: represents the original image; represents the simulated image; represents the original tube voltage; represents the target tube voltage; represents the noise term, which usually follows a Gaussian distribution with mean 0 and standard deviation related to the tube voltage.

[0274] MRI parameter changes mainly affect tissue contrast, which can be simulated by adjusting the signal intensity of different tissues:

[0275] ,

[0276] where: represents the original signal intensity of a specific tissue; represents the simulated signal intensity; represents the original TE value; represents the target TE value; represents the T2 relaxation time of the tissue; represents the exponential function.

[0277] (3) Add real acquisition noise, including Gaussian noise (standard deviation randomly set within the range of 0.01-0.05), Poisson noise, and salt and pepper noise (density randomly set within the range of 0.01-0.03). Noise simulation can improve the robustness of the model to low-quality images.

[0278] (4) The addition formula of Gaussian noise is:

[0279] ,

[0280] where: represents the original image; represents the image after adding noise; represents a Gaussian distribution with mean 0 and standard deviation ​denotes the standard deviation of the noise, which is randomly set in the range of 0.01-0.05.

[0281] Poisson noise simulates the statistical fluctuations of photon counting, which is suitable for low-dose CT images:

[0282] ,

[0283] where: denotes the original image; denotes the image after adding noise; denotes the Poisson distribution with parameter ; denotes the scaling factor, which is usually set to 10-50.

[0284] The addition of salt and pepper noise involves randomly setting the pixel value to the maximum or minimum value:

[0285] ,

[0286] where: denotes the pixel value of the original image at position ; denotes the pixel value of the image after adding noise at position ; denotes a random value at position , ranging from [0, 1]; denotes the noise density, which is randomly set in the range of 0.01-0.03.

[0287] (4) Conditional generation network based on the specified parameters to generate synthetic samples that meet the characteristics of specific tumor types. Conditional Generative Adversarial Network (Conditional GAN) is a generative model that can generate samples that meet specific conditions. In this invention, the conditions can be tumor type, cell morphology parameters, or tissue structure characteristics, etc.

[0288] The objective function of the generator G and discriminator D of the conditional GAN is:

[0289] ,

[0290] where: denotes the generator; denotes the discriminator; denotes the objective function of the generative adversarial network; denotes the real sample; denotes random noise; denotes the condition; denotes the real data distribution; denotes the noise distribution (usually standard normal distribution); denotes the expectation; represents the natural logarithm; Represents the discriminator's discrimination result on the real sample (condition is ); Represents the samples generated by the generator (given noise and conditions ); Represents the discriminator's discrimination result on the generated sample.

[0291] During training, the generator Learning to convert random noise and conditions Mapping to realistic samples , and the discriminator Learning to distinguish between real and generated samples while taking into account the conditions After training is completed, you can enter specific conditions into the generator and random noise To generate samples that meet this condition.

[0292] Optimally, for each tumor type, 100-500 synthetic samples can be generated to expand the training dataset. The quality of the generated samples can be evaluated by expert scoring, and only samples with a score of more than 4.0 (out of 5.0) are included in the training set.

[0293] The Bayesian diagnosis unit 43 is connected to the small sample learning unit 41 and the data enhancement unit 42, and is used to perform Bayesian reasoning based on the associated feature data, tumor type feature prototypes and enhanced samples, output a complete diagnosis probability distribution rather than a single point prediction, generate diagnosis result data, and estimate the prediction uncertainty through Monte Carlo sampling to generate credibility score data.

[0294] In one embodiment of the present invention, Bayesian deep learning is implemented by Monte Carlo Dropout. Specifically, Dropout is kept activated during the test phase, multiple forward propagations are performed to obtain multiple prediction results, and then the distribution characteristics of these results are statistically analyzed. Let the model parameters be , the input is , then the predicted distribution can be expressed as:

[0295] ,

[0296] in: Represents a given input and training data Conditional output The posterior distribution of Represents a given input and model parameters Conditional output the conditional distribution of the predictive distribution of the posterior distribution of the predictive distribution of the parameter of the the parameter of the the number of samples (usually set to 30-50) the integral operation the average of the average of

[0297] Based on the prediction results of multiple sampling, the mean and variance of prediction can be calculated:

[0298] ,

[0299] ,

[0300] where: the mean of prediction the variance of prediction the prediction result of the the average of the average of The prediction variance can be used as a measure of uncertainty, which can be converted into a confidence score:

[0301] ,

[0302] where: Confidence represents the confidence score (0-100); exp represents the exponential function; the scaling factor, usually set to 1-5, so that the confidence score is distributed between 0-100; Var the variance of prediction

[0303] In practical applications, a confidence threshold can be set, such as 85%, when the predicted confidence exceeds the threshold, the system directly outputs the diagnosis result; when the confidence is between 60-85%, the system suggests the doctor to combine with clinical judgment; when the confidence is less than 60%, the system suggests additional examination or expert consultation.

[0304] Referring to Figure 6 , the explainability display module 5 includes an attention visualization unit 51, a feature contribution analysis unit 52, and a diagnosis report generation unit 53.

[0305] The attention visualization unit 51 is configured to generate high-resolution attention heatmaps of key regions of pathological images, three-dimensional attention voxel maps of key regions of medical images, and integrate attention information of different scales to form hierarchical attention representations from the global to the cellular level, and generate multi-level attention visualization data.

[0306] In an embodiment of the present application, the attention visualization is based on Grad-CAM++ technology. Grad-CAM++ is an improved version of Grad-CAM, which can more accurately locate multiple target regions and generate smoother heatmaps. For the intermediate layer feature map of the convolutional neural network , the Grad-CAM++ weight of class c is calculated as follows:

[0307] ,

[0308] wherein: represents the Grad-CAM++ weight of class c for the feature map ; represents the predicted score of class c; represents the activation value of the feature map at position (i,j); represents the second-order partial derivative of with respect to ; represents the first-order partial derivative of with respect to ; represents the rectified linear unit activation function, defined as ; represents the sum over all positions (i,j) of the feature map. Then, the attention heatmap of class c is calculated as follows:

[0309] ,

[0310] wherein: represents the Grad-CAM++ heatmap of class c; represents the weighted sum over all feature maps.

[0311] The resulting heatmaps are usually low-resolution, which need to be up-sampled to the original image resolution and overlaid with the original image for display.

[0312] For the three-dimensional attention voxel map of medical images, Grad-CAM++ can be extended to 3D convolutional networks to calculate the importance score of each voxel. In order to improve the visualization effect, three-dimensional rendering techniques such as maximum intensity projection (MIP) or volume rendering can be used to visually display the 3D attention voxel map.

[0313] To realize the hierarchical attention representation from global to cell level, attention information can be extracted at different levels of the network. The attention map at the shallow layer captures detailed features such as cell morphology, and the attention map at the deep layer captures global features such as tissue structure. By fusing attention maps at different levels, a more comprehensive hierarchical attention representation can be obtained.

[0314] The feature contribution analysis unit 52 is used to quantify the contribution of each feature to the diagnostic result, analyze the relative importance of different modal data for diagnosis, and extract feature points that are crucial to the current diagnosis.

[0315] In an embodiment of the present application, the feature contribution analysis is based on SHAP (SHapley Additive exPlanations) values. SHAP values are Shapley values based on game theory, used to explain the predictions of machine learning models. For feature i, its SHAP value is calculated as follows:

[0316]

[0317] wherein: represents the SHAP value of feature ; represents the set of all features; represents a subset that does not contain feature ; represents the number of features in the subset ; represents the number of all features; represents the model prediction using only the feature subset ; represents the model prediction using the feature subset plus feature ; represents the summation over all subsets that do not contain feature ; represents the combination coefficient, and represents the factorial.

[0318] SHAP values have three important properties: local accuracy, missing consistency, and symmetry, making them an ideal tool for explaining model predictions. By calculating the SHAP value of each feature, its contribution to the diagnostic result can be quantified, thereby identifying key features.

[0319] For the analysis of the relative importance of different modal data, the SHAP values of the features of the same modality can be summed to obtain the overall contribution of the modality:

[0320]

[0321] wherein:​ represents the overall contribution of a certain modality; represents the sum of all features within this modality; represents the feature 's SHAP value.

[0322] By comparing the overall contribution of different modalities, we can determine which modality is most important for the current diagnosis. For example, for chordoma, molecular biology features such as Brachyury expression may have higher contribution; while for chondrosarcoma, pathological features such as mucous matrix features may be more important.

[0323] In order to extract the feature points that are crucial to the current diagnosis, we can set a SHAP value threshold, such as 0.05, and when the absolute SHAP value of a feature exceeds this threshold, it is considered a key feature point. These key feature points will be highlighted in the diagnosis report, helping doctors understand the basis of the diagnosis.

[0324] The diagnosis report generation unit 53 is connected to the attention visualization unit 51 and the feature contribution analysis unit 52, and is used to generate a hierarchical diagnosis report based on the diagnosis result data, the reliability score data, the multi-level attention visualization data and the feature contribution analysis results, provide diagnosis reasoning path and differential diagnosis analysis, and provide risk stratification decision suggestions according to the uncertainty level, and generate a structured diagnosis report.

[0325] The structured diagnosis report generated by the diagnosis report generation unit 53 includes:

[0326] (1) Summary section, containing diagnosis conclusion, reliability score and key evidence summary. The diagnosis conclusion directly gives the tumor type, such as "chordoma", "chondrosarcoma" etc. The reliability score is presented in percentage form, indicating the reliability of the diagnosis. The key evidence summary lists 3-5 findings with the most diagnostic value, such as "strong positive expression of Brachyury", "typical physical vacuole cell morphology" etc.

[0327] (2) Detailed section, containing complete feature description, differential diagnosis analysis and suggested measures. The complete feature description systematically presents pathological, imaging and molecular biology features, such as cell morphology, tissue structure, tumor location, enhancement characteristics, immunohistochemical expression profile and gene mutation status, etc. The differential diagnosis analysis discusses other possible diagnoses and exclusion reasons. The suggested measures give follow-up treatment suggestions according to the diagnosis result and reliability, such as whether additional examination is needed, whether expert consultation is recommended, etc.

[0328] (3) Evidence section, which displays supporting evidence for diagnosis in hierarchical order, including strong evidence, moderate evidence, and ancillary evidence. Strong evidence is a decisive feature with high specificity, such as a characteristic gene mutation or immunohistochemical marker; moderate evidence is a supportive feature but not specific; ancillary evidence is a general feature that provides background support for diagnosis. Each piece of evidence can be traced back to the original data, making it easy for doctors to verify.

[0329] (4) Differential diagnosis section, which lists differential diagnoses in order of likelihood and provides key distinguishing features. For each differential diagnosis, the system calculates its probability and ranks it, while listing key differences from the primary diagnosis. For example, for a case diagnosed as spinal cord tumor, chondrosarcoma may be the primary differential diagnosis, and the key distinguishing features may be "Brachyury expression positive vs. negative", "S100 focal positive vs. diffuse positive", etc.

[0330] (5) Recommendation section, which provides hierarchical recommendations based on uncertainty levels, including clear diagnostic recommendations for high certainty areas, multiple possibilities sorted for medium uncertainty areas, and further examination recommendations for high uncertainty areas. When the credibility is higher than 85%, the system gives a clear diagnostic recommendation; when the credibility is between 60-85%, the system lists multiple possibilities and ranks them; when the credibility is less than 60%, the system suggests additional examination or expert consultation to improve the accuracy of the diagnosis.

[0331] In practical application, structured diagnostic reports are usually generated in PDF or HTML format, containing text, charts and images. The report adopts a hierarchical structure, allowing doctors to quickly access key information from the overview, and also to view detailed content in depth. The images and data in the report can be interactively displayed, facilitating exploratory analysis by doctors.

[0332] The above detailed the specific implementation of the present application based on deep learning Mucoid brain tumor multi-source data assisted differential diagnosis system. The system integrates multi-source heterogeneous medical data, mines cross-modal correlation information, applies small sample learning and data enhancement technology, and provides interpretable diagnostic decision support, effectively solving the accuracy, consistency and interpretability problems in mucoid brain tumor diagnosis, and has significant clinical application value.

Claims

1. A multi-source data-assisted differential diagnosis system for myxoid brain tumors based on deep learning, characterized by: include: Multi-source heterogeneous data collection and standardization module, used for: Collect multi-source heterogeneous medical data including pathological image data, medical imaging data, immunohistochemistry data and molecular biology data; performing standardization processing on the multi-source heterogeneous medical data to generate standardized medical data; Establish patient-level data association and generate associated standardized medical data; The multi-scale feature extraction module is connected to the multi-source heterogeneous data acquisition and standardization module and is used to: receiving the associated standardized medical data; extracting pathological features, imaging features, and molecular biological features from the associated standardized medical data; generating multimodal feature data comprising the pathological features, the imaging features, and the molecular biological features; The cross-modal correlation analysis module is connected to the multi-scale feature extraction module and is used to: receiving the multimodal feature data; Analyzing cross-modal correlations in the multimodal feature data through an inter-modal attention mechanism to generate attention mapping data; Constructing a feature association graph, analyzing the feature association relationship in the multimodal feature data, and generating feature association data; generating associated feature data based on the attention mapping data and the feature association data; The diagnostic reasoning and decision-making module is connected to the cross-modal association analysis module and is used to: receiving the associated feature data; Based on the associated feature data, tumor type identification and differential diagnosis are performed to generate diagnostic result data; Calculating the uncertainty of the diagnosis result data to generate credibility score data; The explainability display module is connected to the diagnostic reasoning and decision-making module and is used to: receiving the diagnosis result data and the credibility score data; Generate multi-level attention visualization data to locate key diagnostic areas; A structured diagnosis report is generated based on the diagnosis result data, the credibility score data and the multi-level attention visualization data.

2. The system according to claim 1, wherein: The multi-source heterogeneous data acquisition and standardization module includes: A multi-channel data acquisition unit is used to acquire the multi-source heterogeneous medical data, wherein the multi-source heterogeneous medical data includes: 40× / 20× / 10× multi-magnification digital pathology slide images; DICOM-compliant CT and MRI sequences; Immunohistochemical marker data including Brachyury, PR, CK7 / 19, S100, EMA, SOX-9, SOX-10, DES, GFAP, SSTR2, TLE1, and CD99; Genetic testing data including IDH1 / 2, EWSR1-FLI1, PDGFRA, PTEN, INI-1, DICER1, and PRKCA; A data standardization processing unit is connected to the multi-channel data acquisition unit and is used to perform standardization processing on the multi-source heterogeneous medical data, wherein the standardization processing includes: Perform color normalization and background area removal on pathological images; Perform grayscale value standardization and spatial standardization on medical images; The immunohistochemical data were quantified for staining intensity and the positive rate was calculated; Normalize expression levels and encode mutations for gene detection data; The patient-level data association unit is connected to the data standardization processing unit and is used to establish multi-source data association for the same patient and generate the associated standardized medical data.

3. The system according to claim 1, wherein: The multi-scale feature extraction module includes: A pathological image feature extraction unit is used to extract the pathological features from the pathological image, wherein the pathological features include: Cell-level characteristics, including cytoplasm and nuclear morphology parameters, cytoplasmic and nuclear chromatin parameters, nuclear-cytoplasmic ratio, and chromatin distribution characteristics; Tissue-level characteristics, including cell arrangement patterns, cell cluster density, mucus matrix characteristics, and tissue structural integrity characteristics; Lesion-level characteristics, including tumor boundary characteristics, necrotic area distribution, and calcification / ossification foci characteristics; A medical image feature extraction unit is used to extract the imaging features from the medical image, wherein the imaging features include: Anatomical localization features, including the precise location of the tumor and its relationship to surrounding structures; Radiomics features, including grayscale histogram features, texture features, and shape features; Multi-parameter quantitative features, including ADC values, perfusion parameters, and enhancement characteristics; A feature extraction unit is used to extract the molecular biological features from the protein level data of immunohistochemistry and the molecular level data of genetic testing, wherein the biological features include: Immunohistochemical marker expression pattern characteristics, including expression intensity, positive rate, and distribution characteristics; Gene mutation characteristics, including driver mutations, fusion genes, and mutation combination patterns; epigenetic signatures, including methylation profiles and histone modification patterns; A multimodal feature fusion unit is connected to the pathological image feature extraction unit, the medical imaging feature extraction unit and the molecular feature extraction unit, and is used to fuse the pathological features, the imaging features and the molecular biology features to generate the multimodal feature data.

4. The system according to claim 1, wherein: The cross-modal association analysis module includes: Inter-modal attention computation unit, used to: Calculate the bidirectional attention weights between pathological features and molecular biological features to generate pathological-molecular attention data; Calculate the bidirectional attention weights between imaging features and pathological features to generate imaging-pathology attention data; Calculate the association attention weights between immunohistochemical features and gene expression features to generate immune-gene attention data; generating the attention mapping data based on the pathology-molecular attention data, the imaging-pathology attention data, and the immune-gene attention data; Feature association graph construction unit, used for: Representing features in the multimodal feature data as nodes of an association graph; Calculate edge weights based on the correlation between features; Analyze direct and indirect connections between nodes through multi-layer graph structure feature propagation; generating the feature association data; A multi-task learning unit, connected to the inter-modal attention calculation unit and the feature association graph construction unit, is used to: Combining the attention mapping data and the feature association data to construct a multi-task learning framework; Simultaneously perform related tasks such as tumor typing, marker prediction, and malignancy assessment; Optimize feature representation by sharing information between tasks; The associated feature data is generated.

5. The system according to claim 1, wherein: The diagnostic reasoning and decision-making module includes: Small sample learning units for: constructing a tumor type feature prototype based on the associated feature data; Optimize the feature space through metric learning to achieve effective learning of 5-10 samples per class; Applying meta-learning strategies to achieve rapid adaptation to new tumor subtypes; Data augmentation unit, used to: Based on the physical model, the data changes under different acquisition conditions are simulated to generate enhanced samples; Create synthetic samples through generative models to expand training data; Apply active learning strategies to identify high-value samples for annotation; A Bayesian diagnosis unit, connected to the small sample learning unit and the data enhancement unit, is used to: performing Bayesian inference based on the associated feature data, the tumor type feature prototype, and the enhanced sample; Output a complete diagnosis probability distribution rather than a single point prediction; generating the diagnosis result data; The prediction uncertainty is estimated through Monte Carlo sampling to generate the credibility score data.

6. The system according to claim 1, wherein: The explainability display module includes: Attention visualization unit, used for: Generate high-resolution attention heatmaps of key areas in pathological images; Generate 3D attention voxel maps of key areas in medical images; Integrate attention information at different scales to form a hierarchical attention representation from the global to the cellular level; generating the multi-level attention visualization data; Feature contribution analysis unit, used to: Quantify the contribution of each feature to the diagnosis result; Analyze the relative importance of different modality data for diagnosis; Extract feature points that are crucial for current diagnosis; A diagnostic report generating unit, connected to the attention visualization unit and the feature contribution analysis unit, is configured to: generating a hierarchical diagnosis report based on the diagnosis result data, the credibility score data, the multi-level attention visualization data, and the feature contribution analysis result; Provide diagnostic reasoning pathways and differential diagnosis analysis; Provide risk-stratified decision recommendations based on uncertainty levels; The structured diagnostic report is generated.

7. The system according to claim 3, wherein: The multimodal feature fusion unit includes: a feature normalization subunit, configured to perform scale unification and dimension alignment on the pathological features, the imaging features, and the molecular biological features; The feature importance evaluation subunit is connected to the feature normalization subunit and is used to: Calculate the importance of each feature based on information gain; Dynamically adjust feature weights for different tumor types; Identify key distinguishing features; The multi-level fusion subunit is connected to the feature importance evaluation subunit and is used to: Perform feature-level fusion to concatenate features from different modalities into a unified vector; Perform decision-level fusion to integrate independent decision results from different modalities; Perform interactive fusion, considering the mutual influence between different modal features; Dynamically adjust the fusion weights of each modality to generate the multimodal feature data.

8. The system according to claim 4, wherein: The feature association graph construction unit constructs the feature association graph by the following steps: Represent cell-level features, tissue-level features, and lesion-level features as nodes at different levels; Establish edge connections based on correlation, mutual information, and conditional probability between features; Assign higher initial weights to key nodes; Update node feature representation through multiple rounds of information transmission; Through graph structure centrality analysis, the feature combination that plays a decisive role in diagnostic decision-making is identified.

9. The system according to claim 5, characterized in that The data enhancement unit generates enhanced samples in the following manner: Simulate different staining conditions for pathology images, including H&E concentration changes (randomly changed within ±15%), staining time differences (randomly adjusted within ±30%), and different batches of staining effects; Simulate different scanning parameters for medical imaging, including changes in CT tube voltage (randomly adjusted within the range of 80-120kV), changes in MRI TR / TE parameters, and the effects of different magnetic field intensities; Add real acquisition noise, including Gaussian noise (standard deviation randomly set in the range of 0.01-0.05), Poisson noise and salt and pepper noise (density randomly set in the range of 0.01-0.03); Based on the conditional generative network, synthetic samples that meet the characteristics of specific tumor types are generated according to specified parameters.

10. The system according to claim 6, wherein: The structured diagnostic report generated by the diagnostic report generating unit includes: Summary section, including diagnostic conclusion, confidence score and summary of key evidence; A detailed section containing a complete description of features, differential diagnosis analysis, and recommended actions; The evidence section displays the evidence supporting the diagnosis according to the strength level, including strong evidence, moderate evidence and supporting evidence; The differential diagnosis section lists the differential diagnoses in order of likelihood and provides key distinguishing features; The recommendations section provides graded recommendations based on the level of uncertainty, including clear diagnosis recommendations in high-certainty areas, multiple possibility rankings in moderate-uncertainty areas, and further examination recommendations in high-uncertainty areas.

Images