Oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system

Abstract

This invention relates to a hierarchical, cross-modal, multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma (OSCC), comprising: a data acquisition and preprocessing module for acquiring and preprocessing multimodal data including imaging, pathology, clinical, and genetic data; a sub-modal feature extraction module for extracting depth features from imaging and pathology data as image-based modalities, and encoding features from clinical and genetic data as structured modalities; a hierarchical cross-modal fusion module for performing three levels of fusion: intra-modal fusion, intra-modal fusion, and cross-modal fusion, and assigning fusion weights according to the clinical priority of each modality; and a multi-task prediction module for simultaneously performing OSCC detection, invasiveness scoring, and metastasis risk prediction tasks. This technical solution solves the problems of insufficient heterogeneous multimodal data fusion, poor model interpretability, single prediction task, and poor robustness when modalities are missing, and is suitable for OSCC-assisted diagnosis and treatment in medical institutions at all levels.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and oral medicine, specifically to a hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma. Background Technology

[0002] Oral squamous cell carcinoma (OSCC) is one of the most common malignant tumors of the head and neck region. Its global incidence has been rising steadily in recent years, becoming a major public health problem threatening oral health and life safety. OSCC is characterized by strong local invasiveness and a high tendency to metastasize to cervical lymph nodes. It not only causes damage to the oral structure and loss of physiological functions such as chewing and swallowing, severely impacting patients' quality of life, but also results in a consistently low overall five-year survival rate, placing a heavy disease burden on patients. With the continued advancement of my country's strategy of shifting the focus of cancer prevention and control to earlier stages, and the explicit requirements of the "Healthy China" initiative for the construction of an early screening, diagnosis, and treatment system for malignant tumors, improving the early identification capability of OSCC and achieving standardization and precision throughout the entire diagnosis and treatment process has become a core clinical challenge urgently needing to be addressed in my country's oral medicine field.

[0003] However, current clinical diagnosis and risk assessment of OSCC still face multiple technical bottlenecks, severely hindering the achievement of early diagnosis and treatment goals. First, OSCC itself is characterized by indistinct lesion boundaries, strong intratumoral and intertumoral heterogeneity, and dynamic multi-stage progression of carcinogenesis. Traditional imaging and pathological diagnosis heavily relies on physicians' clinical experience and subjective judgment, resulting in inherent defects such as time-consuming image reading, poor diagnostic consistency, and high rates of missed or misdiagnosed early small lesions. This makes it difficult to achieve standardized, precise, and quantitative assessment of the invasive characteristics of OSCC, failing to meet the clinical needs of precision medicine. Second, the occurrence and development of OSCC involve complex mechanisms such as altered tumor cell heterogeneity, tumor microenvironment remodeling, and abnormal regulation of multiple pathway signals. Single-dimensional diagnostic strategies can only capture local features of the disease and cannot comprehensively reflect the overall disease picture, making it difficult to achieve comprehensive assessment and precise stratification of OSCC.

[0004] With the rapid development of artificial intelligence and deep learning technologies, AI-assisted diagnostic technology based on multimodal data fusion has become a research hotspot in the field of precision diagnosis and treatment of OSCC. By integrating multi-dimensional data such as imaging, pathology, clinical data, and genes, it is expected to break through the limitations of traditional diagnosis and achieve objective, standardized, and intelligent detection of OSCC.

[0005] However, existing technical solutions still suffer from three major unavoidable core technical defects in clinical application: First, existing OSCC multimodal fusion solutions generally employ unconstrained data-driven attention mechanisms. Relying solely on data fitting to complete the weight allocation between modalities, without embedding the gold standard rules and domain prior knowledge of OSCC clinical diagnosis and treatment, leads to a disconnect between the model's weight allocation logic and the underlying cognition of clinical diagnosis and treatment. This not only easily results in problems such as feature bias and overfitting caused by pure data-driven approaches, but also makes the model output results highly black-box, making it impossible for clinicians to trace or understand them, and making it difficult to truly gain the trust and application of the clinical side.

[0006] Second, most existing OSCC multimodal fusion technologies employ a conventional design of direct splicing and disordered parallel fusion of multimodal features, failing to fully consider the essential differences in the characteristics of commonly used multimodal data in OSCC diagnosis and treatment. CT / MRI images and pathological whole-slice images belong to spatial image data with strong feature homology, while clinical structured data and gene expression data belong to non-image structured data, with significant differences in feature space and dimensionality distribution between the two. The conventional disordered fusion approach cannot fundamentally solve the core problem of feature space mismatch between heterogeneous data, making it difficult to fully explore the complementary features between different modalities, resulting in insufficient model generalization ability and significant performance fluctuations in multi-center clinical applications.

[0007] Third, existing OSCC multimodal AI models almost all require complete input of full-modal data to operate stably, completely ignoring the reality of uneven distribution of medical resources in my country. In particular, in vast regions such as Xinjiang, primary healthcare institutions generally suffer from insufficient digital pathology equipment, inadequate gene testing capabilities, and high rates of missing modal data. This makes it difficult for existing technical solutions to be implemented in primary hospitals, failing to address the real pain points of low early detection rates of OSCC at the grassroots level and significant regional differences in treatment levels, and hindering the penetration and popularization of OSCC early screening technology.

[0008] In summary, current technologies have not yet formed a set of intelligent detection solutions that fit the clinical diagnosis and treatment logic of OSCC, can effectively solve the problem of multimodal heterogeneous data fusion, and are suitable for primary healthcare scenarios. There is an urgent need to develop an OSCC multimodal fusion intelligent detection technology with clinical interpretability, high robustness, and strong applicability, so as to provide standardized and precise auxiliary diagnostic support for clinicians in medical institutions at different levels, and promote the improvement and homogenization of my country's OSCC early diagnosis and treatment system. Summary of the Invention

[0009] In view of this, the purpose of this invention is to provide a hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma. Based on a Transformer-based adaptive perception network, this system employs a hierarchical cross-modal fusion strategy and designs a multi-task collaborative learning framework to simultaneously optimize two highly correlated core clinical tasks: tumor invasiveness score prediction and cervical lymph node metastasis probability prediction. This invention effectively solves the technical problems of insufficient multimodal data fusion, difficulty in capturing deep cross-modal interaction features, and single prediction targets that fail to provide multi-dimensional decision-making support for neck lymph node dissection in existing technologies. It achieves a systematic improvement in the accuracy of assessing the biological behavior of oral squamous cell carcinoma, significantly enhances the ability to identify occult lymph node metastasis, and provides clinicians with interpretable and visualized decision support. Furthermore, it is adaptable to scenarios lacking modalities in primary healthcare, possessing strong clinical application value.

[0010] This invention is achieved through the following technical solution: A hierarchical, cross-modal, multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma includes: The data acquisition and preprocessing module is used to acquire multimodal data of patients with oral squamous cell carcinoma. The multimodal data includes at least imaging data, pathological data, clinical data and genetic data. The module performs case-level alignment and standardization preprocessing on each modality of data and divides the preprocessed dataset into training set, validation set and test set. The modal feature extraction module is used to extract image depth features and pathological depth features from the image data and pathological data as image modalities, respectively; and to extract clinical coding features and gene coding features from the clinical data and gene data as structured modalities, respectively. A hierarchical cross-modal fusion module is used to perform hierarchical fusion of the image-based modalities and the structured modalities. The hierarchical fusion includes at least three levels: intra-modal fusion of images, intra-modal fusion of structured modalities, and cross-modal fusion. Intra-modal fusion of images fuses the image depth features and the pathological depth features into an image-based fusion feature. Intra-modal fusion of structured modalities fuses the clinical coding features and the gene coding features into a structured fusion feature. Cross-modal fusion of images fuses the image-based fusion features and the structured fusion features into a multimodal fusion feature. In the cross-modal fusion, the hierarchical cross-modal fusion module assigns fusion weights to different modalities according to the clinical priority of each modality. A multi-task prediction module is used to simultaneously execute at least two clinical prediction tasks related to oral squamous cell carcinoma based on the multimodal fusion features. An interpretable output module is used to provide visual explanations of the output results of the multi-task prediction module.

[0011] Further specifying, the clinical prediction tasks include oral squamous cell carcinoma detection and classification tasks, tumor invasiveness scoring prediction tasks, and cervical lymph node metastasis risk prediction tasks.

[0012] Furthermore, the hierarchical cross-modal fusion module also includes a modality missing processing unit. This unit sets the image data and clinical data as core modalities and the pathological data and gene data as enhanced modalities. When an enhanced modality is missing, the unit marks the missing modality, completes it using pre-trained missing modality features, and redistributes the fusion weights of the remaining effective modalities.

[0013] Furthermore, the modality missing processing unit performs random masking processing on the augmented modalities in the training set samples with a predetermined probability during the model training phase, and trains the model's ability to adapt to modality missing by comparative learning.

[0014] Further specifying, in the modal feature extraction module, the specific method for extracting the image depth features is as follows: inputting the preprocessed image region of interest data into a pre-trained convolutional neural network, and outputting the image depth features after processing through convolutional layers, pooling layers, and residual blocks; the specific method for extracting the pathological depth features is as follows: dividing the preprocessed pathological whole-slice image into several patches, inputting each patch into a pre-trained pathological image feature extraction network to extract patch-level features, and outputting the pathological depth features after aggregating the features of all patches through an attention mechanism.

[0015] Further specifying, in the modal feature extraction module, the specific method for extracting the clinical coding features is as follows: inputting the standardized clinical features into a multilayer perceptron for nonlinear mapping, and outputting the clinical coding features; the specific method for extracting the gene coding features is as follows: constructing a topological association network containing gene nodes and signal pathway nodes, mapping the standardized gene features to the topological association network, extracting the interaction relationships between genes through a graph convolutional network, and outputting the gene coding features.

[0016] Furthermore, the image-class intramodal fusion is achieved through the self-attention mechanism of the Transformer network, the structured class intramodal fusion is achieved through a fully connected network, and the cross-class cross-fusion is achieved through a cross-modal Transformer network.

[0017] Further defined, the multi-task prediction module includes a shared network layer and multiple task-specific output layers; the shared network layer consists of multiple fully connected layers with residual connections, used to transform the multimodal fusion features into shared hidden states; each task-specific output layer corresponds to a clinical prediction task, and each task-specific output layer makes predictions starting from the shared hidden states; the multi-task prediction module is trained using a joint loss function, which is a weighted sum of the loss functions of each clinical prediction task.

[0018] Further defining the interpretability output module, it includes a lesion region visualization unit, a feature importance ranking unit, and a biological association interpretation unit. The lesion region visualization unit uses a gradient-weighted class activation mapping method to annotate the core regions of interest to the model on the original images and pathological images. The feature importance ranking unit calculates the contribution of each modality and each feature to the prediction results based on SHAP values ​​and generates ranking information. The biological association interpretation unit outputs biological interpretation information based on the association between gene pathways and image phenotypes.

[0019] Further specifying, the imaging data includes CT images and / or MRI images; the pathological data includes H&E-stained digital pathological whole-slice images; the clinical data includes structured clinical records and unstructured clinical texts; and the genetic data includes tumor tissue sequencing data.

[0020] The beneficial effects of this invention are as follows: By adopting a three-tiered fusion architecture of "image-based intra-modal fusion, structured intra-modal fusion, and cross-category fusion", the problem of mismatch in feature space of heterogeneous data is fundamentally solved, the complementary features between multimodal data are fully explored, and the sufficiency and effectiveness of multimodal fusion are improved.

[0021] By embedding a weight allocation mechanism guided by clinical priorities, the model fusion logic conforms to clinical diagnosis and treatment guidelines. Through three levels of interpretability output—visualization of lesion areas, ranking of feature importance, and explanation of biological associations—clinicians can clearly trace the basis of the model's judgments, thereby enhancing the model's interpretability and clinical reliability.

[0022] By simultaneously performing three major tasks—OSCC detection, invasiveness scoring, and metastasis risk prediction—the multi-task prediction module provides clinicians with comprehensive auxiliary diagnostic information, overcoming the limitations of single-task diagnosis and achieving multi-dimensional clinical decision support.

[0023] By employing a core enhanced modality classification mechanism, feature completion and weight redistribution, and mask comparison training strategy, the system can still operate stably when pathological or genetic data is missing, adapting to the actual conditions of primary healthcare institutions and improving the system's robustness to modality loss and its adaptability to primary healthcare settings.

[0024] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0025] Figure 1 A schematic diagram of the overall architecture of a stratified, cross-modal, multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma.

[0026] In the picture: 100. Data Acquisition and Preprocessing Module; 200. Modal Feature Extraction Module; 300. Hierarchical Cross-Modal Fusion Module; 400. Multi-Task Prediction Module; 500. Interpretable Output Module. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0029] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0030] In the above description of the present invention, it should be noted that the terms "one side," "the other side," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is conventionally placed during use. These terms are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] Furthermore, terms such as "identical" do not imply that components must be absolutely identical; minor differences are permissible. The term "perpendicular" simply means that the positional relationship between components is more perpendicular than "parallel," not that the structure must be perfectly perpendicular; a slight tilt is acceptable.

[0032] This invention provides a hierarchical, cross-modal, multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma, comprising: The data acquisition and preprocessing module is used to acquire multimodal data of patients with oral squamous cell carcinoma. The multimodal data includes at least imaging data, pathological data, clinical data and genetic data. The module performs case-level alignment and standardization preprocessing on each modality of data and divides the preprocessed dataset into training set, validation set and test set. The modal feature extraction module is used to extract image depth features and pathological depth features from the image data and pathological data as image modalities, respectively; and to extract clinical coding features and gene coding features from the clinical data and gene data as structured modalities, respectively. A hierarchical cross-modal fusion module is used to perform hierarchical fusion of the image-based modalities and the structured modalities. The hierarchical fusion includes at least three levels: intra-modal fusion of images, intra-modal fusion of structured modalities, and cross-modal fusion. Intra-modal fusion of images fuses the image depth features and the pathological depth features into an image-based fusion feature. Intra-modal fusion of structured modalities fuses the clinical coding features and the gene coding features into a structured fusion feature. Cross-modal fusion of images fuses the image-based fusion features and the structured fusion features into a multimodal fusion feature. In the cross-modal fusion, the hierarchical cross-modal fusion module assigns fusion weights to different modalities according to the clinical priority of each modality. A multi-task prediction module is used to simultaneously execute at least two clinical prediction tasks related to oral squamous cell carcinoma based on the multimodal fusion features. An interpretable output module is used to provide visual explanations of the output results of the multi-task prediction module.

[0033] The clinical prediction tasks include oral squamous cell carcinoma detection and classification tasks, tumor invasiveness scoring prediction tasks, and cervical lymph node metastasis risk prediction tasks.

[0034] The hierarchical cross-modal fusion module also includes a modality missing processing unit. This unit sets the image data and clinical data as core modalities and the pathological data and genetic data as enhanced modalities. When an enhanced modality is missing, the modality missing processing unit marks the missing modality, completes it using pre-trained missing modality features, and redistributes the fusion weights of the remaining effective modalities.

[0035] In this process, the modality missing processing unit performs random masking on the augmented modalities in the training set samples with a predetermined probability during the model training phase, and trains the model's ability to adapt to modality missing by comparative learning.

[0036] Specifically, in the modal feature extraction module, the method for extracting the image depth features is as follows: the preprocessed image region of interest data is input into a pre-trained convolutional neural network, and the image depth features are output after processing through convolutional layers, pooling layers, and residual blocks; the method for extracting the pathological depth features is as follows: the preprocessed pathological whole-slice image is divided into several patches, each patch is input into a pre-trained pathological image feature extraction network to extract patch-level features, and the features of all patches are aggregated through an attention mechanism to output the pathological depth features.

[0037] Specifically, in the modal feature extraction module, the clinical coding features are extracted by inputting the standardized clinical features into a multilayer perceptron for nonlinear mapping and outputting the clinical coding features. The gene coding features are extracted by constructing a topological association network containing gene nodes and signal pathway nodes, mapping the standardized gene features to the topological association network, extracting the interaction relationships between genes through a graph convolutional network, and outputting the gene coding features.

[0038] Specifically, the image-class intramodal fusion is achieved through the self-attention mechanism of the Transformer network, the structured class intramodal fusion is achieved through a fully connected network, and the cross-class cross-fusion is achieved through a cross-modal Transformer network.

[0039] The multi-task prediction module includes a shared network layer and multiple task-specific output layers. The shared network layer consists of multiple fully connected layers with residual connections, used to transform the multimodal fusion features into shared hidden states. Each task-specific output layer corresponds to a clinical prediction task, and each task-specific output layer makes predictions based on the shared hidden states. The multi-task prediction module is trained using a joint loss function, which is a weighted sum of the loss functions of each clinical prediction task.

[0040] The interpretability output module includes a lesion region visualization unit, a feature importance ranking unit, and a biological association interpretation unit. The lesion region visualization unit uses a gradient-weighted class activation mapping method to annotate the core regions of interest to the model on the original images and pathological images. The feature importance ranking unit calculates the contribution of each modality and each feature to the prediction results based on SHAP values ​​and generates ranking information. The biological association interpretation unit outputs biological interpretation information based on the association between gene pathways and image phenotypes.

[0041] The imaging data includes CT images and / or MRI images; the pathological data includes H&E-stained digital pathological whole slide images; the clinical data includes structured clinical records and unstructured clinical texts; and the genetic data includes tumor tissue sequencing data.

[0042] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. These preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0043] Example 1: System Overall Architecture like Figure 1 As shown, it includes a data acquisition and preprocessing module 100, a submodal feature extraction module 200, a hierarchical cross-modal fusion module 300, a multi-task prediction module 400, and an interpretability output module 500.

[0044] The data acquisition and preprocessing module 100 is used to acquire multimodal data of OSCC patients. The multimodal data includes at least imaging data, pathological data, clinical data and genetic data. The module performs case-level alignment and standardization preprocessing on each modality of data and divides the preprocessed dataset into training set, validation set and test set.

[0045] The modal feature extraction module 200 is used to extract image depth features and pathological depth features from the image data and pathological data as image modalities, respectively; and to extract clinical coding features and gene coding features from the clinical data and gene data as structured modalities, respectively.

[0046] The hierarchical cross-modal fusion module 300 is used to perform hierarchical fusion of the image-type modality and the structured-type modality, and output multimodal fusion features.

[0047] The multi-task prediction module 400 is used to simultaneously execute at least two OSCC-related clinical prediction tasks based on the multimodal fusion features.

[0048] The interpretability output module 500 is used to provide visual explanation information for the output results of the multi-task prediction module.

[0049] Example 2: Modal Feature Extraction This embodiment details the specific implementation of the modal feature extraction module 200.

[0050] Image Feature Extraction: The preprocessed image ROI data is input into a pre-trained ResNeXt-101 network. The ResNeXt-101 network effectively extracts deep features and avoids the gradient vanishing problem through grouped convolutions and residual connections. The image data is sequentially processed through convolutional layers, pooling layers, and residual blocks to extract the spatial morphological features, texture features, boundary features, and density distribution features of the tumor. Finally, a 256-dimensional image depth feature vector is output through a global average pooling layer.

[0051] Pathological Feature Extraction: The preprocessed pathological WSI image was divided into 256×256 pixel non-overlapping patches at a magnification of 20×. Each patch was input into a pre-trained HistoBERT network to extract patch-level features. This network is optimized for cell morphology and tissue arrangement characteristics in pathological images. A multi-instance learning strategy was adopted, and the features of all patches were aggregated through an attention mechanism to highlight the feature contribution of the core tumor region, ultimately outputting a 256-dimensional pathological depth feature vector.

[0052] Clinical feature encoding: The standardized clinical features are input into a two-layer multilayer perceptron. The first layer has 256 neurons and the second layer has 128 neurons. Both layers use the GELU activation function. A dropout layer with a dropout rate of 0.2 is set between the two layers. The output is a 128-dimensional clinical encoded feature vector.

[0053] Gene Feature Encoding: A "gene pathway-expression" topological association network is constructed, combining standardized gene features with prior knowledge of key OSCC signaling pathways (such as the PI3K-AKT pathway and the NOTCH pathway). Interactions between genes and deep molecular-level features are extracted using graph convolutional layers, outputting a 128-dimensional gene encoding feature vector.

[0054] Example 3: Hierarchical Cross-Modal Fusion This embodiment details the specific implementation of the hierarchical cross-modal fusion module 300.

[0055] Image-modal fusion: The 256-dimensional image depth feature vector and the 256-dimensional pathological depth feature vector are input into the Transformer fusion layer. The self-attention mechanism is used to capture the deep interaction features between the two image modalities. The feature vector is then aggregated through residual connections and layer normalization to form a 512-dimensional image-modal fusion feature vector.

[0056] Structured class-modal fusion: 128-dimensional clinical coding feature vectors and 128-dimensional gene coding feature vectors are input into a fully connected fusion layer. Through splicing, nonlinear mapping and feature normalization, the correlation between clinical features and gene features is explored and aggregated to form a 256-dimensional structured class-fusion feature vector.

[0057] Cross-category fusion: Embedding the clinical prior rules from the CSCO Head and Neck Tumor Diagnosis and Treatment Guidelines—that "pathological diagnosis is the gold standard for OSCC diagnosis, imaging TNM staging is the core basis for treatment decisions, and clinical and genetic features are important supplements to risk stratification"—the baseline weights for image-based fusion features are set at 0.6, and those for structured fusion features at 0.4. Image-based and structured fusion features are input into a cross-modal Transformer fusion layer. The baseline weights are dynamically adjusted within ±0.1 based on the variance contribution of each modality feature. These dynamic weights are embedded in the cross-modal attention calculation process, performing weighted mapping on different modal features and calculating cross-modal attention scores. The weighted features are then aggregated and nonlinearly mapped using a feedforward neural network and residual connections. After layer normalization, a 768-dimensional multimodal fusion feature is output.

[0058] Modality Missing Adaptive Handling: CT images and clinical structured data are defined as two core mandatory modalities, while pathology and genetic data are optional enhancement modalities. When an optional enhancement modality is missing, the system automatically masks the missing modality and uses the pre-trained mean features of that modality on the training set to complete the feature vector of the missing modality. Based on the modality missing situation, the weights of the remaining valid modalities are automatically redistributed according to the CSCO prior rules, ensuring that the weight of the core mandatory modalities is not less than 80%. For example, if both pathology and genetic data are missing, image-based fusion features retain only image features, and structured data fusion features retain only clinical features, with their weights each adjusted to 0.5, and the sum of their weights being 1. During model training, the optional modalities of the training set samples are randomly masked with a 30% masking probability. A contrastive learning method is used to compare the features of complete modality samples with those of missing modality samples during training, improving the model's adaptability to modality missingness.

[0059] Example 4: Multi-task collaborative learning This embodiment details the specific implementation of the multi-task prediction module 400.

[0060] Task Sharing Layer: The 768-dimensional multimodal fusion features are input into the task sharing network layer. The sharing layer consists of three fully connected layers with residual connections, with 512, 256, and 128 neurons in each layer, respectively. All layers use the GELU activation function. Each fully connected layer is followed by a Dropout layer with a dropout rate of 0.3, and the final output is a 128-dimensional shared hidden state.

[0061] Task-specific output layer: (1) OSCC detection classification head: Two fully connected layers are used, with 64 and 2 neurons respectively. The Softmax activation function is used to output the binary classification result (OSCC positive / negative).

[0062] (2) Tumor invasiveness score regression head: Two fully connected layers are used, with 64 and 1 neurons respectively. The tumor invasiveness score is output by mapping to the [0,5] interval through the Sigmoid function.

[0063] (3) Cervical lymph node metastasis risk prediction head: Two fully connected layers are used, with 64 and 1 neurons respectively. The output metastasis risk probability value is between 0 and 1. A probability value ≥ 0.5 is judged as high metastasis risk.

[0064] Joint loss function The total loss function is: × in, The cross-entropy loss for the OSCC detection classification task is used to measure the difference between the model's classification result and the true label. The calculation formula is as follows: in For the sample size, For the first The true label of each sample (1 = OSCC positive, 0 = OSCC negative). This represents the positive probability predicted by the model. The mean squared error loss for the tumor invasiveness scoring regression task measures the difference between the model-predicted invasiveness score and the true score annotated by pathologists. The calculation formula is as follows: in For the first The true invasiveness score of each sample The invasiveness score predicted by the model; The cross-entropy loss is used in the cervical lymph node metastasis risk prediction task to measure the difference between the model's predicted metastasis probability and the actual metastasis status. The calculation formula is the same as... Consistent, among which For the first The true transition status of each sample (1 = transition, 0 = no transition). The transition probability predicted by the model; The loss balance coefficients for each task are determined using a grid search method, and in this embodiment, they are set as follows: =0.3、 =0.4、 =0.3.

[0065] The model was trained using the AdamW optimizer with an initial learning rate of 1e-4, a weight decay coefficient of 1e-5, a batch size of 16, and 100 training epochs. An early stopping mechanism (patience=10) was used to prevent overfitting; that is, training was automatically terminated when the validation set loss did not decrease for 10 consecutive epochs, and the model with the smallest validation set loss was saved as the optimal model.

[0066] Example 5: Interpretable Output This embodiment details the specific implementation of the interpretability output module 500.

[0067] Visualization of key lesion areas: The Grad-CAM method was used to visualize the feature extraction process of the imaging and pathological models. On the original CT / MRI images and pathological WSI images, the core areas of interest identified by the model were marked, showing the tumor lesion boundaries, highly invasive areas, and suspicious lymph node areas.

[0068] Multimodal feature importance ranking: The contribution of each modality and feature to the final output of the model is calculated based on the SHAP value method, generating a standardized feature importance ranking report. The report clearly marks the top 10 core features with the highest contribution (such as pathological differentiation degree, tumor invasion depth, imaging T stage, TP53 gene mutation status, smoking history, etc.) and quantifies the contribution value of each feature.

[0069] Image-gene association biological interpretation: Based on the "gene pathway-image phenotype" topological association network constructed by the convolutional layer in step 2.2, the association between imaging biomarkers (such as tumor boundary ambiguity and density heterogeneity) strongly associated with the high invasiveness and high metastasis risk of OSCC and corresponding molecular signaling pathways (such as the PI3K-AKT pathway) is explored to establish the association interpretation of "image phenotype-molecular mechanism".

[0070] Example 6: Experimental Verification To verify the effectiveness of this invention, a comparative experiment was conducted on the self-built oral cancer multimodal-cervical lymph node metastasis dataset (OSCC-1K).

[0071] This dataset contains 1200 pathologically confirmed patients with oral squamous cell carcinoma. All cases are accompanied by preoperative CT images, postoperative pathological whole-slice images, and complete clinical data. The experiment employed a five-fold cross-validation method to effectively reduce bias caused by data partitioning. Evaluation metrics included: area under the receiver operating characteristic (AUC), accuracy (ACC), and F1-score for the lymph node metastasis prediction task; and Pearson correlation coefficient (PCC) for the invasiveness score prediction task. A PCC value closer to 1 indicates a stronger correlation between the model's predicted score and the actual score.

[0072] Experimental method setup: This experiment compares the method of the present invention with the following methods: (1) Radiomics + LASSO (traditional method): Based solely on image data, LASSO regression is used for feature selection and prediction; (2) Clinical pathology Nomogram (logistic regression): Based solely on clinical and pathological data, a predictive model is constructed using logistic regression; (3) Simple splicing of three modalities for single task (ResNet+MLP): Clinical + imaging + pathology three modal data are used, and the features are directly spliced ​​and then input into MLP for single task prediction; (4) Tensor Fusion Based Trimodal Network (TFN): Clinical + Imaging + Pathology trimodal data are used, and prediction is performed after multimodal fusion through tensor fusion layer.

[0073] The experimental results are as follows: Results analysis: As can be seen from the experimental results in the table above, the multi-task hierarchical fusion network proposed in this invention performs best among all the comparison methods: in the lymph node metastasis prediction task, it achieves the highest AUC (0.947), accuracy (0.885), and F1-score (0.867), with an AUC improvement of more than 16% compared to traditional methods (radiomics + LASSO, clinicopathology Nomogram), an AUC improvement of 7.4% compared to the single-task trimodal simple splicing model, and an AUC improvement of 4.2% compared to the trimodal network based on tensor fusion. It is significantly better than traditional machine learning methods and single-task simple cross-modal fusion models, indicating that the hierarchical fusion strategy of this invention can effectively mine the complementary features of multimodal data and improve the accuracy of lymph node metastasis prediction, especially significantly enhancing the ability to identify occult lymph node metastasis.

[0074] In the invasiveness score regression task, the Pearson correlation coefficient (PCC) of the method of this invention reached 0.823, which is much higher than other comparative methods (the highest being 0.724 for the TFN model). This indicates that the invasiveness score predicted by the model is highly correlated with the actual invasiveness score labeled by pathology experts. It can achieve accurate quantitative assessment of the invasiveness characteristics of OSCC and provide a reliable basis for the formulation of clinical treatment plans.

[0075] In summary, the experimental results fully demonstrate that the hierarchical cross-modal fusion mechanism and multi-task learning strategy designed in this invention have significant advantages and effectiveness in the comprehensive assessment of the risk of cervical lymph node metastasis and tumor invasiveness in oral squamous cell carcinoma, and have extremely high clinical application value.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A hierarchical, cross-modal, multi-task collaborative intelligent diagnosis and treatment system for oral squamous cell carcinoma, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data of patients with oral squamous cell carcinoma. The multimodal data includes at least imaging data, pathological data, clinical data and genetic data. The module performs case-level alignment and standardization preprocessing on each modality of data and divides the preprocessed dataset into training set, validation set and test set. The modal feature extraction module is used to extract image depth features and pathological depth features from the image data and pathological data as image modalities, respectively; and to extract clinical coding features and gene coding features from the clinical data and gene data as structured modalities, respectively. A hierarchical cross-modal fusion module is used to perform hierarchical fusion of the image-type modality and the structured modality: the hierarchical fusion includes at least three levels: image-type modality intra-fusion, structured modality intra-fusion, and cross-major category cross-fusion. The image-type modality intra-fusion fuses the image depth features and the pathological depth features into an image-type fusion feature. The structured modality intra-fusion fuses the clinical coding features and the gene coding features into a structured fusion feature. The cross-major category cross-fusion fuses the image-type fusion feature and the structured fusion feature into a multimodal fusion feature. The hierarchical cross-modal fusion module assigns fusion weights to different modalities based on their clinical priority during the cross-category fusion process. A multi-task prediction module is used to simultaneously execute at least two clinical prediction tasks related to oral squamous cell carcinoma based on the multimodal fusion features. An interpretable output module is used to provide visual explanations of the output results of the multi-task prediction module.

2. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, The clinical prediction tasks include oral squamous cell carcinoma detection and classification tasks, tumor invasiveness scoring prediction tasks, and cervical lymph node metastasis risk prediction tasks.

3. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, The hierarchical cross-modal fusion module also includes a modality missing processing unit. This unit sets the image data and clinical data as core modalities and the pathological data and genetic data as enhanced modalities. When an enhanced modality is missing, the unit marks the missing modality, completes it using pre-trained missing modality features, and redistributes the fusion weights of the remaining effective modalities.

4. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 3, characterized in that, The modality missing processing unit performs random masking on the augmented modalities in the training set samples with a predetermined probability during the model training phase, and trains the model's ability to adapt to modality missing by comparative learning.

5. The oral squamous cell carcinoma stratified cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, In the modal feature extraction module, the specific method for extracting the image depth features is as follows: the preprocessed image region of interest data is input into a pre-trained convolutional neural network, and the image depth features are output after processing through convolutional layers, pooling layers, and residual blocks; the specific method for extracting the pathological depth features is as follows: the preprocessed pathological whole-slice image is divided into several patches, each patch is input into a pre-trained pathological image feature extraction network to extract patch-level features, and the features of all patches are aggregated through an attention mechanism to output the pathological depth features.

6. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, In the modal feature extraction module, the specific method for extracting the clinical coding features is as follows: the standardized clinical features are input into a multilayer perceptron for nonlinear mapping, and the clinical coding features are output. The specific method for extracting the gene coding features is as follows: a topological association network containing gene nodes and signal pathway nodes is constructed, the standardized gene features are mapped to the topological association network, and the interaction relationships between genes are extracted through a graph convolutional network, and the gene coding features are output.

7. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, The image-class intramodal fusion is achieved through the self-attention mechanism of the Transformer network, the structured class intramodal fusion is achieved through a fully connected network, and the cross-class cross-fusion is achieved through a cross-modal Transformer network.

8. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, The multi-task prediction module includes a shared network layer and multiple task-specific output layers. The shared network layer consists of multiple fully connected layers with residual connections, used to transform the multimodal fusion features into shared hidden states. Each task-specific output layer corresponds to a clinical prediction task, and each task-specific output layer makes predictions starting from the shared hidden states. The multi-task prediction module is trained using a joint loss function, which is a weighted sum of the loss functions of each clinical prediction task.

9. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, The interpretable output module includes a lesion region visualization unit, a feature importance ranking unit, and a biological association interpretation unit; the lesion region visualization unit uses a gradient-weighted class activation mapping method to annotate the core regions of interest to the model on the original image and pathological image; The feature importance ranking unit calculates the contribution of each modality and each feature to the prediction result based on the SHAP value and generates ranking information. The biological association interpretation unit outputs biological interpretation information based on the association between gene pathways and image phenotypes.

10. The oral squamous cell carcinoma hierarchical cross-modal multi-task collaborative intelligent diagnosis and treatment system according to claim 1, characterized in that, The imaging data includes CT images and / or MRI images; the pathological data includes H&E-stained digital pathological whole slide images; the clinical data includes structured clinical records and unstructured clinical texts; and the genetic data includes tumor tissue sequencing data.

Images