A multimodal oral pathology large model system and method for aiding diagnosis and teaching
By constructing a multimodal oral pathology model, the problems of long diagnosis time and poor consistency in oral pathology have been solved, enabling rapid and accurate auxiliary diagnosis and teaching, and improving the efficiency of oral pathology diagnosis and teaching.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2026-05-13
- Publication Date
- 2026-06-23
AI Technical Summary
Current oral pathology diagnosis relies on manual slide reading, which is time-consuming and has poor diagnostic consistency. General pathology AI models lack oral pathology specificity, have difficulty processing ultra-high resolution WSI images, lack multimodal information fusion mechanisms, and have limited functionality, failing to meet the needs of auxiliary diagnosis and teaching.
A large-scale multimodal oral pathology model is constructed, including modules for data acquisition, knowledge graph, multi-scale visual coding, multimodal semantic alignment, and reasoning and decision-making. It supports the fusion analysis of pathological images, clinical text, and imaging data, and is combined with the oral pathology knowledge graph for assisted diagnosis and teaching.
It enables rapid and accurate oral pathology diagnosis and teaching, improves diagnostic efficiency and interpretability, supports multi-scenario applications, and enhances the efficiency of pathology teaching and the training effect for young doctors.
Smart Images
Figure CN122266732A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of medical artificial intelligence and oral pathology, specifically relating to a multimodal oral pathology large model system and method for auxiliary diagnosis and teaching. Background Technology
[0002] Oral pathology is one of the important basic disciplines of oral medicine. Its diagnostic results directly affect the formulation of clinical treatment plans and prognostic assessments for oral squamous cell carcinoma, potential oral malignancies (such as oral leukoplakia and oral lichen planus), odontogenic cysts, odontogenic tumors, and various inflammatory diseases.
[0003] However, the current oral pathology diagnosis has long relied on the manual reading experience of pathologists, which has the following prominent problems: (1) Oral pathology subtypes are complex, and the histological differences between different lesions are subtle. The consistency of diagnosis is significantly affected by the reading experience and is highly subjective; (2) Whole Slide Image (WSI) is huge in size, with the number of pixels in a single slide reaching 10 9 -10 10 (3) Primary healthcare institutions generally lack professional oral pathologists, and the distribution of high-quality medical resources is seriously uneven, resulting in poor accessibility of pathological diagnosis. (4) Existing general pathological artificial intelligence systems mainly focus on major pathological fields such as lung cancer, breast cancer, and gastrointestinal tumors. Their training data and model architecture lack adaptation to specific tissue structures (such as odontogenic epithelial remnants and special keratinization patterns of oral mucosa) and lesion patterns, leading to a significant decrease in recognition accuracy in oral pathology scenarios. (5) Existing artificial intelligence systems mostly support only single image classification tasks, making it difficult to combine pathological images, clinical history, imaging data, and natural language information to achieve comprehensive analysis. They cannot simulate the comprehensive decision-making logic of "morphology + clinical information" in clinical pathological diagnosis. (6) In the traditional teaching process, young doctors and medical students' understanding of complex oral pathological images is highly dependent on expert guidance. There is a lack of interactive, interpretable, and traceable intelligent teaching platforms, resulting in low reuse rate of teaching resources.
[0004] In recent years, the development of Large Language Model (LLM) and multimodal artificial intelligence technology has provided a new technical path for medical artificial intelligence. However, existing general multimodal models still have the following shortcomings in the field of oral pathology: (1) lack of oral pathology professional knowledge map support, making it difficult to embed the hierarchical logic and expert experience rules of oral pathology diagnosis; (2) lack of multi-scale feature extraction capability for ultra-high resolution pathological slices, making it impossible to effectively balance the capture of global context and local subtle morphology; (3) lack of deep semantic understanding of oral disease-specific pathological morphology (such as dyskeratosis, intercellular bridges, basal cell vacuolar degeneration, etc.); (4) difficulty in achieving accurate semantic alignment between pathological images and clinical text, resulting in a semantic gap in image-text interaction; (5) difficulty in taking into account multiple application scenarios such as auxiliary diagnosis, teaching Q&A, and scientific research analysis, and the model output lacks interpretability and clinical operability.
[0005] Therefore, there is an urgent need to develop a multimodal oral pathology model that can integrate pathological images, clinical texts, and knowledge in the field of oral pathology, in order to achieve intelligent assistance in oral pathology diagnosis, teaching support, and sharing of medical resources. Summary of the Invention
[0006] In view of the above-mentioned shortcomings, the purpose of this invention is to overcome the deficiencies of the prior art and provide a multimodal large-scale oral pathology model system and method to solve the following technical problems:
[0007] (1) Existing general pathology AI models lack oral pathology specificity and are difficult to accurately identify oral pathology subtypes and subtle histological differences;
[0008] (2) Existing models are difficult to effectively process ultra-high resolution WSI images and cannot extract global slice-level features and local cell-level features at the same time;
[0009] (3) Existing systems lack cross-modal precise alignment and fusion mechanisms for pathological images and clinical texts and imaging data;
[0010] (4) The existing system lacks knowledge constraints in the field of oral pathology, and the interpretability of the diagnostic logic is poor;
[0011] (5) The existing system has limited functionality and cannot simultaneously meet the needs of multiple scenarios such as assisted diagnosis, structured report generation and intelligent teaching.
[0012] To achieve the above objectives, the present invention employs the following technical means:
[0013] This invention first provides a multimodal oral pathology large model system, comprising:
[0014] The data acquisition module is configured to acquire fully digital slide images of oral pathology, clinical history text data, and medical imaging data; wherein, the fully digital slide images of oral pathology include pathology slide images at different magnifications, the clinical history text data includes basic patient information, lesion location, past medical history, and pathological description information, and the medical imaging data includes at least one of oral CT, MRI, CBCT, and intraoral photographs; the data acquisition module is further configured to perform timestamp matching, format standardization, and anonymization processing on data from different sources.
[0015] The oral pathology knowledge graph module is configured to store and manage oral pathology entity relationships and diagnostic rules; wherein, the oral pathology entity relationships include at least one of disease type, histological features, immunohistochemical markers, pathological classification and differential diagnosis relationships; the diagnostic rules include auxiliary diagnostic rules and teaching reasoning rules based on combinations of pathological features; the knowledge graph module is further configured to support dynamic updates and historical case association retrieval.
[0016] A multi-scale visual coding module is configured to extract multi-resolution features from the fully digital slice image to obtain pathological morphological features; wherein, the pathological morphological features include at least one of cellular atypia features, keratinization features, infiltration boundary features, nuclear-cytoplasmic ratio features, and tissue structure features; the multi-scale visual coding module further includes a hybrid structure of convolutional neural network and visual Transformer, used to simultaneously extract local microscopic features and overall tissue structure features of the fully digital slice image.
[0017] The multimodal semantic alignment module is configured to perform cross-modal semantic mapping and fusion of the pathological morphological features, the clinical history text data, and the medical image data to generate a multimodal fusion representation; wherein, the cross-modal semantic mapping includes alignment learning between image feature vectors and text semantic vectors; the multimodal semantic alignment module is further configured to calculate the importance weights of different modal information based on an attention mechanism.
[0018] The reasoning and decision-making module is configured to perform at least one of the following tasks based on the multimodal fusion representation and the diagnostic rules of the oral pathology knowledge graph module: oral pathology auxiliary diagnosis, structured report generation, and teaching interaction. The oral pathology auxiliary diagnosis includes at least one of benign / malignant lesion classification, pathological subtyping identification, and lesion region localization. The structured report generation includes automatic generation of pathological diagnostic terminology and extraction of diagnostic evidence. The teaching interaction includes differential diagnosis explanation, case reasoning, and knowledge-based question answering.
[0019] The user interaction module is configured to receive natural language query commands and output diagnostic suggestions, teaching explanations, or pathology reports; wherein, the natural language query commands include at least one of pathology diagnosis queries, differential diagnosis queries, pathology knowledge Q&A, and case teaching commands; the user interaction module is further configured to output lesion heatmaps, key pathological area annotation results, and corresponding explanatory information in a combination of text and images.
[0020] This invention also provides a multimodal oral pathology-assisted diagnosis and teaching method, comprising the following steps:
[0021] S1. Acquire whole-slide images of oral pathology, clinical history text data, and medical imaging data; wherein, the whole-slide images of oral pathology are WSI (Whole Slide Image) images digitized by a pathology scanning device, with an image resolution of not less than 20× scan magnification; the clinical history text data includes at least one of the following: chief complaint, present illness, pathology examination information, description of lesion site, and previous medical records; the medical imaging data includes at least one of the following: oral CT, CBCT, MRI, and intraoral photographs; and perform data standardization, missing value filtering, coordinate alignment, and anonymization processing on the data.
[0022] S2. Construct an oral pathology knowledge graph; wherein the oral pathology knowledge graph includes at least a disease entity layer, a histological feature layer, a molecular marker layer, and a diagnostic rule layer; further, the disease entity layer includes disease categories such as oral squamous cell carcinoma, oral leukoplakia, ameloblastoma, odontogenic cysts, and oral mucosal diseases; the histological feature layer includes pathological features such as keratin pearls, nuclear atypia, invasive borders, epithelial dysplasia, and disordered cell arrangement; the diagnostic rule layer is used to establish the mapping relationship between combinations of pathological features and disease categories; the knowledge graph further includes differential diagnostic associations and teaching reasoning paths.
[0023] S3. Multi-resolution feature extraction is performed on the fully digital slice image to obtain pathological morphological features; wherein, the multi-resolution feature extraction includes low-magnification tissue structure feature extraction and high-magnification cell morphology feature extraction; the pathological morphological features include at least cell density features, nuclear-cytoplasmic ratio features, keratinization features, infiltration pattern features, and tissue structure disorder features; furthermore, a combination network of convolutional neural network and visual Transformer is used to encode features of fully digital slice image regions at different scales, and key lesion regions are screened through an attention mechanism.
[0024] S4. Perform cross-modal semantic mapping and fusion on the pathological morphological features, the clinical history text data, and the medical image data to generate a multimodal fusion representation; wherein, the cross-modal semantic mapping includes a unified embedding representation of pathological visual feature vectors, text semantic feature vectors, and medical image feature vectors; the fusion process adopts a cross-modal attention mechanism to establish the association between pathological morphological features and clinical semantic information; the multimodal fusion representation is used to characterize the comprehensive biological characteristics and clinical risk information of diseased tissues.
[0025] S5. Based on the multimodal fusion representation and the diagnostic rules of the oral pathology knowledge graph, perform at least one task among oral pathology auxiliary diagnosis, structured report generation, and teaching interaction; wherein, the auxiliary diagnosis includes lesion benign and malignant classification, pathological subtype identification, lesion area localization, and cancer risk prediction; the structured report generation includes standardized generation of pathological diagnostic terminology, automatic completion of pathological descriptions, and output of diagnostic basis associations; the teaching interaction includes pathological knowledge Q&A, differential diagnosis reasoning, typical case association recommendation, and explanatory annotation of key pathological areas.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1) This invention is the first to construct a multimodal, dedicated large-scale model for the field of oral pathology. It is optimized to address the complex tissue structures, diverse disease subtypes, and reliance on expert experience in oral pathology diagnosis. It can support auxiliary analysis of over thirty common oral diseases, including oral squamous cell carcinoma, leukoplakia, and odontogenic cysts. Compared to traditional single-task artificial intelligence models, this system can complete preliminary analysis of pathological slides and lesion localization within seconds, significantly improving diagnostic efficiency.
[0028] 2) This invention possesses multimodal image understanding and professional natural language processing capabilities, enabling the combined analysis of pathological slides, clinical photographs, and medical history information. It also supports professional pathology Q&A and interactive teaching. For example, after a user inputs "What are the main differentiating points between this lesion and well-differentiated squamous cell carcinoma?", the system can automatically generate explanatory analysis results based on pathological features. Testing has shown that the system's accuracy rate in answering typical teaching case questions can reach over 90%.
[0029] 3) This invention possesses intelligent interactive and interpretable analysis capabilities, automatically marking key pathological areas such as keratinized pearls, infiltrative boundaries, and nuclear atypia, and generating visual heatmaps and diagnostic explanations, thereby improving the effectiveness of pathology teaching and training for young physicians. Compared to traditional static digital pathology systems, this invention can reduce case preparation time by approximately 50% and improve teaching interactivity and learning efficiency. Attached Figure Description
[0030] Figure 1This is a schematic diagram of the overall architecture of the multimodal oral pathology large model system provided in an embodiment of the present invention;
[0031] Figure 2 This is a schematic diagram of the intelligent teaching interactive interface provided in an embodiment of the present invention. Detailed Implementation
[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0033] Example 1
[0034] This embodiment provides a multimodal oral pathology large model system for auxiliary diagnosis and teaching.
[0035] like Figure 1 As shown, the system includes a pathological image acquisition module, a multimodal data processing module, a pathological feature extraction module, a multimodal large language model module, a knowledge graph module, and an intelligent interactive teaching module.
[0036] First, oral pathology slide images are acquired using a fully digital pathology scanner, and the patient's clinical data, including age, gender, lesion location, medical history description, and clinical photographs, are imported simultaneously. After image preprocessing, the pathology slide images are divided into multiple image blocks and input into a pathological feature extraction network for analysis.
[0037] Furthermore, the system employs an attention-based multi-instance learning algorithm to automatically identify key lesion areas in pathological slides and generate lesion heatmaps. Subsequently, a multimodal large language model combines pathological image features, clinical text information, and oral pathology knowledge graphs to comprehensively analyze lesion types.
[0038] In this embodiment, taking oral squamous cell carcinoma as an example, the system can automatically identify keratinized beads, cellular atypia, and invasive growth areas in pathological sections, and output the auxiliary diagnostic result "considering well-differentiated oral squamous cell carcinoma." Simultaneously, the system generates a corresponding lesion area annotation map, stratifies the lesion into malignant, suspicious, and non-malignant lesions based on risk, and further provides reference opinions for the four histological subtypes of squamous cell carcinoma. Testing showed that the model maintains high sensitivity while exhibiting a low false positive rate, with AUC greater than 0.90 for each subtype task.
[0039] Furthermore, in the teaching mode, users can input questions such as "What are the main differentiating points between this case and oral leukoplakia?" The system will automatically call up the pathological knowledge graph and historical case database to generate an explanatory answer that combines text and images, including differences in pathological structure, typical histological features, and relevant differential diagnostic criteria.
[0040] Tests have shown that the system in this embodiment can complete the preliminary analysis of a single fully digital pathology slide within 10 seconds, and the accuracy rate of answers in typical oral pathology teaching Q&A tasks reaches over 90%, which can effectively improve the efficiency of pathology teaching and auxiliary diagnostic capabilities.
[0041] This embodiment is only one of the preferred embodiments of the present invention and is not intended to limit the scope of protection of the present invention.
[0042] Furthermore, to verify the beneficial effects of this invention, this embodiment collected approximately 120,000 oral pathology WSI images from three tertiary hospitals, covering 31 lesion types including oral squamous cell carcinoma, ameloblastoma, and oral leukoplakia. After being annotated by senior oral pathologists, the images were divided into training, validation, and test sets in a 7:2:1 ratio.
[0043] Experimental results show that:
[0044] (1) In the auxiliary diagnostic task, the overall diagnostic accuracy of the system of the present invention on the independent test set reached 93.8%, which is higher than the 86.4% of the general pathology AI model and the 88.7% of the single-modality image classification model; the AUC-ROC reached 0.962, the sensitivity reached 95.1%, and the specificity reached 92.6%. Among them, in key disease categories such as oral squamous cell carcinoma, oral leukoplakia, and ameloblastoma, the AUC of each category was greater than 0.90, indicating that the system has a good auxiliary identification ability for oral pathology diseases. This data range is comparable to the classification accuracy of about 94% reported by multimodal WSI pathology models in recent years. Related studies have also shown that the fusion of WSI with textual information such as pathology reports can improve the tumor classification and interpretable localization ability.
[0045] (2) In the multimodal fusion ablation experiment, the diagnostic accuracy of the complete model was 93.8%; after removing the clinical history text modality, the accuracy decreased to 90.6%, a reduction of 3.2 percentage points; after removing clinical photographs and imaging data, the accuracy decreased to 91.4%, a reduction of 2.4 percentage points; when only pathological images were retained as input, the accuracy was 88.7%. These results indicate that pathological images, clinical text, and other clinical multimodal information have a synergistic effect. Recent studies on multimodal pathological basic models have also shown that training with image alignment to pathology reports or text descriptions can improve WSI-level characterization, report generation, classification, and retrieval capabilities.
[0046] (3) In the intelligent teaching task, 120 typical teaching cases were selected to construct a test set, and 20 oral pathology residents conducted a trial evaluation. The results showed that the differential diagnosis interpretations generated by the system were 91.5% consistent with the expert consensus, the completeness of the interpretation of key pathological features reached 89.7%, and the teaching satisfaction score was 4.6 out of 5. Compared with traditional static teaching slides, the system can automatically highlight key areas such as keratin pearls, cellular atypia, epithelial dysplasia, and infiltration boundaries, and generate explanatory content combining text and images, which helps improve the efficiency of young physicians in understanding complex oral pathology cases.
[0047] (4) In the report generation task, the structured pathology reports automatically generated by the system, after expert review, achieved a format compliance rate of 96.8% and a key information completeness rate of 95.6%. The generation time for a single report was reduced from 9.5 minutes for manual writing to less than 40 seconds. These results suggest that this invention can not only be used to assist in diagnosis, but also for standardized report generation, teaching demonstrations, and remote pathology collaboration at the grassroots level. Existing multimodal WSI basic model research also lists pathology report generation as an important application direction, supporting the feasibility of such technology in pathology workflows.
[0048] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
Claims
1. A multimodal oral pathology large model system, comprising: The data acquisition module is configured to acquire fully digital slice images of oral pathology, clinical history text data, and medical imaging data; The oral pathology knowledge graph module is configured to store and manage oral pathology entity relationships and diagnostic rules; A multi-scale visual coding module is configured to extract multi-resolution features from the all-digital slice image to obtain pathological morphological features; The multimodal semantic alignment module is configured to perform cross-modal semantic mapping and fusion of the pathological morphological features, the clinical history text data, and the medical image data to generate a multimodal fusion representation; The reasoning and decision-making module is configured to perform at least one task among oral pathology auxiliary diagnosis, structured report generation, and teaching interaction based on the diagnostic rules of the multimodal fusion representation and the oral pathology knowledge graph module. The user interaction module is configured to receive natural language query commands and output diagnostic suggestions, teaching materials, or pathology reports.
2. The multimodal oral pathology large model system according to claim 1, wherein: The data acquisition module is configured to acquire fully digital slide images of oral pathology, clinical history text data, and medical imaging data; among which, The fully digital slide images include pathological slide images at different magnifications; The clinical medical history text data includes basic patient information, lesion location, past medical history, and pathological description information; the medical imaging data includes at least one of oral CT, MRI, CBCT, and intraoral photographs. The data acquisition module is further configured to perform timestamp matching, format standardization, and anonymization on data from different sources.
3. The multimodal oral pathology large model system according to claim 1, wherein: The oral pathological entity relationships include at least one of the following: disease type, histological features, immunohistochemical markers, pathological classification, and differential diagnosis. The diagnostic rules include auxiliary diagnostic rules based on combinations of pathological features and teaching reasoning rules; The oral pathology knowledge graph module is further configured to support dynamic updates and historical case association retrieval.
4. The multimodal oral pathology large model system according to claim 1, wherein: The pathological morphological features include at least one of the following: cellular atypia, keratinization, infiltration boundary, nuclear-cytoplasmic ratio, and tissue structure. The multi-scale visual coding module further includes a hybrid structure of convolutional neural network and visual Transformer, used to simultaneously extract local microscopic features and overall tissue structure features of the fully digital slice image.
5. The multimodal oral pathology large model system according to claim 1, wherein: The cross-modal semantic mapping includes alignment learning between image feature vectors and text semantic vectors; The multimodal semantic alignment module is further configured to calculate the importance weights of different modal information based on an attention mechanism.
6. The multimodal oral pathology large model system according to claim 1, wherein: The oral pathology auxiliary diagnosis includes at least one of the following: classification of benign and malignant lesions, identification of pathological subtypes, and localization of lesion areas. The structured report generation includes automatic generation of pathological diagnostic terminology and extraction of diagnostic evidence; The interactive teaching methods include differential diagnosis explanation, case reasoning, and knowledge quizzes.
7. The multimodal oral pathology large model system according to claim 1, wherein: The natural language query commands include at least one of the following: pathological diagnosis query, differential diagnosis query, pathological knowledge Q&A, and case teaching commands. The user interaction module is further configured to output lesion heat maps, key pathological area annotation results, and corresponding explanatory information in a combination of text and images.
8. A multimodal oral pathology-assisted diagnosis and teaching method, comprising the following steps: S1. Acquire fully digital slice images of oral pathology, clinical history text data, and medical imaging data, and perform data standardization, missing value filtering, coordinate alignment, and anonymization on the above data; S2. Construct an oral pathology knowledge graph, which includes at least: a disease entity layer, a histological feature layer, a molecular marker layer, and a diagnostic rule layer; the graph further includes differential diagnostic relationships and teaching reasoning paths. S3. Perform multi-resolution feature extraction on the fully digital slice image to obtain pathological morphological features; S4. Perform cross-modal semantic mapping and fusion on the pathological morphological features, the clinical history text data and the medical image data to generate a multimodal fusion representation; S5. Based on the multimodal fusion representation and the diagnostic rules of the oral pathology knowledge graph, perform at least one task in oral pathology auxiliary diagnosis, structured report generation, and teaching interaction.
9. The multimodal oral pathology auxiliary diagnosis and teaching method according to claim 8, wherein: Step S3, the multi-resolution feature extraction, includes low-magnification tissue structure feature extraction and high-magnification cell morphology feature extraction; the pathological morphological features include at least cell density features, nuclear-cytoplasmic ratio features, keratinization features, infiltration pattern features, and tissue structure disorder features; Furthermore, a combination of convolutional neural networks and visual Transformers is used to encode features of fully digital slice image regions at different scales, and key lesion regions are screened through an attention mechanism.
10. The multimodal oral pathology auxiliary diagnosis and teaching method according to claim 8, wherein: The cross-modal semantic mapping described in step S4 includes a unified embedding representation of pathological visual feature vectors, text semantic feature vectors, and medical image feature vectors; The fusion process employs a cross-modal attention mechanism to establish a correlation between pathological morphological features and clinical semantic information; the multimodal fusion representation is used to characterize the comprehensive biological characteristics and clinical risk information of diseased tissues.