Cervical cell interpretable classification method guided by pathological semantics

Abstract

The invention discloses a cervical cell interpretable classification method guided by pathological semantics, and belongs to the technical field of medical image processing. The method comprises the following steps: firstly, constructing a pathological semantic knowledge graph containing cell structures, morphological characteristics and diagnosis concepts, and obtaining semantic vector representation of the pathological semantic knowledge graph; then extracting cell individual characteristics of the cervical cell image and cell population relation characteristics based on a graph attention network; thirdly, fine-grained alignment and fusion are carried out on the visual features and the pathological semantic vectors through a cross-modal attention alignment module, and semantic-enhanced fusion feature vectors are generated; and finally, performing classification prediction based on the features, and obtaining key pathological semantic concepts according to alignment weight traceability to form interpretable output. According to the method, pathological knowledge is explicitly embedded into the classification model, so that the accuracy of cervical cell classification and the interpretability of model decision are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer-aided diagnostic technology, and in particular to an interpretable classification method for cervical cells guided by pathological semantics. Background Technology

[0002] Cervical cancer is a common malignant tumor that seriously threatens women's health, but early screening can effectively reduce its incidence and mortality. Cervical cytology is one of the main methods of cervical cancer screening, which involves staining exfoliated cervical cells and observing them under a microscope to assess morphological abnormalities. However, traditional cervical cell classification diagnosis faces many challenges: First, manual interpretation of slides is inefficient and heavily influenced by the doctor's experience, resulting in poor diagnostic consistency among different observers; second, the morphology of cells in cervical cell images is complex and variable, and the morphological differences between diseased cells and normal cells, as well as between cells of different grades of disease, can be very subtle, increasing the difficulty of accurate classification.

[0003] In recent years, deep learning-based computer-aided diagnostic systems have made significant progress in cervical cell image analysis. Existing technologies mainly include the following types: First, end-to-end classification methods based on convolutional neural networks, which directly learn features from cell images and perform classification; second, classification methods that combine cell context information, such as the proposed cervical cell classification method that integrates personalized cell relationships and background information. This method constructs similar and different cell relationship graphs, utilizes graph attention networks to enhance the modeling of cell relationships, and simultaneously uses a multi-head attention module to capture smear background information; third, multimodal fusion methods, such as the proposed cervical image classification method that integrates multimodal prior pathological depth features. This method integrates features from the original cervical image, cervical iodine image, and cervical acetic acid whitening image through three-stage training; and fourth, methods that combine localization and classification, such as the proposed method for localizing and identifying cervical cells on a whole slide. This method first uses a target detection model to locate suspected lesion cells, and then uses a fine-grained classification model to classify the lesion type.

[0004] Despite some progress in existing technologies, there are still problems such as insufficient integration of pathological knowledge, poor model interpretability, and inadequate modeling of cell relationships.

[0005] Furthermore, existing technologies often overlook the rich semantic information contained in pathology report text when processing cervical cell images. As shown in [examples of images], pathology reports contain structured information such as diagnostic conclusions, morphological descriptions, and differential diagnoses. This information can serve as a knowledge source to guide the extraction and understanding of image features. Therefore, how to combine pathological semantic knowledge with visual features to construct an accurate and interpretable cervical cell classification method has become an urgent technical problem to be solved. Summary of the Invention

[0006] To achieve the above objectives, this invention provides a method for interpretable classification of cervical cells guided by pathological semantics, comprising the following steps:

[0007] Step 1: Construct a pathological semantic knowledge graph containing pathological entities of cervical cells and the relationships between entities, and obtain the semantic vector representation of the pathological entities in the pathological semantic knowledge graph;

[0008] Step 2: Extract visual features from the cervical cell images to be classified. The visual features include individual cell features obtained based on image segmentation networks and cell group relationship features modeled based on graph attention networks.

[0009] Step 3: Through the cross-modal attention alignment module, the visual features and the semantic vector representation are finely aligned and fused to generate a semantically enhanced fused feature vector;

[0010] Step 4: Perform classification prediction based on the semantically enhanced fusion feature vector, and based on the alignment weights generated by the cross-modal attention alignment module, trace back to obtain at least one key pathological semantic concept that supports the classification prediction result, forming an interpretable output.

[0011] Preferably, the construction of the pathological semantic knowledge graph in step 1 specifically includes:

[0012] Step 1.1: Extract pathological entities and relationships from multi-source pathological data, which includes structured pathology reports, professional medical literature texts, and cervical cell image databases with diagnostic annotations; the pathological entities include cell structure entities, cell morphology characteristic entities, and cell lesion diagnostic entities; the relationships include attribute relationships, causal relationships, and hierarchical relationships.

[0013] Step 1.2: Standardize the terminology of the extracted pathological entities, mapping different but identical terms to standard pathological concepts, which are defined according to the International Cytopathology Terminology System.

[0014] Step 1.3: Construct a graph-structured pathological semantic knowledge graph by using standardized pathological entities as nodes and relationships between entities as edges;

[0015] Step 1.4: Use a graph neural network to perform representation learning on the pathological semantic knowledge graph to obtain a semantic vector representation with fixed dimensions for each pathological entity. This semantic vector representation encodes the semantics of the pathological entity itself and its structural context information in the graph.

[0016] Preferably, step 2, which involves extracting visual features from the cervical cell image to be classified, specifically includes:

[0017] Step 2.1: Perform staining standardization and quality screening preprocessing on the input cervical cell images;

[0018] Step 2.2: Use a cell segmentation network based on an encoder-decoder architecture to perform semantic segmentation of cell nuclei and cytoplasm in the preprocessed image, and extract bounding box image patches of individual cells;

[0019] Step 2.3: Input the bounding box image patch of a single cell into a pre-trained multi-scale convolutional neural network, extract multi-scale feature maps of the cell image patch from the shallow, middle and deep layers of the multi-scale convolutional neural network respectively, and perform adaptive fusion through a feature pyramid network to form a cell individual feature vector;

[0020] Step 2.4: Based on the spatial proximity distance of cells and the cosine similarity of individual cell feature vectors, construct a cell relationship graph with cells as nodes and spatial and morphological relationships as edges;

[0021] Step 2.5: The individual cell feature vector is used as the initial feature of the node and input into the graph attention network. Through multi-layer message passing and neighborhood feature aggregation, the visual feature vector that integrates cell population context information is updated.

[0022] Preferably, the cross-modal attention alignment module in step 3 performs the following operations:

[0023] Step 3.1: Calculate the cosine similarity between the visual feature vector and the semantic vector representations of all pathological entities in the pathological semantic knowledge graph;

[0024] Step 3.2: Normalize the cosine similarity to generate the attention alignment weight of the visual feature vector corresponding to each pathological entity. The attention alignment weight represents the correlation strength between the visual feature region and a specific pathological semantic concept.

[0025] Step 3.3: Based on the attention alignment weights, perform a weighted summation of the semantic vector representations of the relevant pathological entities to generate a semantic context vector corresponding to the visual features;

[0026] Step 3.4: Design a feature fusion gating mechanism. The feature fusion gating mechanism takes the visual feature vector as input, dynamically generates fusion weights, and uses the fusion weights to linearly combine the semantic context vector with the original visual feature vector to output the semantically enhanced fusion feature vector.

[0027] Preferably, step 4, which generates interpretable output, specifically includes:

[0028] Step 4.1: Input the semantically enhanced fusion feature vector into the classifier to obtain the predicted probability of cervical cells belonging to each lesion category;

[0029] Step 4.2: Select the K pathological semantic concepts most relevant to the classification prediction results, and select the top K attention alignment weights that are generated by the cross-modal attention alignment module and have the highest contribution to the final classification prediction probability, where K is an integer not less than 1;

[0030] Step 4.3: Based on the selected K key pathological semantic concepts and their corresponding attention alignment weights and contributions, generate a structured interpretability report; the interpretability report uses natural language description to point out the main pathological morphological basis that leads to the classification prediction result;

[0031] Step 4.4: Map the attention alignment weights corresponding to the key pathological semantic concepts back to the original cervical cell image space to generate a visual attention heatmap. In the visual attention heatmap, the brightness of an image region indicates the contribution of that region to activating a specific key pathological semantic concept.

[0032] Preferably, step 1.1, which involves extracting pathological entities and relationships from a cervical cell image database with diagnostic annotations, specifically includes:

[0033] A vision-language pre-trained model is used to perform cross-modal association learning between image region features and corresponding diagnostic description text;

[0034] The vision-language pre-trained model automatically establishes a correspondence between the visual patterns of cell images and morphological descriptive terms in pathological report text, thereby transforming visual evidence into textual descriptions that can be associated with the pathological semantic knowledge graph, and then extracting new entity relationship pairs to expand the pathological semantic knowledge graph.

[0035] Preferably, the graph attention network in step 2.5 employs a multi-head attention mechanism, wherein different attention heads are constrained to focus on relationships of different properties between cells;

[0036] The different types of relationships include, but are not limited to: spatial proximity relationships based on Euclidean distance, morphological homogeneity relationships based on feature similarity, and diagnostic consistency relationships based on cell category prediction probability. The graph attention network ultimately aggregates information from multiple relational neighborhoods to form a more comprehensive set of cell group relational features.

[0037] Preferably, during the model training phase, the cross-modal attention alignment process in step 3 introduces a semantic consistency loss function based on contrastive learning;

[0038] The semantic consistency loss function encourages cell samples belonging to the same diagnostic category and having similar key pathological semantic concepts to have their semantically enhanced fusion feature vectors close to each other in the feature space, while causing the semantically enhanced fusion feature vectors of cell samples with different diagnostic categories or having different key pathological semantic concepts to be far apart in the feature space, thereby constraining the model to learn a feature space with a clear pathological semantic structure.

[0039] Preferably, the structured interpretability report generated in step 4.3 is further linked and retrieved with the historical pathological diagnosis case database;

[0040] The specific method is as follows: using the semantically enhanced fusion feature vector or the set of key pathological semantic concepts as query conditions, searching for confirmed cases with similar characteristics in the historical pathological diagnosis case database, and incorporating the diagnostic conclusions, diagnostic basis and images of the retrieved similar cases as auxiliary reference information into the output content of the interpretability report.

[0041] Preferably, the method is applied to the auxiliary analysis of cervical liquid-based thin-layer cytology examination images or digital images of cervical tissue pathology sections; the method is deployed on a server or cloud computing platform, receives cervical cell images uploaded by front-end devices through an application programming interface, and returns classification results, interpretability reports, and visual attention heatmaps to the front-end devices.

[0042] The beneficial effects of this invention are:

[0043] 1. By constructing a pathological semantic knowledge graph and using it as prior knowledge, a deep learning model is guided to extract visual features with clear pathological significance, overcoming the blindness in feature selection inherent in traditional data-driven methods. In particular, the modeling of cell population relationships based on graph attention networks enables the model to simulate the diagnostic process of pathologists observing cell arrangement, polarity, and other population characteristics. This allows for more accurate and stable judgments when faced with samples with subtle morphological differences or complex backgrounds, effectively reducing the rates of misdiagnosis and missed diagnosis.

[0044] 2. This invention's pioneering integrated "classification-source tracing" mechanism not only outputs diagnostic conclusions but also automatically generates natural language explanations and corresponding visual heatmaps pointing to specific pathological semantic concepts (such as "increased nucleocytoplasmic ratio" and "coarsened chromatin"). This explanation directly maps to the physician's professional knowledge system, enabling them to clearly understand the model's decision-making basis, verify the rationality of the diagnostic logic, and thus establish deep trust in the AI-assisted diagnostic system, overcoming the "black box" obstacle in the application of deep learning models in the medical field.

[0045] 3. By structurally integrating pathological knowledge scattered across literature, reports, and expert experience into a computable knowledge graph and model, it is equivalent to building a tireless "digital pathology expert." This not only helps standardize diagnostic criteria in clinical practice and reduce subjective differences among physicians, but also serves as a powerful teaching tool. Through visualized feature alignment and source tracing results, it assists junior physicians in quickly and deeply understanding the key morphological indicators of various lesions, shortening their learning curve. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0048] Figure 2 This is a flowchart of the steps in step 2 of the method of the present invention to extract visual features of the cervical cells to be classified from the image.

[0049] Figure 3 This is a flowchart of the cross-modal attention alignment module in step 3 of the method of the present invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0051] Please see Figures 1-3 This invention provides an interpretable classification method for cervical cells guided by pathological semantics. The method sequentially performs four key steps: constructing a pathological semantic knowledge graph and obtaining semantic vector representations, extracting visual features from images of cervical cells to be classified, generating semantically enhanced fusion feature vectors through a cross-modal attention alignment module, and performing classification prediction based on fusion features and tracing key pathological semantic concepts to form interpretable output.

[0052] The visual feature extraction process integrates individual cell features based on image segmentation and cell group relationship features based on graph attention network modeling. The core of this method lies in establishing an explicit mapping relationship between cellular visual representation and standardized pathological semantic concepts through a cross-modal attention mechanism.

[0053] This method transforms abstract pathological diagnostic knowledge into a computable and guided structured form, enabling the entire classification decision-making process to retain the feature learning capabilities of deep learning models while being endowed with an interpretable framework that conforms to professional logic, fundamentally changing the decision-making mode of traditional black box models.

[0054] In one possible implementation, firstly, cell structure entities, morphological feature entities, and lesion diagnosis entities are extracted from structured pathology reports, electronic texts of professional medical textbooks, and cervical cell image databases with multiple annotations exported from the hospital information system, using a named entity recognition model based on a pre-trained language model.

[0055] Subsequently, the identified entities were standardized using the International Cytopathology Terminology System, unifying the inconsistent synonyms to standard concepts. Next, the standardized entities were used as nodes, and the attribute relationships, causal relationships, and hierarchical relationships between entities were used as directed edges, constructing a structured knowledge graph in the graph database.

[0056] Finally, a graph convolutional network is used to learn the representation of the knowledge graph, so that each pathological entity is encoded as a dense vector containing its own semantics and graph structure information.

[0057] This process enables the effective integration and structuring of multi-source heterogeneous pathological knowledge, providing an accurate and unified semantic space foundation for subsequent visual-semantic alignment, enabling machines to "understand" the relationships and hierarchies of pathological concepts.

[0058] In one possible implementation, the input full-view digital slice is first preprocessed with staining normalization and quality screening. Then, a semantic segmentation network with an encoder-decoder structure is used to perform pixel-level parsing of the image, segmenting the nucleus and cytoplasm regions of individual cells and extracting their image patches. Each cell image patch is fed into a pre-trained multi-scale convolutional neural network, where multi-scale features from shallow, mid-, and deep layers are fused through a feature pyramid network to form an individual cell feature vector.

[0059] Then, based on the spatial proximity distance between cells and the cosine similarity of their individual feature vectors, a cell relationship graph is constructed, with cells as nodes and spatial and morphological relationships as edges. Finally, the individual feature vectors are used as the initial feature input to the node attention network, which aggregates neighborhood information through multi-layer message passing and outputs a final visual feature vector that incorporates group context information.

[0060] This process simulates the observation logic of pathologists, moving from local to global and from individual to group. The extracted features contain both detailed cell morphology information and crucial intercellular relationship information, providing a more comprehensive visual basis for accurate classification.

[0061] In one possible implementation, firstly, the cosine similarity between the cell visual feature vector and the semantic vectors of all pathological entities in the knowledge graph is calculated. Then, these similarities are normalized using a Softmax function to generate an attention alignment weight for each pathological entity corresponding to the visual feature. This weight quantifies the relevance strength between the visual region and a specific semantic concept.

[0062] Then, the semantic vectors of the relevant pathological entities are weighted and summed according to these weights to generate a comprehensive semantic context vector. Finally, a gating mechanism is designed with visual feature vectors as input. This mechanism dynamically generates a set of fusion weights and uses these weights to linearly combine the original visual feature vectors with the semantic context vectors, outputting the final semantically enhanced fusion feature vector.

[0063] This module enables the adaptive fusion of visual information and pathological knowledge at a fine-grained level, allowing the model to selectively enhance semantic features related to diagnosis and suppress irrelevant features, thereby generating feature representations with stronger discriminative power and clear pathological significance.

[0064] In one possible implementation, firstly, the semantically enhanced fused feature vector is input into the classifier to obtain the predicted probability of a cell belonging to each lesion category. Then, from the weights generated by the cross-modal attention alignment module, the top K weights that contribute the most to the final classification prediction and their corresponding pathological semantic concepts are selected as key decision bases. Next, based on these key concepts and their contributions, a structured natural language description report is automatically generated, clearly indicating the main pathological morphological basis leading to the classification result.

[0065] Meanwhile, the attention weights corresponding to these key concepts are back-mapped to the original cell image space to generate a visual attention heatmap. The highlighted areas in the heatmap intuitively show the image parts that contribute the most to activating specific semantic concepts.

[0066] This process provides bimodal interpretable output, which simultaneously satisfies pathologists' diagnostic habits of verifying through textual logic and confirming through visual positioning, greatly enhancing the trust and adoption of AI-assisted results in clinical decision-making.

[0067] In one possible implementation, a dual-tower visual-language pre-trained model is employed. One tower is a convolutional neural network used to extract features from cell image regions, and the other tower is a text encoder used to encode the corresponding morphological description text. By training on large-scale image-text pairs, the model learns to approximate similar visual patterns with textual descriptions in a shared semantic space.

[0068] For labeled cell images in the image database, this model is used to establish associations between their visual features and pathological report descriptions, thereby automatically uncovering implicit entity relationship pairs in the images that have not been captured by traditional text mining. These new relationships are then used to expand and enrich the existing pathological semantic knowledge graph.

[0069] This method breaks through the limitation of extracting knowledge only from text, and realizes the direct transformation from visual evidence to semantic knowledge. This allows the knowledge graph to continuously evolve and become more refined as image data accumulates, making it closer to real diagnostic observation experience.

[0070] In one possible implementation, the network is designed with multiple independent attention heads operating in parallel, each constrained or guided to focus on relationships between cells of different properties. For example, one attention head might specifically compute spatial proximity weights based on Euclidean distance, another might compute morphological homogeneity weights based on feature cosine similarity, and yet another might focus on diagnostic relationship weights based on the consistency of initial cell classification predictions. During forward propagation, each attention head independently computes its relationship weights and aggregates neighbor information; finally, the outputs of all heads are concatenated or averaged to form the updated features of the nodes.

[0071] This multi-faceted division of labor mechanism enables the model to simultaneously and explicitly model multiple intercellular relationships that are crucial for pathological diagnosis, and flexibly integrate information on these heterogeneous relationships, thereby capturing complex cell population patterns in diseased tissues more comprehensively and accurately.

[0072] In one possible implementation, during model training, for cell samples within a batch, the most relevant set of key pathological semantic concepts is first determined based on their cross-modal attention weights. Subsequently, positive and negative sample pairs are constructed: samples with the same diagnostic label and highly overlapping sets of key semantic concepts form positive sample pairs; samples with different diagnostic labels or significantly different sets of key semantic concepts form negative sample pairs. The core of this loss function is to encourage the semantic enhancement fusion feature vectors of positive sample pairs to have the smallest possible distance in the feature space, while simultaneously maximizing the feature vector distance of negative sample pairs.

[0073] This loss function imposes constraints at the learning objective level, forcing the model to take the similarity of pathological semantics as an important principle for organizing the feature space, so that the learned feature space has a clear semantic structure, which not only improves classification performance, but also provides an inherent consistency guarantee for subsequent semantic tracing.

[0074] In one possible implementation, the system maintains a database of historical confirmed cases. Each case stores the fusion feature vector of its cell image, a set of key pathological semantic concepts, the final diagnosis, and a complete report. When generating an interpretable report for a new cell, the system uses the fusion feature vector or key concept set of the new cell as query criteria to perform a similarity search in the historical database. The search employs an approximate nearest neighbor search algorithm to quickly find the most similar historical cases. The diagnostic conclusions, detailed diagnostic evidence text, and images from these similar cases are extracted and appended as supplementary reference information to the newly generated interpretable report.

[0075] This feature provides pathologists with valuable cross-sectional comparative references. By presenting the treatment methods and outcomes of similar cases in the past, it further corroborates the rationality of current diagnostic recommendations, playing a dual role in assisting decision support and clinical teaching, and enhancing the practical value of the system.

[0076] In one possible implementation, the entire method is deployed on a cloud server cluster. The hospital pathology department's front-end digital slide scanner or workstation uploads cervical cell TCT images to the cloud service interface. The cloud system automatically invokes the entire process of the method, including image preprocessing, feature extraction, semantic alignment, classification, and interpretation generation. After processing, the cloud returns the classification results, a structured natural language interpretation report, and a visual attention heatmap data package to the front-end workstation. The application software on the front-end workstation receives and parses the data, labels the classification results on the images, and clearly displays the text report and interactive heatmap to the pathologist.

[0077] This deployment model enables computationally intensive intelligent analysis to be completed in the cloud, lowering the local hardware barrier for hospitals. At the same time, it provides services through standardized interfaces, facilitating the rapid promotion and application of this advanced assisted diagnostic technology in different medical institutions.

[0078] Example;

[0079] Example: Intelligent auxiliary diagnostic system for cervical liquid-based cytology examination applied in the pathology department of a tertiary hospital;

[0080] 1. Application Scenarios and Problem Description

[0081] This embodiment is applied to the routine cervical cancer screening work of the pathology department of a large tertiary hospital. The pathology department processes hundreds of cervical liquid-based thin-layer cytology (TCT) samples daily. In the traditional workflow, senior pathologists spend a significant amount of time observing cell morphology under a microscope or on fully scanned digital slides (WSI), looking for abnormalities such as enlarged nuclei, imbalanced nuclear-cytoplasmic ratios, deeply stained nuclei, and thickened chromatin, and then classifying and diagnosing them according to the Bethesda Reporting System (TBS) (e.g., NILM, ASC-US, LSIL, HSIL). This process has three prominent drawbacks: first, manual slide reading is inefficient, leading to physician fatigue and long diagnostic cycles; second, diagnostic criteria are highly subjective, with inconsistencies in diagnosis between different physicians, and even among the same physician at different times, regarding the same slide; third, training a pathologist capable of independent and accurate diagnosis is a lengthy process. This embodiment aims to deploy an intelligent assisted diagnostic system to solve the above problems and improve the accuracy, consistency, and efficiency of screening.

[0082] 2. Detailed explanation of specific implementation steps;

[0083] The core of this system is the execution of the classification method described in this invention. A complete diagnostic process begins with receiving digital pathology slides and ends with generating a report containing classification results and interpretability.

[0084] Step 1: Construct a semantic knowledge graph of cervical cell pathology;

[0085] This step is fundamental to integrating and structuring pathological knowledge. The implementation process is as follows:

[0086] First, multi-source pathological data were collected and preprocessed. Data sources included: 1) nearly 100,000 structured TBS diagnostic reports from the past five years exported from the hospital laboratory information system (LIS), with fields including a unique number, diagnostic conclusion (e.g., "high-grade squamous intraepithelial lesion, HSIL"), and morphological description text (e.g., "significantly enlarged nuclei, increased nucleus-cytoplasmic ratio, and coarse chromatin"); 2) electronic texts of authoritative textbooks such as the translation of "Bethesda Reporting System for Cervical Cytology" and "Diagnostic Cytopathology"; 3) a collaboratively constructed image library of 5,000 labeled cervical cells, each image not only having TBS category labels confirmed by multiple senior pathologists, but also pixel-level annotations of the outlines of the nucleus and cytoplasm.

[0087] Next, pathological entities and relationships are extracted. For text data, a BERT model pre-trained on a large amount of Chinese medical literature (specifically using the "RoBERTa-wwm-ext" architecture, due to its better coverage of Chinese vocabulary) is used as the foundation. First, named entity recognition is performed: after segmenting the diagnostic report and textbook text into sentences, the data is input into the model to identify and label three types of entities. For example, from the sentence "visible atypical cells with increased nuclear-cytoplasmic ratio," the cellular structural entities "nucleus" and "cytoplasm," and the morphological entity "increased nuclear-cytoplasmic ratio" are identified. Then, relationships are extracted: a BERT-based sequence labeling model is used to identify the relationships between entity pairs within the same sentence. For example, a "featured" attribute relationship is identified between "nucleus" and "increased nuclear-cytoplasmic ratio"; from the textbook paragraph "significantly increased nuclear-cytoplasmic ratio is an important indicator of high-grade lesions," a "hint" causal relationship between "significantly increased nuclear-cytoplasmic ratio" and "high-grade lesions" is extracted. For the image database, a dual-tower visual-language pre-trained model is used. One tower of the model is a ResNet-50 convolutional neural network used to extract image region features; the other tower is the aforementioned BERT text encoder used to encode the corresponding morphological description text for the image. By contrastively learning a loss function, the model is trained to align image regions describing the same morphological features with their text descriptions in a shared semantic space. After training, for new, unlabeled cell image patches, the model can automatically associate them with the most relevant text descriptions by calculating the similarity between their visual features and the features of all pathological description texts, thereby indirectly extracting potential entity relationship pairs to expand the knowledge base.

[0088] Next, terminology standardization and graph construction were performed. A thesaurus was built using the standard terminology provided by the TBS system. For example, different expressions such as "increased nucleoplasmic ratio," "increased nucleoplasmic ratio," and "increased N / C ratio" were all uniformly mapped to the standard term "increased nucleoplasmic ratio." All standardized pathological entities were stored as nodes using the graph database Neo4j. Nodes have a "type" attribute, such as "structure," "morphology," and "diagnosis." Relationships between entities are connected as directed edges, with each edge having a "relationship type" attribute. The final knowledge graph contains approximately 350 core entities and over 2000 relational edges, forming a network-like knowledge structure.

[0089] Finally, graph representation learning is performed. A Graph Convolutional Network (GCN) layer from a Graph Neural Network (GNN) is used to encode the knowledge graph. The initial features of each entity node are word vectors trained using its standard term name through a Word2Vec model. The GCN updates the representation of each node by aggregating the features of neighboring nodes. After two layers of GCN propagation, each pathological entity is encoded as a 256-dimensional dense vector, i.e., a semantic vector representation. This vector not only contains the semantics of the entity itself but also its positional information within the graph structure (e.g., the vector for "increased nuclear-cytoplasmic ratio" is close to the vectors for "enlarged nuclear mass" and "high-grade lesion").

[0090] Step 2: Extract visual features from the images of cervical cells to be classified;

[0091] This step is responsible for extracting diagnostically valuable visual information from the raw WSI. Taking a TCT digital slice with a 40x magnification as an example, the specific implementation is as follows: First, image preprocessing is performed. The Macenko method is used to normalize the color space of the input WSI, unifying the color space of all slices to the reference color template to eliminate color differences caused by different color batches. At the same time, the Laplacian variance algorithm is used to calculate the sharpness of each image region, automatically filtering out blurry areas caused by focusing issues.

[0092] Next, cell segmentation and individual feature extraction are performed. A U-Net semantic segmentation network is used to perform pixel-level classification on the preprocessed images. This network is trained on images with labeled cell nuclei and cytoplasm, outputting a probability map of each pixel belonging to "cell nucleus," "cytoplasm," or "background." Bounding boxes for each individual cell are extracted through connected component analysis. Each cell bounding box image is resized to a uniform size (e.g., 64x64 pixels) and then fed into a pre-trained Feature Pyramid Network (FPN). This FPN uses ResNet-50 as its backbone. We extract the output feature maps from stages 2, 3, 4, and 5 of the backbone network, denoted as C2, C3, C4, and C5, respectively. The C2 feature map has high spatial resolution and contains rich texture details (e.g., chromatin granules); the C5 feature map has low resolution but strong semantic information (e.g., overall cell morphology). The FPN fuses high-level semantic features with low-level detail features through top-down and lateral connections, generating a set of multi-scale feature maps (P2, P3, P4, P5). For each cell region, perform RoI Align operation at the corresponding feature layer level (e.g., P2 for small cells and P3 for large cells) to extract a feature vector of a fixed size, which serves as the individual feature vector for that cell.

[0093] Next, a cell relationship graph is constructed. For all cells in the current field of view, each cell is treated as a node, and the initial features of the node are its individual feature vectors. Edges are constructed based on two rules: 1) Spatial adjacency edges: If the Euclidean distance between the centroids of two cell nuclei is less than 50 pixels (approximately equivalent to the average diameter of the two cells), then an edge is established between them. 2) Morphological similarity edges: The cosine similarity between all pairwise individual feature vectors of all cells is calculated; if the similarity is greater than 0.7, then an edge is established. This constructs an undirected graph.

[0094] Finally, the group relationship features are aggregated. The cell relationship graph is input into a two-layer multi-head graph attention network (GAT). The first layer of GAT has four attention heads. For node i and its neighbor j, each attention head independently calculates an attention coefficient, which represents the importance of node j's features to node i. For example, one attention head may specifically learn to focus on the spatially nearest cell, while another head may specifically learn to focus on the morphologically most anomalous cell. The second layer of GAT averages the outputs of all heads in the first layer and further aggregates the information. After GAT processing, the feature vector of each cell node is updated to a new vector that incorporates its local cell group context information, i.e., the final visual feature vector. This vector not only encodes the cell's own morphology but also its arrangement relationship and similarity with surrounding cells.

[0095] Step 3: Perform feature alignment and fusion through cross-modal attention;

[0096] This step is crucial for connecting vision and semantics and achieving knowledge guidance. The specific steps are as follows:

[0097] First, the attention alignment weights are calculated. For a cell's visual feature vector v (256 dimensions), cosine similarity is calculated between it and the semantic vector representations {s_1, s_2, ..., s_N} (all 256 dimensions) of N entities in the pathological semantic knowledge graph. The formula for calculating cosine similarity is sim(v, s_N). k )=(v·s k ) / (||v||×||s k Then, the Softmax function is used to normalize all similarities to obtain the attention weight α of the cellular visual feature corresponding to each pathological entity. k The calculation formula is α k =exp(sim(v,s k )) / Σ{n=1 to N}exp(sim(v,s_n)). Weight α k It is a value between 0 and 1, all α kThe sum of these values ​​is 1, which intuitively represents "the extent to which the visual features of this cell represent the k-th pathological semantic concept".

[0098] Next, semantic context vectors are generated. Based on the calculated attention weights, the semantic vectors of all entities in the knowledge graph are weighted and summed: c = Σ{k=1 to N}(α) k ×s k The resulting vector c is the aggregated representation of the pathological semantic concept most relevant to the visual features of the cell, called the semantic context vector. Then, a feature fusion gating mechanism is designed and applied. To effectively fuse visual information v with semantic information c, we designed a gating vector g. The gating vector g is generated from the visual feature v through a fully connected layer (followed by a Sigmoid activation function): g = σ(W g ×v+b g ), where W g and b g σ is a learnable parameter, and σ is the sigmoid function. Each element of the gating vector g has a value between 0 and 1. The final semantically enhanced fusion feature vector f is calculated using the following formula: f = g⊙v + (1-g)⊙c, where ⊙ denotes element-wise multiplication. This mechanism allows the model to adaptively decide, for the current cell, whether to rely more on its original visual features or more on semantic features inferred from knowledge.

[0099] Finally, a contrastive learning loss is introduced for optimization. During model training, in addition to the standard cross-entropy classification loss, a semantic consistency contrastive loss is introduced. For a cell sample pair (i,j) in a training batch, the top 3 pathological entities with the highest weights in each sample are first identified as their key semantic concept set based on their attention weights. If samples i and j have the same TBS diagnostic label and their key semantic concept set overlap (Jaccard similarity coefficient) is greater than 0.5, they are considered positive sample pairs; otherwise, they are considered negative sample pairs. The contrastive loss L_cont is defined using the InfoNCE loss function, encouraging positive sample pairs to have closer fused feature vectors in the feature space and negative sample pairs to have farther distances. This loss function effectively constrains the feature space structure learned by the model to maintain consistency with the pathological semantic logic.

[0100] Step 4: Generate an interpretable classification and tracing report;

[0101] This step outputs the final diagnostic result and its basis. For a processed cell, the following procedure is followed: First, classification prediction is performed. The semantically enhanced fusion feature vector f is input into a fully connected classifier. This classifier consists of two fully connected layers, using ReLU activation and Dropout regularization in between, and finally outputs a 4-dimensional vector, corresponding to the predicted probabilities of the four categories: NILM, ASC-US, LSIL, and HSIL. The category with the highest probability is taken as the preliminary classification result for the cell.

[0102] Next, the etymology of key pathological semantic concepts is traced. The attention weights {α} calculated in step three are used... k In this process, the top K weights with the highest values ​​and their corresponding pathological entities are selected. In this embodiment, K=5. These 5 entities are the key pathological semantic concepts supporting the current classification decision. Simultaneously, the system calculates the contribution of each key concept to the final classification score. Specifically, this is done through gradient backpropagation, calculating the gradient of the classification probability relative to the attention weight corresponding to each concept, and then normalizing it to obtain a percentage contribution.

[0103] Then, a structured natural language explanation report is generated. The system combines the classification results with the source information to automatically generate a descriptive text. For example: "This cell is classified as high-grade squamous intraepithelial lesion (HSIL). The main criteria for judgment are: 1. Significantly increased nuclear-cytoplasmic ratio (contribution 35.2%); 2. Deeply stained nucleus (contribution 28.7%); 3. Irregular nuclear membrane (contribution 19.1%); 4. Coarse chromatin (contribution 12.5%); 5. Enlarged nucleus (contribution 4.5%)." In addition, the system also performs a similarity search between the cell's fused feature vector f and the historical case database, finding the three most similar historical confirmed cases, and attaching their diagnostic reports and image thumbnails as "similar case references" to the report.

[0104] Finally, a visual attention heatmap is generated. The attention weights corresponding to key pathological semantic concepts (such as "increased nucleoplasmic ratio") are mapped back to the image space of the original cell through backpropagation. Specifically, the gradient of the classification probability with respect to the feature map of the cell nucleus region output by the U-Net segmentation network in step two is calculated, and the gradient information is used to generate a weighted heatmap. On the final user interface, a semi-transparent color heatmap is overlaid on the original cell image. The red area represents the location that contributes the most to activating the concept of "increased nucleoplasmic ratio" (usually the nucleus region), and the blue area represents the area with a smaller contribution. Users can interactively click on different key concepts, and the heatmap will dynamically switch, intuitively showing the spatial location of different morphological evidence on the cell.

[0105] 3. System effectiveness verification and comparative analysis;

[0106] To objectively evaluate the performance of the system constructed in this embodiment, we conducted experiments on a dataset containing 3000 independent test WSI images, none of which were used in the training process. We compared the following five methods:

[0107] The baseline CNN method uses ResNet-50 to directly classify cell image patches without any interpretable modules.

[0108] The Vision Transformer method uses the ViT-B / 16 model and leverages its self-attention mechanism to provide a visual attention map.

[0109] Cell Graph NN method: Constructs a cell relationship graph and uses GAT, but does not introduce a pathological knowledge graph.

[0110] Multimodal fusion method: An advanced method for fusing multi-scale features of images, but also lacks explicit semantic knowledge guidance.

[0111] The method of this invention refers to the complete system described in this embodiment.

[0112] Evaluation metrics included classification accuracy metrics (accuracy, precision, recall, and F1 score) and interpretability metrics. Interpretability scoring was conducted independently by three senior pathologists not involved in the project. They reviewed the interpretation reports and heatmaps generated by the system for 100 randomized cases, assigning scores from 1 to 10 based on three dimensions: "clinical relevance," "logical rationality," and "auxiliary value to diagnosis." The average score was used as the final interpretability score.

[0113] The experimental results are shown in the table below:

[0114] method Accuracy (%) Accuracy (%) Recall rate (%) F1 score (%) Interpretability score (average) Benchmark CNN methods 81.5 80.2 79.8 80.0 2.1 Visual attention methods 84.0 82.9 83.1 83.0 3.5 Cell graph neural network method 86.2 85.0 85.5 85.2 3.8 Multimodal fusion method 87.8 86.7 86.9 86.8 4.0 Method of the present invention 92.3 91.5 91.8 91.6 8.7

[0115] Analysis and Conclusion:

[0116] Accuracy Advantage: The method of this invention significantly outperforms other comparative methods on all objective classification metrics. This indicates that, guided by a pathological semantic knowledge graph and through cross-modal alignment fusion, the model learns more discriminative feature representations that conform to pathological principles. Particularly in distinguishing between categories with subtle morphological differences, such as ASC-US and LSIL, the method shows the most significant improvement in F1 score, demonstrating the value of semantic knowledge in solving fine-grained classification challenges.

[0117] Explainability Advantage: The interpretability score of this method is significantly higher than other methods. Pathologists have reported that benchmark CNN methods offer no explanation; while heatmaps provided by visual attention and graph network methods highlight important areas, they fail to explain "why they are important"; whereas the natural language report provided by this invention directly relates to specific pathological concepts such as "increased nuclear-cytoplasmic ratio," and the heatmap accurately corresponds to specific morphological features, greatly enhancing physicians' trust in the model's decisions. Physicians believe that this explanation has a direct and clear auxiliary role in their review and diagnosis.

[0118] Clinical Value: In simulated auxiliary diagnostic scenarios, after introducing this system, the consistency between the diagnosis of junior pathologists and the gold standard of senior pathologists increased from 75% to 89%, and the average reading time per case decreased by approximately 40%. The "similar case reference" function provided by the system is particularly popular and has played a significant role in teaching and reference.

[0119] In summary, this embodiment fully demonstrates the entire process of the method of the present invention, from knowledge construction, feature extraction, semantic alignment to interpretable generation. Through detailed comparative experiments, it is proven that this method not only achieves an advanced level in the accuracy of cervical cell classification, but also makes a breakthrough in the key clinical requirement of model interpretability, and has good practicality and prospects for promotion.

[0120] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0121] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for interpretable classification of cervical cells guided by pathological semantics, characterized in that, Includes the following steps: Step 1: Construct a pathological semantic knowledge graph containing pathological entities of cervical cells and the relationships between entities, and obtain the semantic vector representation of the pathological entities in the pathological semantic knowledge graph; Step 2: Extract visual features from the cervical cell images to be classified. The visual features include individual cell features obtained based on image segmentation networks and cell group relationship features modeled based on graph attention networks. Step 3: Through the cross-modal attention alignment module, the visual features and the semantic vector representation are finely aligned and fused to generate a semantically enhanced fused feature vector; Step 4: Perform classification prediction based on the semantically enhanced fusion feature vector, and based on the alignment weights generated by the cross-modal attention alignment module, trace back to obtain at least one key pathological semantic concept that supports the classification prediction result, forming an interpretable output.

2. The cervical cell classification method guided by pathological semantics according to claim 1, characterized in that, The construction of the pathological semantic knowledge graph in step 1 specifically includes: Step 1.1: Extract pathological entities and relationships from multi-source pathological data, which includes structured pathology reports, professional medical literature texts, and cervical cell image databases with diagnostic annotations; the pathological entities include cell structure entities, cell morphology characteristic entities, and cell lesion diagnostic entities; the relationships include attribute relationships, causal relationships, and hierarchical relationships. Step 1.2: Standardize the terminology of the extracted pathological entities, mapping different but identical terms to standard pathological concepts, which are defined according to the International Cytopathology Terminology System. Step 1.3: Construct a graph-structured pathological semantic knowledge graph by using standardized pathological entities as nodes and relationships between entities as edges; Step 1.4: Use a graph neural network to perform representation learning on the pathological semantic knowledge graph to obtain a semantic vector representation with fixed dimensions for each pathological entity. This semantic vector representation encodes the semantics of the pathological entity itself and its structural context information in the graph.

3. The cervical cell classification method guided by pathological semantics according to claim 1, characterized in that, The specific steps in step 2 of extracting the visual features of the cervical cell images to be classified include: Step 2.1: Perform staining standardization and quality screening preprocessing on the input cervical cell images; Step 2.2: Use a cell segmentation network based on an encoder-decoder architecture to perform semantic segmentation of cell nuclei and cytoplasm in the preprocessed image, and extract bounding box image patches of individual cells; Step 2.3: Input the bounding box image patch of a single cell into a pre-trained multi-scale convolutional neural network, extract multi-scale feature maps of the cell image patch from the shallow, middle and deep layers of the multi-scale convolutional neural network respectively, and perform adaptive fusion through a feature pyramid network to form a cell individual feature vector; Step 2.4: Based on the spatial proximity distance of cells and the cosine similarity of individual cell feature vectors, construct a cell relationship graph with cells as nodes and spatial and morphological relationships as edges; Step 2.5: The individual cell feature vector is used as the initial feature of the node and input into the graph attention network. Through multi-layer message passing and neighborhood feature aggregation, the visual feature vector that integrates cell population context information is updated.

4. The cervical cell classification method guided by pathological semantics according to claim 1, characterized in that, The cross-modal attention alignment module in step 3 performs the following operations: Step 3.1: Calculate the cosine similarity between the visual feature vector and the semantic vector representations of all pathological entities in the pathological semantic knowledge graph; Step 3.2: Normalize the cosine similarity to generate the attention alignment weight of the visual feature vector corresponding to each pathological entity. The attention alignment weight represents the correlation strength between the visual feature region and a specific pathological semantic concept. Step 3.3: Based on the attention alignment weights, perform a weighted summation of the semantic vector representations of the relevant pathological entities to generate a semantic context vector corresponding to the visual features; Step 3.4: Design a feature fusion gating mechanism. The feature fusion gating mechanism takes the visual feature vector as input, dynamically generates fusion weights, and uses the fusion weights to linearly combine the semantic context vector with the original visual feature vector to output the semantically enhanced fusion feature vector.

5. The cervical cell classification method guided by pathological semantics according to claim 1, characterized in that, Step 4, which generates interpretable output, specifically includes: Step 4.1: Input the semantically enhanced fusion feature vector into the classifier to obtain the predicted probability of cervical cells belonging to each lesion category; Step 4.2: Select the K pathological semantic concepts most relevant to the classification prediction results, and select the top K attention alignment weights that are generated by the cross-modal attention alignment module and have the highest contribution to the final classification prediction probability, where K is an integer not less than 1; Step 4.3: Based on the selected K key pathological semantic concepts and their corresponding attention alignment weights and contributions, generate a structured interpretability report; the interpretability report uses natural language description to point out the main pathological morphological basis that leads to the classification prediction result; Step 4.4: Map the attention alignment weights corresponding to the key pathological semantic concepts back to the original cervical cell image space to generate a visual attention heatmap. In the visual attention heatmap, the brightness of an image region indicates the contribution of that region to activating a specific key pathological semantic concept.

6. The cervical cell classification method guided by pathological semantics according to claim 2, characterized in that, Step 1.1, which involves extracting pathological entities and relationships from a cervical cell image database with diagnostic annotations, specifically includes: A vision-language pre-trained model is used to perform cross-modal association learning between image region features and corresponding diagnostic description text; The vision-language pre-trained model automatically establishes a correspondence between the visual patterns of cell images and morphological descriptive terms in pathological report text, thereby transforming visual evidence into textual descriptions that can be associated with the pathological semantic knowledge graph, and then extracting new entity relationship pairs to expand the pathological semantic knowledge graph.

7. The cervical cell classification method guided by pathological semantics according to claim 3, characterized in that, The graph attention network in step 2.5 employs a multi-head attention mechanism, where different attention heads are constrained to focus on relationships of different properties between cells; The different types of relationships include, but are not limited to: spatial proximity relationships based on Euclidean distance, morphological homogeneity relationships based on feature similarity, and diagnostic consistency relationships based on cell category prediction probability. The graph attention network ultimately aggregates information from multiple relational neighborhoods to form a more comprehensive set of cell group relational features.

8. The cervical cell classification method guided by pathological semantics according to claim 4, characterized in that, During the model training phase, the cross-modal attention alignment process in step 3 introduces a semantic consistency loss function based on contrastive learning. The semantic consistency loss function encourages cell samples belonging to the same diagnostic category and having similar key pathological semantic concepts to have their semantically enhanced fusion feature vectors close to each other in the feature space, while causing the semantically enhanced fusion feature vectors of cell samples with different diagnostic categories or having different key pathological semantic concepts to be far apart in the feature space, thereby constraining the model to learn a feature space with a clear pathological semantic structure.

9. The cervical cell classification method guided by pathological semantics according to claim 5, characterized in that, The structured interpretability report generated in step 4.3 is further linked and retrieved with the historical pathological diagnosis case database; The specific method is as follows: using the semantically enhanced fusion feature vector or the set of key pathological semantic concepts as query conditions, searching for confirmed cases with similar characteristics in the historical pathological diagnosis case database, and incorporating the diagnostic conclusions, diagnostic basis and images of the retrieved similar cases as auxiliary reference information into the output content of the interpretability report.

10. A method for interpretable classification of cervical cells guided by pathological semantics according to any one of claims 1 to 9, characterized in that, The method is applied to auxiliary analysis scenarios of cervical liquid-based thin-layer cytology examination images or digital images of cervical tissue pathology sections; the method is deployed on a server or cloud computing platform, receives cervical cell images uploaded by front-end devices through an application programming interface, and returns classification results, interpretability reports, and visual attention heatmaps to the front-end devices.

Images