A cell image classification method based on morphological semantic guidance

Abstract

This invention discloses a cell image classification method based on morphological semantic guidance, relating to the fields of deep learning and image understanding. The method first trains a model and then uses the model to predict categories. The training process involves: first, constructing pathological morphological description text for cell categories and generating text features through a text encoder; second, extracting visual features from cell images and recombining these features through a semantic adaptation module; and third, utilizing a bidirectional attention mechanism to achieve fine-grained cross-modal alignment between visual and textual features. The prediction process involves classifying based on the similarity between cell image features and category text features. This invention utilizes prior medical knowledge to guide feature learning, enhancing the model's interpretability. Furthermore, by employing a visual-semantic space mapping mechanism, it endows the model with zero-sample reasoning capabilities for unseen categories, effectively solving the problem of traditional methods' inability to flexibly expand to new categories. This method is suitable for the auxiliary diagnosis of medical pathological images.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a cell image classification method based on morphological semantic guidance. Background Technology

[0002] With the rapid development of deep learning technology, computer-aided diagnosis has made significant progress in the field of medical pathology image analysis (Litjens, G., et al. 2017. A survey on deep learning in medical image analysis. Medical image analysis, 42, 60-88.). Especially in the specific application scenario of cervical cancer screening, liquid-based cytology is the globally accepted "gold standard" for initial screening. Its core task is to classify cells into various subtypes, such as normal, low-grade lesions, high-grade lesions, and squamous cell carcinoma, according to the internationally accepted Bethesda system standards. However, manual screening is not only labor-intensive but also prone to missed or misdiagnosed cases due to subtle morphological differences. Therefore, utilizing automated algorithms to assist doctors in conducting efficient and objective initial screening has become an urgent clinical need.

[0003] Currently, for the classification and diagnosis of pathological images, academia and industry have developed relatively mature technical approaches, mainly relying on single-modal deep learning models such as convolutional neural networks or visual Transformers. For example, convolutional neural network models have solved the problem of training difficulties in deep networks through residual connections (He, K., Zhang, X., Ren, S., and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778.). Based on this, Campanella et al. proposed a deep learning system based on multiple instance learning (MIL), which uses a convolutional neural network as a feature extractor to achieve clinical-grade classification of prostate cancer, basal cell carcinoma and breast cancer metastasis without pixel-level annotation, demonstrating the great potential of deep learning in whole slide analysis (WSI) (Campanella, G., et al. 2019. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8), 1301-1309.).

[0004] Meanwhile, visual Transformers based on self-attention mechanisms have also been introduced into pathological image analysis due to their superior global context modeling capabilities. Visual Transformers capture global features of images through self-attention (Dosovitskiy, A., et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.). For example, the GasHis-Transformer proposed by Chen et al. utilizes the Transformer structure to capture long-range dependencies in histological images, achieving superior performance compared to traditional Convolutional Neural Networks (CNNs) on gastric cancer pathological classification tasks (Chen, H., et al. 2021. GasHis-Transformer: A multi-scale visual transformer approach for gastrichistopathological image classification. arXiv preprint arXiv:2104.14528.).

[0005] However, despite the success of the above technologies in areas such as prostate cancer and gastric cancer, the following significant defects and shortcomings still exist when applied to fine-grained classification of cervical cells and actual clinical auxiliary diagnosis:

[0006] 1. Heavy reliance on large amounts of labeled data and lack of generalization and zero-shot capability: Traditional unimodal classification models are closed-set recognition systems, only able to recognize categories already present in the training set. Faced with the prevalent long-tailed distribution problem in medical scenarios, the model has a low recognition rate for rare samples (Zhang, Y., Kang, B., Hooi, B., Yan, S., and Feng, J. 2021. Deep long-tailed learning: A survey. International Journal of Computer Vision, 129, 1239–1274.); furthermore, when new lesion subtypes appear in clinical practice (not included in the training set), existing models cannot perform effective reasoning, requiring the collection of large amounts of new data and retraining, which is time-consuming and labor-intensive.

[0007] 2. Lack of semantic understanding and interpretability: Existing end-to-end classification models tend to learn the pixel statistical regularities of images rather than morphological features in a pathological sense. Although the model can output classification results, it cannot establish a correspondence between image features and medical morphological descriptions such as "increased nuclear-cytoplasmic ratio" and "irregular nuclear membrane". It is a "black box" system (Rudin, C. 2019. Stop explaining black box machine learning models for highstakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206-215.). This makes it difficult for doctors to understand the decision-making basis of the model, which limits its trust in clinical auxiliary diagnosis.

[0008] 3. Limited applicability of general multimodal models in medical scenarios: Although visual-language pre-trained models, represented by CLIP, have demonstrated powerful image-text alignment capabilities (Radford, A., et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748-8763.), their direct application in pathological analysis faces two major challenges: First, data modality mismatch. Most existing medical datasets only contain "image-label" pairs, lacking "image-text description" pairs similar to those in internet data, making it impossible to directly use models that rely on image-text pair pre-training. Second, feature granularity mismatch. General multimodal models typically perform global feature alignment from whole images to whole sentences, resulting in coarse granularity. This makes it difficult to capture small but crucial fine-grained features in pathological images (such as chromatin texture or blurred edges inside cell nuclei), limiting performance in refined pathological classification tasks.

[0009] In summary, there is an urgent need for a novel image classification method that does not require expensive paired image-text annotation, can utilize medical knowledge bases to guide feature extraction, and possesses local fine-grained alignment and zero-shot reasoning capabilities. Summary of the Invention

[0010] The purpose of this invention is to address several problems existing in current medical pathology image classification technologies: first, they heavily rely on massive amounts of labeled data and lack zero-shot reasoning capabilities, making it difficult to identify new categories not included in the training set; second, they lack semantic understanding of pathological morphology, resulting in a lack of interpretability in the model's decision-making process; and third, general multimodal models (such as CLIP) struggle to acquire high-quality paired image-text labeled data and are insufficient in aligning fine-grained features such as cell nuclear texture. Therefore, this invention provides a cell image classification method based on morphological semantic guidance.

[0011] The objective of this invention is achieved through the following technical solution: a cell image classification method based on morphological semantic guidance, comprising the following steps:

[0012] Step 1: Obtain the morphological attributes of the category to be classified and convert its structured description into natural language morphological description text. Use a text encoder to encode the morphological description text and construct a category feature library containing global semantic features and fine-grained text features.

[0013] Step 2: Input the cell image to be classified into an image encoder to extract multi-scale visual features, including global visual features and local visual features;

[0014] Step 3: Calculate the similarity between the global visual features in Step 2 and the global semantic features of all categories in the category feature library, generate coarse-grained classification prediction results, and calculate the coarse-grained classification loss value;

[0015] Step 4: Utilize the attention mechanism to perform cross-modal alignment between local visual features and global semantic features, as well as between fine-grained text features and global visual features, to generate text-guided visual features and visual-guided text features, and calculate the fine-grained alignment loss value.

[0016] Step 5: Combine the coarse-grained classification loss value and the fine-grained alignment loss value to optimize the image encoder parameters;

[0017] Step 6: Extract the global visual features of the cell image to be classified, calculate the similarity between the global visual features and the global semantic features of each category in the category feature library, and generate the category prediction result based on the similarity. For newly added category images, use a text encoder to generate the global semantic features of the newly added category image, calculate the similarity between its global visual features and its global semantic features, and generate the category prediction result based on the similarity.

[0018] Furthermore, step one includes:

[0019] Based on medical pathology standards, morphological attributes of the categories to be classified are obtained. A recursive flattening algorithm is used to transform the structured description of the morphological attributes into natural language morphological description text. The morphological description text is then encoded using a text encoder to construct a category feature library containing global semantic features and fine-grained text features. The global semantic features represent the overall semantic prototype of the category, and the fine-grained text features represent the fine-grained semantic features of each word or symbol in the morphological description text.

[0020] Furthermore, step two includes:

[0021] The cell images to be classified are preprocessed. The image encoder extracts local and global features from the preprocessed cell images. Then, feature projection and normalization are performed to obtain global and local visual features.

[0022] Furthermore, the similarity is cosine similarity.

[0023] Furthermore, in step four, visual feature adaptation is performed before the cross-modal alignment operation, including:

[0024] A visual adapter module is constructed, which includes a 3×3 convolutional layer, a first batch of normalization layers, a non-linear activation function layer, a 1×1 convolutional layer, and a second batch of normalization layers connected in series.

[0025] The visual adapter module is used to process the local visual features output by the image encoder. Local neighborhood information is aggregated and spatial noise is filtered through a 3×3 convolutional layer, and then the semantic features of the channel dimension are reorganized through a 1×1 convolutional layer to output the adapted local visual features.

[0026] Furthermore, step four includes:

[0027] The global semantic features of the category are used as the query vector, and the adapted local visual features are used as the key vector and value vector. Attention weights are generated by calculating the dot product similarity between the query vector and the key vector, and the local visual features are aggregated using these weights to generate text-guided visual features.

[0028] The global visual features of the cell image to be classified are used as the query vector, and the fine-grained text features of the category are used as the key vector and value vector. Attention weights are generated by calculating the dot product similarity between the query vector and the key vector, and the fine-grained text features are aggregated using these weights to generate visually guided text features.

[0029] A temperature-dependent contrast loss function is used to calculate the similarity between text-guided visual features and the original global semantic features of the category, and the similarity between visually guided text features and the original global visual features. Based on the two similarities, a fine-grained alignment loss value is calculated using multi-positive-sample contrast loss.

[0030] Furthermore, step five includes:

[0031] The coarse-grained classification loss and the fine-grained alignment loss are weighted and summed to construct the total loss function. The backpropagation algorithm is then used to perform end-to-end optimization of the image encoder and projection layer.

[0032] To achieve the above objectives, the present invention also provides a cell image classification device based on morphological semantic guidance, comprising one or more processors for implementing the above-described cell image classification method based on morphological semantic guidance.

[0033] To achieve the above objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described morphology-semantic guided cell image classification method.

[0034] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described morphological semantic-guided cell image classification method.

[0035] Compared with the prior art, the present invention has the following significant advantages:

[0036] 1. Overcoming the limitations of a single modality to achieve high accuracy and strong interpretability: This invention introduces morphological description as semantic guidance, enabling the model to no longer merely fit the pixel statistics of images, but to "understand" medically defined pathological features (such as identifying irregular nuclear membranes). Experimental results show that this invention significantly outperforms traditional linear classification baseline models based on full-scale fine-tuning in both classification accuracy and F1 score, effectively solving the problem of "black box" models lacking clinical trust in medical image classification.

[0037] 2. No pairwise annotation required, significantly reducing data costs: Unlike general multimodal models such as CLIP, which require massive and expensive image-text pairwise data, this invention only requires single-modal images and a general category description library built from medical guidelines for training. This weakly supervised learning paradigm at the image-label level greatly reduces the labeling threshold and cost of medical professional data.

[0038] 3. Excellent zero-sample transfer and expansion capabilities: The classification paradigm established in this invention does not rely on fixed category label IDs, but rather on semantic similarity matching. This enables the model to have open-set recognition capabilities. When faced with rare lesions or new categories not seen in the training set, there is no need to retrain the model parameters; only the corresponding text descriptions need to be added to achieve effective diagnosis, greatly improving the clinical applicability of the system.

[0039] 4. Enhanced noise resistance and fine-grained feature capture: By introducing a visual fine-tuning adapter and a cross-modal fine-grained alignment mechanism, the model can effectively filter background noise in non-lesion areas and accurately focus on diagnostically significant fine morphological structures (such as chromatin granularity), maintaining stable high performance under different experimental settings. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process of a cell image classification method based on morphological semantic guidance according to the present invention;

[0041] Figure 2 This is a comparison chart of the hyperparameter sensitivity analysis and robustness verification results of the temperature coefficient of the fine-grained alignment module in this embodiment of the invention;

[0042] Figure 3 This is a schematic diagram of the device structure of the present invention;

[0043] Figure 4 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0045] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0046] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0047] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0048] This embodiment takes the cervical liquid-based cytology pathological image classification task as an example, and combines specific algorithm formulas to provide a detailed description of the cell image classification method based on morphological semantic guidance proposed in this invention.

[0049] I. Experimental Environment and Data Preparation.

[0050] This embodiment is implemented using the PyTorch deep learning framework, and the hardware platform uses NVIDIA GPU H20 for accelerated computing.

[0051] Experimental data were collected using the HiCervix liquid-based cytology dataset. Considering that the proposed method relies on high-quality morphological descriptions as semantic guidance, and to ensure that the semantic prototypes of each category have authoritative and unambiguous medical evidence, this embodiment underwent rigorous screening during dataset construction: using the internationally recognized Bethesda System (TBS) as the gold standard, 21 fine-grained categories that could precisely align with the standard morphological descriptions of TBS were selected from the original dataset. This screening ensured that the experiment was conducted based on high-quality, standardized image-text alignment under the premise of the finest-grained classification. All images were preprocessed and uniformly adjusted to a resolution of 256×256 pixels.

[0052] II. Overall Process.

[0053] like Figure 1 As shown, the method of the present invention mainly includes: semantic library construction, visual feature extraction, coarse-grained classification prediction, fine-grained cross-modal alignment, joint model optimization, and semantic matching and zero-shot classification.

[0054] III. Detailed Implementation Steps.

[0055] 1. Model training phase.

[0056] Step 1: Semantic Library Construction. This step aims to transform the morphological knowledge of medical experts into feature vectors that the model can understand.

[0057] (1.1) Morphological Description Acquisition: Morphological attributes of the category to be classified are obtained according to medical pathology standards (such as TBS); the category to be classified is all categories included in the training samples, including various malignant lesions and benign lesions. Taking HSIL (high-grade squamous intraepithelial lesion) as an example, its structured description is as follows:

[0058] Nuclear-to-plasma ratio (N / C ratio): Markedly increased;

[0059] Nuclear membrane: Highly irregular, often with prominent indentations and grooves;

[0060] Chromatin: Can be fine or coarse granular;

[0061] Cytoplasm: Scant and immature; may appear delicate and lace-like... (small in quantity and immature, possibly lace-like);

[0062] Special Features: Hyperchromatic crowded groups (HCGs) of immature cells may be formed.

[0063] (1.2) Text serialization: The key-value pairs of the above JSON structure are converted into a coherent natural language text sequence using a recursive flattening algorithm.

[0064] (1.3) Text feature encoding and projection: Use a pre-trained text encoding model (such as PathologyBERT) as a text encoder to encode the above text and build a category feature library containing global semantic features and fine-grained text features, thereby introducing expert prior knowledge to guide model learning.

[0065] Global semantic features ( ): Take the [CLS] vector (768 dimensions) output by BERT and input it into the global projection head composed of "fully connected layer-batch normalization-ReLU-fully connected layer-batch normalization". The output dimension is B×256, which represents the overall semantic prototype of the category (such as HSIL).

[0066] Fine-grained text features ): Take the last hidden state sequence from the BERT output and input it into a local projection head consisting of "1×1 convolution-batch normalization-ReLU-1×1 convolution-batch normalization". The output is a feature sequence with dimensions B×L×256 (L is the word sequence length), representing the fine-grained semantic features of each word or token in the morphological description text.

[0067] Step 2: Visual Feature Extraction. This step focuses on image modality processing. The cell images to be classified... Inputting a ResNet-50 backbone network (image encoder) to extract multi-scale visual features:

[0068] (2.1) Image preprocessing: The input cell image with a size of 256×256 is upsampled to a resolution of 299×299 by bilinear interpolation to ensure that the subsequent feature map has sufficient spatial resolution for fine-grained alignment.

[0069] (2.2) Backbone network extraction: Input the upsampled image into the ResNet-50 backbone network.

[0070] Local feature extraction: Extract the output of the third feature layer (Layer 3) of ResNet-50. At this point, the feature map size is 19×19 (i.e., N=361), and the number of channels is 1024. This feature preserves relatively rich local texture and morphological details.

[0071] Global feature extraction: Continue forward propagation to the fourth feature layer (Layer 4) and perform global average pooling (GAP) to obtain the original global vector of 2048 dimensions.

[0072] (2.3) Feature projection: Similar to the text side, it is used to align the feature space.

[0073] Global visual features ( ): Input the 2048-dimensional original global vector into the GlobalEmbedding head and map it to a B×256 feature.

[0074] Local visual features ( ): Input the 1024-dimensional original local features into the local embedding head and map them into a B×361×256 feature sequence.

[0075] Feature normalization: for the projected and Perform L2 normalization (F.normalize) to obtain the final features used for alignment.

[0076] Step 3: Coarse-grained classification prediction, calculating the global visual features of the projected and normalized image. Global semantic features of all K categories The cosine similarity between them is used to generate coarse-grained classification prediction results (Logits).

[0077]

[0078] in, Represents the global semantic features of all K categories. The characteristic matrix formed, The temperature coefficient (set to 10.0 in this embodiment) is located in the upper right corner. This represents the transpose operation. The cross-entropy loss function is calculated based on these Logits, denoted as... .

[0079] Step 4: Fine-grained cross-modal alignment, which is the core step of this invention. An attention mechanism is used to perform cross-modal alignment between local visual features and global semantic features, as well as between fine-grained text features and global visual features, generating text-driven visual features and visual-driven text features, and calculating the fine-grained alignment loss value.

[0080] (4.1) Visual Fine-tuning Adaptation (VF Adapter): In order to further eliminate noise and enhance the semantic expression of local features, local visual features are processed by the visual adapter module before attention interaction. The module structure is as follows:

[0081] 3×3 convolution (Padding=1) → Batch Normalization (BatchNorm) → ReLU activation → 1×1 convolution → Batch Normalization (BatchNorm). The adapted local visual features are denoted as... Through the Perform global average pooling and normalization to obtain the adapted global visual features. .

[0082] The visual adapter module is used to process the local visual features output by the image encoder. First, local neighborhood information is aggregated and spatial noise is filtered through a 3×3 convolutional layer. Then, the semantic features of the channel dimension are reorganized through a 1×1 convolutional layer, and the adapted local visual features are output for subsequent alignment calculation.

[0083] The specific technical functions of each layer of components are as follows:

[0084] Batch Normalization (BN) layer: Used to normalize and reconstruct the feature distribution of the convolution output. In this invention, the BN layer not only accelerates model convergence, but more importantly, it unifies the feature distribution differences between different batches of cell images (eliminating internal covariate shifts), ensuring that the local features input to the subsequent attention module have a stable scale and distribution, and preventing feature drift caused by staining differences.

[0085] Nonlinear activation function layer (ReLU): While introducing nonlinear factors to enhance the model's ability to fit complex cell morphologies (such as irregular nuclear membranes), ReLU's one-sided inhibition property (i.e., suppressing negative values) is used to set low-confidence or irrelevant background noise features to zero (sparsening), thereby achieving implicit noise reduction and purification of local visual features.

[0086] (4.2) Two-way attention interaction.

[0087] Text-Driven Visual Alignment (T2V): The global semantic features of the category are used as the query vector, and the adapted local visual features are used as the key vector and value vector. Attention weights are generated by calculating the dot product similarity between the query vector and the key vector, and the local visual features are aggregated using these weights to generate text-guided visual features.

[0088] Specifically, utilizing the global semantic features of categories (Query) scans a local region of the image. (Key, Value). Generate visual features for text guidance. :

[0089]

[0090]

[0091] in, is the attention weight, and d represents the square root of the feature channel dimension.

[0092] Visual-driven text alignment (V2T): The global visual features of the cell image to be classified are used as the query vector, and the fine-grained text features of the category are used as the key vector and value vector. Attention weights are generated by calculating the dot product similarity between the query vector and the key vector, and the fine-grained text features are aggregated using these weights to generate visually guided text features.

[0093] Specifically, utilizing the adapted global visual features (Query) scans for local sequence features in the text. (Key, Value). Generate visually guided text features. :

[0094]

[0095]

[0096] in It is attention weight.

[0097] Fine-grained alignment loss calculation: By maximizing the consistency between the above-mentioned guiding features and the original features, the model is forced to automatically learn the precise correspondence between pixel-level features and text-level semantics without manually labeled bounding boxes.

[0098] Fine-grained alignment loss calculation (including temperature coefficient) To maximize the consistency between the guided features and the original modality features, this invention employs a temperature-coefficient-adjusted contrastive loss function. First, the scaled similarity matrix between features is calculated. For the T2V task, the guided visual features are calculated. Compared with the original global semantic features of this category similarity between :

[0099]

[0100] For V2T tasks, calculate guided text features. Compared with the original global visual features similarity between :

[0101]

[0102] Based on the above similarity, the final fine-grained alignment loss is calculated using Multi-positive Contrastive Loss. :

[0103]

[0104] The meanings of the mathematical symbols are as follows:

[0105] : Represents the final calculated bidirectional fine-grained cross-modal alignment loss value;

[0106] : Indicates the size of the current training batch;

[0107] : indicates the first in the current batch One sample index;

[0108] : Represents a set Any sample index in the dataset (used as a sample in the denominator of the comparative loss calculation);

[0109] : Represents the sample A set of positive samples belonging to the same category (Positive Set), which includes samples other than the positive samples. Other similar samples besides itself;

[0110] : Represents the set of all samples in the current training batch (Anchor Set), used as the denominator for contrastive learning;

[0111] : Indicates a sample With sets Positive samples Text-driven visual alignment similarity between the two (this value has been determined by a temperature coefficient) (Adjust scaling)

[0112] : Indicates a sample With sets Positive samples Visually driven text alignment similarity between the two (this value has been determined by a temperature coefficient) (Adjust scaling)

[0113] : Indicates a sample With sets Samples in Text-driven visual alignment similarity between the two (this value has been determined by a temperature coefficient) (Adjust scaling)

[0114] : Indicates a sample With sets Samples in Visually driven text alignment similarity between the two (this value has been determined by a temperature coefficient) (Adjust scaling)

[0115] : Represents an exponential function with the natural constant e as the base.

[0116] Step 5: Joint Model Optimization. The coarse-grained classification loss... and fine-grained alignment loss Perform weighted summation to construct the total loss function. The backpropagation algorithm is used to perform end-to-end optimization of the image encoder and projection layer. The formula for calculating the total loss function is as follows:

[0117]

[0118] in, and The hyperparameters are pre-defined weights. During the optimization process, the following parameters are updated synchronously using gradient descent:

[0119] Image encoder parameters: Fine-tune the convolution kernel weights of each residual layer in the image encoder, and adjust the convolution layer parameters through gradient backpropagation so that it can accurately extract more recognizable fine-grained morphological features such as cell nuclear membrane and chromatin from the original pixels of the cell image.

[0120] Projection layer parameters: Update the linear transformation matrix and batch normalization layer parameters in the global and local projection heads, aiming to map visual features to a feature space consistent with morphological semantics.

[0121] 2. Model-based prediction stage.

[0122] Step Six: Semantic Matching and Zero-Shot Classification. Receive the input cell image to be classified, and extract its normalized global visual features using a trained image encoder. Calculate the cosine similarity between the global visual features and the semantic prototypes of each category in the category feature library, and select the category with the highest similarity as the classification result. If the cell image to be classified belongs to a new category, receive the morphological description text of the new category and generate normalized global semantic features for the new category using a text encoder. Calculate the cosine similarity between the global visual features and the global semantic features of all candidate categories, and select the category with the highest similarity as the final classification result.

[0123] Conventional reasoning: Extract the global visual features (projected) of the cell image to be classified, calculate the similarity with the global semantic features (cached) of 21 known categories, and output the classification result.

[0124] Zero-shot inference: When it is necessary to identify new categories not included in the training set, there is no need to retrain the network. Only the morphological description text of the new category needs to be used to generate the corresponding semantic feature vector through a text encoder and projector head, and then added to the category feature library. Subsequently, the visual features of the cell image to be classified are extracted, the similarity between the visual features and the semantic features in the updated category feature library is calculated, and the classification result is determined and output based on the similarity.

[0125] IV. Verification of Experimental Results.

[0126] 1. Experimental Setup and Evaluation Metrics To comprehensively evaluate the effectiveness and robustness of the method of this invention, the experiment was conducted on the HiCervix dataset.

[0127] Multimodal encoder configuration: The image encoder uses the ResNet-50 architecture. Input images are cropped to a uniform 224×224 pixels and upsampled before being fed into the encoder to improve feature resolution. The output of the third convolutional module of the ResNet-50 is extracted as multi-scale visual features, and its feature map is flattened into 361 local feature vectors (PatchTokens). The initial model weights are pre-trained on the ImageNet-1K dataset based on a supervised image classification task.

[0128] The text encoder employs the PathologyBERT model based on the Transformer architecture, and its parameters are frozen during training. After text segmentation, truncation or padding is uniformly performed to fix the sequence length at 256 tokens. Over 5 million abstracts and full-text paragraphs of biomedical research papers were collected from PubMed and PubMed Central databases. Additionally, texts from pathology textbooks, clinical pathology reports (after anonymization), and pathology diagnostic guidelines were integrated to form a dedicated pathology language corpus, totaling approximately 800,000 documents. Pre-training was performed based on the task of predicting the next sentence from the preceding text. Feature alignment and projection: Both image embedding features and text embedding features are processed through corresponding projectors, uniformly mapping the feature dimensions to 256 dimensions to construct a common cross-modal alignment space.

[0129] The training process for the cell image classification task used the AdamW optimizer, with an initial learning rate of 2×10⁻⁶. -5 The weight decay is 0.05. The learning rate adjustment uses a cosine annealing strategy with restart.

[0130] Loss function parameters: coarse-grained classification loss during joint optimization. fine-grained alignment loss Weighted hyperparameters ( and All values ​​are set to 1.0. Furthermore, the contrastive learning temperature coefficient used in the fine-grained alignment loss calculation is... Set it to 0.1 or 0.3.

[0131] Random seed setting: In order to eliminate the influence of randomness on experimental results and ensure the reproducibility of results, the experiment was repeated under multiple different random seeds, and the mean and standard deviation were reported.

[0132] Evaluation metrics: Micro-Acc, Macro-F1 score, and Macro-AUC were used as the core evaluation metrics. Among them, Macro-F1 is more sensitive to class imbalance and better reflects the model's auxiliary diagnostic ability in minority classes (rare diseases).

[0133] 2. Comparative Experiment Method: This experiment constructed four models with different configurations for horizontal comparison (all backbones were ResNet-50):

[0134] Baseline 1 (Linear Classification Head): ResNet-50 extracts features and then directly connects to a fully connected linear layer for classification (full fine-tuning).

[0135] Baseline 2 (Multilayer Perceptron Classification Head): ResNet-50 is connected to a multilayer perceptron (MLP) for classification (full fine-tuning).

[0136] Method 3 (Semantic Classification): Introduces the text semantic feature library of this invention, and classifies the text using the cosine similarity between image features and text prototypes, but does not include a fine-grained attention alignment module.

[0137] Method 4 (Cross-modal semantic classification - this invention): Based on Method 3, this invention adds a bidirectional cross-modal cross-attention mechanism (T2V&V2T) for fine-grained feature alignment.

[0138] Analysis 5 (Linear Probing): Feature quality assessment experiment. The parameters of the image encoder trained with Method 4 are frozen, and only a linear classifier head is trained on the original dataset. This is used to verify whether the visual features learned in this invention possess better linear separability.

[0139] 3. Comparison and in-depth analysis of experimental results.

[0140] Table 1: Experimental Results

[0141]

[0142] Overall Performance Advantages and Ablation Validation: The proposed method (Method 4) achieved optimal performance across all metrics (F1 score 68.81%). On one hand, compared to the traditional linear baseline (Baseline 1, 66.73%), this invention achieved a significant improvement of 2.08%, strongly demonstrating that incorporating medical morphological descriptions as prior knowledge can overcome the performance bottleneck of relying solely on pixel statistical regularities. On the other hand, compared to semantic classification methods without fine-grained alignment (Method 3, 68.33%), the addition of a bidirectional cross-modal cross-attention mechanism further improved performance, achieving an optimal AUC of 96.94%. This deeply validates the effectiveness of the fine-grained alignment module, indicating that the model successfully captured key pathological morphological details such as "nuclear membrane irregularities" and "deep chromatin staining" through this mechanism, thereby achieving more accurate classification.

[0143] Feature Representation Quality Validation (Linear Probe Analysis): The superiority of the features extracted by this invention was further verified through the Analysis 5 (Linear Probe) experiment. After freezing the encoder trained by this invention, training only a simple linear head (67.67% F1) still outperformed Baseline 1 (66.73% F1) with full fine-tuning. This shows that the advantage of this invention lies not only in the final classification method, but also in the encoder itself learning higher-quality and more robust visual representations. Morphological semantic guidance enables the image encoder to distinguish samples of different categories better in the feature space (stronger linear separability), achieving excellent results even without relying on complex semantic matching heads, using only a simple linear layer. At the same time, the fact that Method 4 (68.81%) outperformed Analysis 5 (67.67%) also demonstrates that the semantic matching classification paradigm (based on similarity matching) has greater flexibility and accuracy than the traditional fixed linear classification boundary.

[0144] Model stability analysis: Under multiple different random seeds, the accuracy standard deviation of this invention (Method 4) is only 0.24, significantly lower than the 0.87 of Baseline 1. This indicates that by introducing stable medical prior knowledge, the model's sensitivity to initialization parameters decreases, demonstrating extremely high robustness.

[0145] 4. Zero-shot inference performance verification.

[0146] To further verify the scalability and generalization performance of the present invention in the absence of labeled samples, this embodiment designed a zero-shot classification experiment.

[0147] Experimental setup: The 21 categories of the original HiCervix dataset were divided into "visible category set" and "unseen category set".

[0148] Training phase (visible class): 16 common categories are selected as the training set. During the training process, the model only comes into contact with these 16 image categories and their corresponding text descriptions. This is mainly used to learn general visual feature extraction capabilities and cross-modal alignment mechanisms.

[0149] Testing Phase (Unseen Classes): The remaining 5 categories (including cervical adenocarcinoma, atypical glandular cells (prone to neoplasia), atypical squamous cells (not excluding high-grade lesions), herpes simplex virus, and squamous cell carcinoma) were selected as the test set. These 5 image categories had never appeared during training, and the model had not learned their corresponding classifier parameters.

[0150] Inference Method: The text encoder described in this invention generates morphological semantic feature vectors for these five unseen categories and adds them to the category feature library. The model directly performs classification prediction by calculating the similarity between the visual features of the input image and these five newly generated semantic prototypes.

[0151] Table 2: Specific performance metrics for zero-shot classification

[0152]

[0153] Results analysis:

[0154] Cross-category transfer capability: Experimental results show that the present invention achieved a macro-average AUC of 0.6517 in five unseen categories. This demonstrates that the "morphological semantic guidance" mechanism proposed in this invention successfully establishes an essential connection between visual features and textual descriptions, enabling the model to identify unfamiliar lesion types by understanding "morphological descriptions," rather than simply relying on rote memorization of training data.

[0155] 5. Hyperparameter sensitivity and robustness analysis.

[0156] To further verify the dependence of the fine-grained alignment mechanism proposed in this invention on hyperparameter settings, especially regarding the temperature coefficient defined in step four. Parameter sensitivity analysis experiments were conducted. The experimental results are as follows: Figure 2 As shown.

[0157] Experimental setup: Temperature coefficient The model was adjusted in increments of 0.2 within the range [0.1, 0.9], and the trends of the micro-average accuracy (Micro-Acc, marked by solid circles in the figure) and macro-average F1 score (marked by solid squares in the figure) on the test set were observed. The corresponding dashed and dotted lines in the figure represent the fixed performance levels of the traditional linear baseline, respectively.

[0158] Performance advantage analysis: Experimental results show that throughout the entire test range ( From 0.1 to 0.9), the performance curve (solid line with markers) of the method of the present invention always lies above the baseline level (dashed and dotted lines). Especially... When the accuracy is 0.1, this invention achieves a 1.17% improvement in accuracy and a 2.08% improvement in F1 score compared to the linear baseline. This fully demonstrates that regardless of parameter fluctuations, the introduction of morphological semantic guidance and fine-grained alignment mechanisms can bring stable positive gains.

[0159] Robustness analysis: Observing the curve trend, it can be seen that as... Despite variations in the values, the performance fluctuations of the model in this invention are relatively small (variance remains at a low level, as shown in the error bars), and no drastic performance drops occur. This indicates that the method of this invention is not limited to a specific "optimal parameter," possesses good robustness and generalization ability, can adapt to different experimental environment settings, and reduces the difficulty of parameter tuning during actual clinical deployment.

[0160] Corresponding to the aforementioned embodiments of the morphological semantic-guided cell image classification method, the present invention also provides embodiments of a morphological semantic-guided cell image classification device.

[0161] See Figure 3 The morphological semantic-guided cell image classification device provided in this embodiment of the invention includes one or more processors for implementing the morphological semantic-guided cell image classification method in the above embodiment.

[0162] The embodiments of the morphological semantic-guided cell image classification device of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device, including the morphological semantic-guided cell image classification device of this invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0163] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0164] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0165] Corresponding to the aforementioned embodiments of the morphology-semantic-guided cell image classification method, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the morphology-semantic-guided cell image classification method as described above. Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities for the morphological semantic-guided cell image classification method provided in this application embodiment, except... Figure 4 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0166] Corresponding to the aforementioned embodiments of the morphological semantic-guided cell image classification method, this embodiment of the invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the morphological semantic-guided cell image classification method described in the above embodiments.

[0167] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0169] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for classifying cell images based on morphological semantic guidance, characterized in that, Includes the following steps: Step 1: Obtain the morphological attributes of the category to be classified and convert its structured description into natural language morphological description text. Use a text encoder to encode the morphological description text and construct a category feature library containing global semantic features and fine-grained text features. Step 2: Input the cell image to be classified into an image encoder to extract multi-scale visual features, including global visual features and local visual features; Step 3: Calculate the similarity between the global visual features in Step 2 and the global semantic features of all categories in the category feature library, generate coarse-grained classification prediction results, and calculate the coarse-grained classification loss value; Step 4: Utilize the attention mechanism to perform cross-modal alignment between local visual features and global semantic features, as well as between fine-grained text features and global visual features, to generate text-guided visual features and visual-guided text features, and calculate the fine-grained alignment loss value. The calculation of fine-grained alignment loss includes: calculating the scaling similarity matrix between features, and calculating the visual features guided by the text. Compared with the original global semantic features of this category similarity between : ； In the formula, denotes the temperature coefficient, denotes the transposition operation; Computing vision-guided text features Similarity between original global vision features :​ ； Based on the above similarity , The final fine-grained alignment loss value is calculated using multi-positive-sample contrastive loss. : ； In the formula, Indicates the size of the current training batch; Indicates the first in the current batch One sample index; Represents a set Any sample index in the dataset; Representation and Sample A set of positive samples belonging to the same category; This represents the set of all samples in the current training batch; Indicates sample With sets Positive samples Text-driven visual alignment similarity between them; Indicates sample With sets Positive samples Visually driven text alignment similarity between them; Indicates sample With sets Samples in Text-driven visual alignment similarity between them; Indicates sample With sets Samples in Visually driven text alignment similarity between them; This represents an exponential function with the natural constant e as its base. Step 5: Combine the coarse-grained classification loss value and the fine-grained alignment loss value to optimize the image encoder parameters; Step 6: Extract the global visual features of the cell image to be classified, calculate the similarity between the global visual features and the global semantic features of each category in the category feature library, and generate the category prediction result based on the similarity. For newly added category images, use a text encoder to generate the global semantic features of the newly added category image, calculate the similarity between its global visual features and its global semantic features, and generate the category prediction result based on the similarity.

2. The cell image classification method based on morphological semantic guidance according to claim 1, characterized in that, Step one includes: Based on medical pathology standards, morphological attributes of the categories to be classified are obtained. A recursive flattening algorithm is used to transform the structured description of the morphological attributes into natural language morphological description text. The morphological description text is then encoded using a text encoder to construct a category feature library containing global semantic features and fine-grained text features. The global semantic features represent the overall semantic prototype of the category, and the fine-grained text features represent the fine-grained semantic features of each word or symbol in the morphological description text.

3. The method of claim 1, wherein the morphological semantic guidance based cell image classification method is characterized by, Step two includes: The cell images to be classified are preprocessed, and local and global features are extracted from the preprocessed cell images by an image encoder. Then, feature projection and normalization are performed to obtain global and local visual features.

4. The cell image classification method based on morphological semantic guidance according to claim 1, characterized in that, The similarity is cosine similarity.

5. The method of claim 1, wherein the morphological semantic guidance based cell image classification method is characterized by, In step four, visual feature adaptation is performed before the cross-modal alignment operation, including: A visual adapter module is constructed, which includes a 3×3 convolutional layer, a first batch of normalization layers, a non-linear activation function layer, a 1×1 convolutional layer, and a second batch of normalization layers connected in series. The visual adapter module is used to process the local visual features output by the image encoder. Local neighborhood information is aggregated and spatial noise is filtered through a 3×3 convolutional layer, and then the semantic features of the channel dimension are reorganized through a 1×1 convolutional layer to output the adapted local visual features.

6. The cell image classification method based on morphological semantic guidance according to claim 5, characterized in that, Step four includes: The global semantic features of the category are used as the query vector, and the adapted local visual features are used as the key vector and value vector. Attention weights are generated by calculating the dot product similarity between the query vector and the key vector, and the local visual features are aggregated using these weights to generate text-guided visual features. The global visual features of the cell image to be classified are used as the query vector, and the fine-grained text features of the category are used as the key vector and value vector. Attention weights are generated by calculating the dot product similarity between the query vector and the key vector, and the fine-grained text features are aggregated using these weights to generate visually guided text features. A temperature-dependent contrast loss function is used to calculate the similarity between text-guided visual features and the original global semantic features of the category, and the similarity between visually guided text features and the original global visual features. Based on the two similarities, a fine-grained alignment loss value is calculated using multi-positive-sample contrast loss.

7. The method of claim 1, wherein the morphological semantic guidance-based cell image classification method is characterized by, Step five includes: The coarse-grained classification loss and the fine-grained alignment loss are weighted and summed to construct the total loss function. The backpropagation algorithm is then used to perform end-to-end optimization of the image encoder and projection layer.

8. A cell image classification apparatus based on morphological semantic guidance, characterized by, It includes one or more processors for implementing the morphology-semantic-guided cell image classification method according to any one of claims 1-7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the morphological semantic-guided cell image classification method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the morphological semantic-guided cell image classification method as described in any one of claims 1-7.

Images