Intelligent auxiliary method for cell pathology based on Clip-Retrival model

Abstract

The invention provides a cell pathology intelligent auxiliary method based on a CLIP-RETRIEVAL model, and relates to the technical field of medical image processing, and the method comprises the following steps: S10, obtaining data of a cervical cell pathology image and a pathology text, and constructing a structured data set of an image-text pair; s20, preprocessing the cervical cell pathological image, and respectively generating corresponding feature vectors according to the preprocessed cervical cell pathological image and pathological text; s30, constructing a unified medical semantic embedding space, aligning feature vectors of the image-text pairs through the dynamic projection matrix P, and outputting alignment features; and S40, inputting the alignment features into an LLM model, and generating a structured pathology report. Through data acquisition, preprocessing, multi-modal feature extraction, semantic alignment and vertical question and answer function adaptation, the problems of high misuse rate of medical terms and semantic segmentation of images and texts are solved, and the accuracy and interpretability of pathological diagnosis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to an intelligent cytopathology auxiliary method based on a Clip-Retrieval model. Background Art

[0002] In the fields of artificial intelligence and medical image processing, multimodal models are widely used in pathology-assisted diagnosis. Traditional technologies rely on static embedding spaces for semantic matching between images and text, and language models lack vertical optimization for the medical field. This leads to a high rate of misuse of medical terminology in diagnostic results, and makes it difficult to dynamically align the semantic associations between images and text, failing to meet clinical demands for accurate diagnosis.

[0003] With the increasing application of artificial intelligence in the medical field, pathology diagnosis is placing higher demands on the semantic alignment accuracy and domain adaptability of multimodal models. Multimodal models must dynamically associate medical image features with pathology text terms in a unified semantic space, while also incorporating medical domain knowledge to improve diagnostic accuracy and interpretability.

[0004] However, existing technologies have the following limitations. First, traditional multimodal models rely on static embedding spaces of fixed dimensions, making it difficult to capture the dynamic semantic associations between cell morphology and pathological terms. Second, general multimodal models are not optimized for the accuracy and contextual logic of medical terminology, resulting in a high rate of misuse of professional terms in generated reports and a lack of interpretability of diagnostic results. Third, existing methods do not introduce dynamic optimization methods such as contrastive learning or knowledge distillation, and are unable to adaptively adjust semantic alignment strategies according to the complexity of pathological features.

[0005] Related technologies, when processing multimodal medical data, rely solely on static feature mapping and general language models, making it difficult to address the semantic gap between image feature diversity and text terminology accuracy in pathological diagnosis. For example, traditional models achieve less than 80% semantic matching accuracy for key features such as an abnormal nuclear-cytoplasmic ratio, and are unable to dynamically distinguish subtle differences between papillary arrangement (a characteristic of malignancy) and follicular cell clusters (a characteristic of benign disease), resulting in a false positive rate exceeding 15%. Summary of the Invention

[0006] In view of the shortcomings of the above-mentioned existing technologies, the technical problem to be solved by the present invention is to provide an intelligent assisted method for cytopathology based on the Clip-Retrieval model. Through data acquisition, preprocessing, multimodal feature extraction, semantic alignment and vertical question-answering function adaptation, it solves the problems of high misuse rate of medical terminology and semantic separation between images and text, and improves the accuracy and interpretability of pathological diagnosis.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is as follows: the present invention provides an intelligent cytopathology assisted method based on the Clip-Retrieval model, comprising the following steps:

[0008] S10, obtaining data of cervical cell pathology images and pathology texts, and constructing a structured dataset of image-text pairs;

[0009] S20, preprocessing the cervical cell pathology image, and generating corresponding feature vectors according to the preprocessed cervical cell pathology image and pathology text;

[0010] S30, constructing a unified medical semantic embedding space, aligning the feature vectors of the image-text pair through the dynamic projection matrix P, and outputting the aligned features;

[0011] S40: Input the alignment features into the LLM model to generate a structured pathology report.

[0012] In a preferred solution, in step S10, the structured data includes a cervical cell pathology image in a specific format, a pathology text corresponding to the cervical cell pathology image, and an annotation label.

[0013] In a preferred solution, the specific steps of generating the feature vector of the image-text pair in step S20 are as follows:

[0014] S21. Based on the ViT model, the preprocessed cervical cell pathology image is divided into image blocks of a specific size, mapped into embedding vectors through a linear projection layer, and a learnable position encoding is added. The cell morphology is extracted through a multi-layer Transformer encoder to generate a 256-dimensional image feature vector f. ViT ;

[0015] S22. Input the key terms in the pathology text into the BERT model pre-trained in the medical field. After the input pathology text is segmented by WordPiece, a word embedding vector is generated. The dynamic representation of words in the context is learned through the Transformer encoder, and the context-aware 768-dimensional text feature vector f is extracted. BERT .

[0016] In a preferred solution, the specific steps in step S30 are as follows:

[0017] S31, through the learnable matrix and The 256-dimensional image feature vector f output by the ViT model ViT The 768-dimensional text feature vector f output by the BERT model BERT Mapped to 768-dimensional medical semantic space, generating image-text pairs (f I ,fT );

[0018] S32, based on the projected image-text pair (f I ,f T ), optimize the semantic space by contrastively learning the triples of the loss function L, so that the similarity of matching pairs is higher than that of non-matching pairs;

[0019] S33. Introduce IGTA into the contrastive learning of step S32 to guide the attention allocation of text features based on image features and strengthen the semantic matching of key terms.

[0020] S34, input image feature f I To the fine-tuned LLM model, generate the term probability distribution p LLM (t|f I );

[0021] Clip Retrieval generates term probability distribution p through softmax Student (t|f I );

[0022] Through the knowledge distillation loss function L KD The divergence measures the probability distribution of terms p Student (t|f I ) and the term probability distribution p LLM (t|f I ) distribution difference;

[0023] S35. Combine contrastive learning loss L with knowledge distillation loss L KD Integration, make f I With f T Semantically align in a unified space to generate aligned multimodal feature vectors

[0024] In the preferred solution, in step S31, the learnable projection matrix The 256-dimensional image feature vector f ViT Mapping to the 768-dimensional medical semantic space, the formula is:

[0025] f I =LayerNorm(P I ·f ViT +b I );

[0026] in, is the image projection matrix, and the nonlinear mapping from image features to medical semantic space is learned through back propagation. is the bias vector that compensates for the semantic offset in the projection process. LayerNorm is the layer normalization operation that adjusts the feature distribution to a mean of 0 and a variance of 1.

[0027] For the 768-dimensional text feature vector f BERT Using residual connection optimization, the formula is:

[0028] f T =LayerNorm(P T ·f BERT +f BERT )=LayerNorm((P T +I)·f BERT );

[0029] in, is the text projection matrix, which is added to the identity matrix I to form a residual connection, preserving the original semantics while optimizing the cross-modal mapping.

[0030] In the preferred solution, the specific steps of step S32 are as follows:

[0031] S321, let triple and pathological complexity label c∈{1,2,3,4,5};

[0032] Among them, the triplet includes anchor samples, positive samples and negative samples. The anchor sample is the image feature vector Positive samples are matching text feature vectors Negative samples are random non-matching text feature vectors

[0033] S322, based on the anchor image feature vector Dynamically adjust the comparative learning margin value based on the pathological complexity;

[0034] S323, forcing the similarity of positive sample pairs to be higher than that of negative sample pairs by at least margin, maximizing the similarity between the anchor image and the matching text, and minimizing the similarity with the non-matching text, the contrast loss function formula is:

[0035]

[0036] in, is the cosine similarity, with a value range of [1,1];

[0037] When the average similarity of positive sample pairs is ≥ 0.9 and that of negative sample pairs is ≤ 0.5, the contrastive learning training is stopped.

[0038] In the preferred solution, the specific steps of step S32 are as follows:

[0039] S331, the text feature vector f output by BERT TDecompose into sub-vectors of single tokens

[0040] Where N is the number of terms in the text, obtained through BERT's Token vector;

[0041] S332, through image feature f I With each Token sub-vector Calculate attention weights;

[0042] S333. Aggregate term features by attention weight. The formula is:

[0043]

[0044] Among them, α i is the attention weight of the i-th term, is the feature of the i-th term in the text, and g is the gating vector.

[0045] In a preferred embodiment, the present invention further provides a computer device / equipment / system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the steps of the above-mentioned intelligent assisted method for cytopathology based on the Clip-Retrieval model.

[0046] In a preferred embodiment, the present invention further provides a computer non-transitory readable storage medium having a computer program / instruction stored thereon, characterized in that when the computer program / instruction is executed by a processor, the steps of any of the above-mentioned intelligent assisted methods for cytopathology based on the Clip-Retrieval model are implemented.

[0047] In a preferred embodiment, the present invention further provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by one or more processors, it implements the steps of any of the above-mentioned intelligent assisted methods for cytopathology based on the Clip-Retrieval model.

[0048] The present invention provides an intelligent assisted cytopathology method based on the Clip-Retrieval model. Through the coordination of the above structures, the following beneficial effects can be achieved:

[0049] First, through a dynamic semantic alignment framework, namely cross-modal projection + contrastive learning + IGTA + knowledge distillation, it overcomes the static alignment flaw of traditional CLIP in the medical field and achieves a three-level precise mapping of "image features - pathological terms - diagnostic logic";

[0050] Second, it can reduce the risk of pathologists misdiagnosing benign cases, especially in differentiating benign and malignant thyroid nodules, by reducing the number of false positive reports by approximately 70%.

[0051] Third, it improves the diagnostic efficiency of complex cases by providing interpretable semantic alignment results, such as automatically labeling "gravel body corresponding image area", to assist doctors in quickly completing report review, reducing the average review time from 10 minutes to 3 minutes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings and examples:

[0053] Figure 1 This is an example diagram of the prediction of thyroid and cervical cells of the present invention;

[0054] Figure 2 This is a predicted example diagram of cervical squamous epithelial cells;

[0055] Figure 3 It is a schematic diagram of the hardware and software of the present invention;

[0056] Figure 4 It is a flowchart of the AI ​​server of the present invention;

[0057] Figure 5 This is a schematic diagram illustrating the report generation of the present invention;

[0058] Figure 6 It is a schematic flow chart of the present invention;

[0059] Figure 7 This is the architecture diagram of ViT and BERT in the present invention;

[0060] Figure 8 is a comparative learning loss curve graph in the present invention;

[0061] Figure 9 It is a schematic diagram of the structure of the computer device / equipment / system of the present invention. DETAILED DESCRIPTION

[0062] In order to better understand the purpose, structure and function of the present invention, the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0063] In the fields of artificial intelligence and medical image processing, multimodal models are widely used in pathology-assisted diagnosis. Traditional technologies rely on static embedding spaces for semantic matching between images and text, and language models lack vertical optimization for the medical field. This results in a high rate of misuse of medical terminology in diagnostic results, with an error rate exceeding 15%. Dynamic alignment of semantic associations between images and text is difficult, making it impossible to meet clinical demands for accurate diagnosis.

[0064] With the in-depth application of artificial intelligence technology in the medical field, pathological diagnosis has placed higher demands on the semantic alignment accuracy and domain adaptation capabilities of multimodal models. Multimodal models need to dynamically associate medical image features with pathological text terms in a unified semantic space, while incorporating medical domain knowledge to improve diagnostic accuracy and interpretability. However, existing technologies have the following limitations: First, static semantic matching defects. Traditional multimodal models rely on static embedding spaces of fixed dimensions, making it difficult to capture the dynamic semantic associations between cell morphology and pathological terms.

[0065] Second, the domain adaptation capability is insufficient. The general multimodal model is not optimized for the accuracy and contextual logic of medical terminology, resulting in a high rate of misuse of professional terms in the generated reports and a lack of interpretability of the diagnostic results.

[0066] Third, there is a lack of dynamic optimization mechanism. Existing methods do not introduce dynamic optimization methods such as contrastive learning or knowledge distillation, and cannot adaptively adjust the semantic alignment strategy according to the complexity of pathological features.

[0067] When processing medical multimodal data, related technologies only use static feature mapping and general language models, which makes it difficult to solve the semantic gap problem of "image feature diversity-text terminology accuracy" in pathological diagnosis. For example, the semantic matching accuracy of traditional models for key features such as "abnormal nuclear-cytoplasmic ratio" is less than 80%, and it is unable to dynamically distinguish the subtle differences between "papillary arrangement" (malignant feature) and "follicular cell clusters" (benign feature), resulting in a false positive rate of more than 15%. This embodiment significantly improves the processing capabilities of medical multimodal data through dynamic semantic alignment mechanism and vertical classification model optimization, providing an innovative solution for intelligent auxiliary diagnosis of pathology.

[0068] In this embodiment, the Vision Transformer model, the Bidirectional EncoderRepresentations from Transformers model, and the Clip-Retrieval model will be used;

[0069] Vision Transformer (ViT) is a visual model based on the self-attention mechanism. It divides the image into fixed-size blocks (such as 16×16 pixels), generates patch embeddings through linear embedding + position encoding, and extracts global features through the Transformer encoder.

[0070] Bidirectional Encoder Representations from Transformers (BERT) is a bidirectional pre-trained language model based on Transformer. It learns text context semantic representations through masked language model (MLM) and next sentence prediction (NSP) and supports cross-task transfer learning.

[0071] Clip-Retrieval is a multimodal model that maps images and text into the same embedding space through contrastive learning, enabling semantic retrieval and alignment.

[0072] Example 1

[0073] like Figures 1 to 8 As shown, a cytopathology intelligent auxiliary method based on the Clip-Retrieval model is characterized by comprising the following steps:

[0074] In this embodiment, Figure 1 、 2 As shown in FIG9 , in the prediction example of thyroid and cervical cells, step S10 inputs data of cervical cell pathology images and pathology texts. The specific input data is as follows:

[0075] Cervical liquid-based cytology (LCT) and thyroid fine needle aspiration (FNA) smears, covering categories such as normal (N), atypical squamous cells (ASC), low-grade squamous intraepithelial lesions (LSIL), high-grade squamous intraepithelial lesions (HSIL), and squamous cell carcinoma (SCC), totaling 1 million cervical samples and 100,000 thyroid samples.

[0076] Each image corresponds to a TBS diagnosis report (cervix) or a TBSRTC classification report (thyroid), which includes key terms such as "nuclear hyperchromasia," "ground-glass nucleus," "squamous epithelial atypia," "glandular dysplasia," or "sammoth bodies," diagnostic conclusions such as "consistent with HSIL," and clinical metadata such as sample number and staining type.

[0077] Extract key terms from pathology reports. Use OCR technology to extract text from paper reports, combine regular expressions to filter non-medical terms (such as "patient number" and "examination date"), and retain pathology-related vocabulary.

[0078] A cross-modal annotation system was constructed, and a two-person annotation system of pathology experts was adopted to establish fine-grained mappings between cell regions in the image, such as single abnormal cells and cell clusters, and text terms. For example, "nuclear groove" corresponds to the concave area of ​​the cell nucleus in the image, forming 1 million cervical cell image-text pairs + 100,000 thyroid image-text pairs of annotation data, solving the problem of inaccurate semantic alignment caused by traditional coarse-grained annotation (only annotating image categories). A large-scale data set was constructed through collaborative data from multiple institutions to improve the generalization ability of the model.

[0079] Table 1

[0080]

[0081] Among them, the structured dataset includes cervical cell images in PNG / JPEG format, corresponding pathological text in TXT / JSON format, and annotation labels classified as "normal / abnormal" and "CIN grade".

[0082] After double-blind review by doctors, the labeling accuracy is >99%.

[0083] like Figure 3 、 4 As shown, a local AI pathology diagnosis system with high scalability and easy maintenance is provided. It realizes local processing of the entire process of pathology slide scanning, AI analysis, and report generation. It supports flexible deployment from single device to multi-device clusters, and reduces operation and maintenance costs through a visual interface and log system, including scanners and AI servers.

[0084] Among them, the scanner is equipped with a high-resolution optical sensor, supports automatic slide loading and scanning, and outputs lossless images such as TIFF / PNG. It is compatible with pathological slide types such as cervical cells, thyroid, and DNA.

[0085] The AI ​​server is configured with at least one NVIDIA A100 GPU, a 32-core CPU such as Intel Xeon Silver 4310, 256GB of memory, and 20TB of SSD storage, and deploys a MySQL database, Redis cache, ELK log system, and a local report generation module.

[0086] In this embodiment, the extended architecture adopts one of a single device mode or a multi-device mode;

[0087] In one feasible approach, the expansion architecture adopts a single-device mode, 1 scanner + 1 AI server, which is suitable for small laboratories.

[0088] The expansion architecture adopts a multi-device mode, which includes horizontal expansion and vertical expansion;

[0089] Among them, horizontal expansion is 1 scanner + N AI servers (AI tasks are distributed through load balancing);

[0090] Vertical expansion to M scanners + N AI servers (multi-source data synchronization through database queues), supporting simultaneous processing of multiple types of pathology samples.

[0091] In this embodiment, the MySQL database is used as the data center and includes four modules: scanning module, AI service module, reporting module and log module;

[0092] Among them, the scanning module stores the original image data of the scanner into MySQL, and triggers the Redis queue to notify the AI ​​service to retrieve the data.

[0093] The AI ​​service module is built based on the Cerely asynchronous framework and supports the parallel running of multiple AI algorithms (such as YOLOv8 cell detection and ResNet pathology classification). The processed results are temporarily stored in Redis and updated to MySQL after verification.

[0094] The reporting module reads AI analysis results from MySQL, automatically generates PDF reports (including cell counts, lesion probability, reference maps, etc.), and supports custom template configuration.

[0095] The log module uses ELK (Elasticsearch+Logstash+Kibana) to achieve full-process log collection, analysis, and visualization, recording information such as data flow, algorithm time consumption, and hardware status.

[0096] When in use, place the cervical cell slide into the scanner, start the scanning task, generate a 200MB / TIFF format image, and store it in the "raw_images" table of the MySQL database. The fields include "sample_id, image_path, scan_time". The MySQL trigger detects the insertion of new data and pushes the task message (including sample_id) to the Redis queue "ai_task_queue". The Cerely framework schedules idle GPU instances to obtain tasks from Redis, loads the YOLOv8 cervical cell detection model, identifies the cell area and outputs the coordinate data. The results are stored in the Redis "processed_results" cache. The scheduled task pulls the processed results from Redis, combines the sample information in MySQL, and generates a report through PythonReportLab, including cell density statistics, abnormal cell markers and risk levels (low / medium / high). Events such as scan start / end, AI task scheduling, and report generation are all written to E.

[0097] In this embodiment, Figure 7As shown, S20, image preprocessing: preprocessing the cervical cell pathology image obtained by the scanner, and the preprocessing adopts the following steps:

[0098] S21, median filtering denoising: Use a 3×3 sliding window to remove salt and pepper noise and retain cell boundary details.

[0099] For each pixel in the image, the median of the pixel values ​​within a 3×3 window is taken and replaced with the original pixel value to effectively eliminate salt and pepper noise, such as dye particle interference. A 5×5 window is used based on dynamic selection of noise intensity to address the grid-like noise common in cervical cell images. For thyroid smears, due to the high cell density, a 3×3 window is used to preserve details, and a Gaussian kernel with σ = 1.5 is used to smooth the image and reduce spot interference. For liquid-based cytology samples, morphological operations (such as opening operations) are used to remove background impurities.

[0100] S22. Grayscale normalization: Normalize the grayscale of the image to compensate for brightness differences between different devices or staining conditions, and map the pixel values ​​to the range of [0, 1]. The formula is:

[0101] Where μ is the mean grayscale value of the image, σ is the standard deviation, and the pixel values ​​are normalized to the range [0, 255] to eliminate the brightness differences between different devices.

[0102] Histogram equalization is used to enhance the contrast of cell boundaries, which is convenient for subsequent feature extraction and enhances the contrast of cell boundaries. The formula is:

[0103] Where L is the number of gray levels (256), N is the total number of pixels, and n j is the number of pixels at the jth gray level.

[0104] Specifically, an adaptive normalization strategy is designed for different staining methods (such as Papanicolaou staining and HE staining) to compensate for the brightness difference caused by uneven staining, and dynamic histogram equalization is introduced to avoid noise amplification caused by over-enhancement.

[0105] Noise reduction effect: PSNR (peak signal-to-noise ratio) is improved from 22.3dB to 28.7dB (the average of the existing technology is 25.1dB).

[0106] After normalization, the image contrast is improved by 30% and the accuracy of cell boundary detection is improved by 18%.

[0107] The preprocessed cervical cell pathology image is output. The preprocessed cervical cell pathology image has reduced noise, uniform brightness, and clear cell boundaries, making it suitable as the input of the Vision Transformer (ViT) model.

[0108] S23. The pre-processed cervical cell pathology image is input into the Vision Transformer (ViT) model to extract high-dimensional pathological features and capture cell morphology and arrangement patterns, as follows:

[0109] S231, image segmentation: resize the pre-processed cervical cell pathology image to 224×224 pixels and segment it into image blocks of fixed size. In a preferred embodiment, the image blocks are segmented into 14×14=196 patches, where each patch is 16×16 pixels.

[0110] S232, Embedding: Through Linear Projection Layer Each patch is converted into a 256-dimensional vector and a learnable position encoding (Pos Embedding) is added to preserve spatial information.

[0111] S233, Multi-layer Transformer Encoding: uses a 12-layer Transformer Encoder, each layer of which contains Multi-Head Attention (MHA) and a Feedforward Neural Network (FFN);

[0112] Calculate the global dependency between image blocks using the formula:

[0113]

[0114] Among them, Q, K, V are query, key, and value matrices respectively, d k To embed the dimension, the attention weights between patches are calculated to capture cell morphological features (such as nuclear membrane irregularity) and arrangement patterns (such as papillary structures).

[0115] The ViT model outputs a high-dimensional image feature vector (768 dimensions) containing the spatial distribution information of cell structures.

[0116] Specifically, a pathological feature enhancement module (PFE Module) is introduced into the ViT output layer: through the weighted attention mechanism, the cell nuclear region features are enhanced (for example, the feature vector weight of cells with a high nuclear-cytoplasmic ratio is increased by 20%), and background noise interference is suppressed. The quantitative results show that the feature vector activation value of key pathological features (such as nuclear grooves) is increased by 35% (compared with standard ViT), and the semantic relevance between visual features and pathological terms (assessed by CLIP) is increased from 0.68 to 0.82. On the Cervical Cancer Dataset, the ViT model's recognition accuracy for CIN grading (82.2%) is significantly higher than that of traditional CNN (ResNet-152, 68.4%).

[0117] In terms of response time, ViT takes 0.45 seconds to process a single image (the average time for existing technologies is 1.2 seconds).

[0118] The final output image feature vector 256-dimensional image feature vector f ViT .

[0119] S24, BERT encoding text features: key terms such as "squamous epithelial atypia", "glandular dysplasia", "CIN Ⅰ", "CIN Ⅲ" in pathological text and diagnostic conclusions are converted into a format that can be processed by the model, and the contextual semantics of medical terms are retained, that is, the original text is converted into a format that can be used with the image feature vector The aligned vector representation includes WordPiece segmentation and context-aware encoding. WordPiece segmentation is a subword-based segmentation method that splits unknown words into common subword units, improving the coverage of medical terms and retaining punctuation marks (such as commas and periods) to assist in context understanding.

[0120] Here is an example:

[0121] Original text: "Squamous epithelial atypia with nuclear enlargement".

[0122] Word segmentation results: ["squamous","epithelial","atypia",",","cell","nucleus","enlargement"].

[0123] Context-aware encoding is to learn the dynamic representation of words in context through the Transformer encoder. The specific steps are as follows:

[0124] S241, Medical Tokenizer: Based on improvements to BioBERT-v1.1, it integrates SNOMED CT and the UMLS medical terminology database to accurately segment complex terms such as "atypical squamous cell predisposition high-grade lesion (ASC-H)." It converts the segmented text into a numerical vector representation, capturing word meaning and position information. It also adds special tokens, including [CLS], [SEP], and [MASK], to pathology text after tokenization to generate word embeddings, position embeddings, and paragraph embeddings.

[0125] Among them, [CLS] marks the beginning of a sentence, and its final output vector is used to represent the semantics of the entire sentence. [SEP] separates sentence pairs (such as the comparison of two pathological terms), and [MASK] is used for pre-training tasks (which may no longer be used in the subsequent fine-tuning stage).

[0126] Example input format: [CLS] Squamous epithelial atypia, nuclear enlargement [SEP]

[0127] Stop word filtering is to remove common words such as "of", "is", "examination", etc., and retain medical terms such as "atypia", "nuclear enlargement". The filtering rule is to define a stop word list based on a medical dictionary (such as SNOMED-CT).

[0128] Among them, Token Embedding (word embedding) is to map each token to a 768-dimensional vector (consistent with the ViT image feature dimension), and the vocabulary size is 30,000 sub-word units (such as the vocabulary of BioBERT).

[0129] Weight learning is to learn word vectors through pre-training tasks (such as masked language models) to make similar terms (such as "atypia", "nuclear enlargement") close in the vector space.

[0130] Position Embedding is to assign a unique 768-dimensional vector to the position of each token (the first word, the second word, etc.), helping the model perceive the sequence order. The method is to generate fixed position encodings through sine / cosine functions, or through a learnable position embedding matrix.

[0131] Segment Embedding is for single-sentence input, and the Segment Embedding of all tokens is 0; if processing sentence pairs (such as the association between terms and diagnostic conclusions), it is marked as 0 or 1 to distinguish different sentences and assist the model in understanding the context relationship.

[0132] S242: Generate a 768-dimensional initial vector through the vector combination of Token Embedding + Segment Embedding + Position Embedding, and the maximum sequence length is 128.

[0133] S243: Context semantic encoding: Capture the context association of terms through masked language modeling (MLM)

[0134] S243: Bidirectional Transformer: Use 12 layers of Encoder to learn the dependency relationship between terms through the self-attention mechanism. For example, for "nuclear enlargement", the model will pay attention to the association between "nucleus" and "enlargement", and at the same time establish connections with other terms (such as "atypia").

[0135] S244: Feed-forward neural network (FFN): Each attention layer is followed by two fully connected networks (input → intermediate layer → output) to enhance the non-linear expression ability.

[0136] The parameters are set as input dimension: 768, intermediate layer dimension: 3072, output dimension: 768

[0137] S245, fine-tuning and feature extraction: Optimize the model for the specific needs of pathology reports and generate text vectors adapted to ViT image features.

[0138] Domain-Specific Fine-tuning: Use a pathology report corpus (e.g., 100,000 annotated data) for fine-tuning.

[0139] Text classification: predicting diagnostic conclusions (such as CIN grading).

[0140] Term matching: Determine the relevance of terms to images (e.g., matching “sand body” to thyroid cancer images).

[0141] Loss function: cross entropy loss (for classification) + contrastive loss (for term matching).

[0142] Feature extraction: Output [CLS] tagged 768-dimensional text feature vector f BERT , as the 768-dimensional semantic vector of the entire sentence.

[0143] Term-level vector: If fine-grained matching is required, extract the word segmentation vector of each term (such as the vector of "heterogeneity").

[0144] Here is an example:

[0145] Enter the text: "Squamous epithelial atypia, nuclear enlargement".

[0146] Output vector: 768-dimensional vector (output of [CLS] tag), representing the pathological semantics of the entire sentence.

[0147] S244. Domain knowledge injection: 1 million PubMed pathology articles were added during the pre-training phase to enable the model to understand the semantics of rare terms such as "intranuclear pseudoinclusions".

[0148] Design a special prompt template for pathology report generation to guide BERT to generate feature descriptions that conform to pathology logic.

[0149] The dedicated prompt template includes [image features] prompting [diagnostic direction], combined with [clinical information], and considering [conclusion];

[0150] Validation data: The accuracy of term context association (such as "nuclear dark staining → LSIL") increased from 72% to 89%; the initial semantic similarity between text features and visual features (before alignment) increased from 0.55 to 0.71.

[0151] S30. To solve the alignment problem between image and text features caused by dimensional heterogeneity and semantic space fragmentation, dynamic cross-modal mapping and domain knowledge injection are used to construct a unified feature space with consistent medical semantics, achieving a precise association of "visual features → diagnostic terms → pathological conclusions". Clip-Retrieval dynamic semantic alignment: image features and text features are mapped to the same semantic space, and the association between pathological images and terms is established. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, discriminative features are learned, and the weight distribution of image and text features is dynamically adjusted to capture multi-scale semantic associations.

[0152] The following describes Clip-Retrieval dynamic semantic alignment through a core logic chain: dimension alignment (solving feature space heterogeneity) → dynamic contrastive learning (strengthening matching features and suppressing false associations) → image-guided attention (focusing on key diagnostic terms) → knowledge distillation (injecting expert diagnostic logic) → joint optimization (balancing data-driven and domain knowledge). The specific steps are as follows:

[0153] S31. Construction of a unified embedding space: This step achieves dynamic semantic alignment of images and texts through a five-layer technology stack. Each layer contains clear input and output, algorithm logic, and optimization goals to form a complete technical solution. The image feature dimension (256 dimensions) output by ViT is incompatible with the BERT text feature dimension (768 dimensions), and dimensional alignment must be achieved through a learnable linear transformation. Traditional fixed mappings (such as fully connected layers) are prone to semantic distortion, while the present invention uses normalized residual projections to ensure that the feature distribution is stable and retains the original semantics, so that the two features can be directly compared in a unified semantic space, as follows:

[0154] S311, Image Feature Dimensionality Up-Projection and Normalization: Through Learnable Projection Matrix The 256-dimensional image feature vector f ViT Mapped to the 768-dimensional medical semantic space, the formula is: I =LayerNorm(P I ·f ViT +b I );

[0155] in, The image projection matrix is ​​used to learn the nonlinear mapping of image features to medical semantic space through back propagation, for example, mapping the visual feature of "nuclear groove" to the dimension containing the semantics of "nuclear membrane irregularity"; is a bias vector that compensates for semantic offset in the projection process; LayerNorm is a layer normalization operation that adjusts the feature distribution to a mean of 0 and a variance of 1 to prevent gradient disappearance.

[0156] Furthermore, the formula of LayerNorm is:

[0157] Among them, μ, σ are the feature mean and variance, γ, β are the learnable scaling factors and offsets.

[0158] S312, constant dimension optimization of text features: 768-dimensional text feature vector f BERT Using residual connection optimization, the formula is: f T =LayerNorm(P T ·f BERT +f BERT )=LayerNorm((P T +I)·f BERT );

[0159] in, is the text projection matrix, which is added to the identity matrix I to form a residual connection, ensuring that the original semantics are preserved while optimizing the cross-modal mapping;

[0160] Specifically, the loss of text semantics caused by direct projection, such as the dilution of the specific semantics of "gritty" text, is avoided, and feature adaptability is enhanced through residual learning.

[0161] The image feature f I With the text feature f T They are all converted into 768-dimensional medical semantic vectors, and the correlation can be directly measured by cosine similarity. For example, the similarity between the text "nuclear groove image" and "irregular nuclear membrane" is increased from 0.5 to 0.8.

[0162] S32, Dynamic Contrast Learning: Figure 8 As shown in the figure, based on the projected features, triple contrast learning is used to strengthen the semantic association of positive sample pairs (matching images and text) and suppress the false association of negative sample pairs (mismatching images and text). An adaptive margin mechanism is introduced to dynamically adjust the optimization target according to the feature complexity.

[0163] Specifically, pathological images vary significantly in complexity. For example, the feature distinction between typical lesions and atypical hyperplasia is low. Traditional contrastive learning with a fixed margin struggles to balance the optimization goals for simple and complex cases. This invention leverages a feature complexity perception mechanism to dynamically adjust the margin value.

[0164] Traditional contrastive learning uses a fixed margin (e.g., 0.2), which cannot distinguish the alignment difficulty between simple cases (e.g., typical LSIL) and complex cases (e.g., atypical hyperplasia suspected of cancer). This paper uses a pathological feature complexity perception mechanism to achieve dynamic adjustment of the margin value. The specific steps are as follows:

[0165] S321. Sample pair construction:

[0166] Let the anchor sample be the image feature vector Positive samples are matching text feature vectors (Same case text), negative samples are random non-matching text feature vectors (across case texts and with no common terms with the anchor image).

[0167] Specifically, the positive sample pair is the image-text pair of the same case The text is required to contain at least two diagnostic terms related to the image, such as "image of cervical cells with nuclear grooves" and "irregular nuclear membrane";

[0168] Negative sample pairs are image-text pairs across cases The term index exclusion method is used to ensure that the term set of the negative sample text has no intersection with the positive sample. For example, the positive sample contains "sand body" and the negative sample only contains irrelevant terms such as "red blood cells". The batch size is set to 256 pairs / batch, training is 100 epochs, and the learning rate is 1e-4.

[0169] S322, dynamic margin calculation: according to the anchor image feature vector For example, the pathological complexity of nuclear morphological irregularities dynamically adjusts the contrast learning margin, and the formula is as follows:

[0170] in, is the pathology complexity weight vector, which is trained to learn the contribution of each dimension feature to the diagnostic difficulty. For example, the dimension weight corresponding to "nuclear-cytoplasmic ratio > 3:1" is higher. σ(·) is the Sigmoid function, which compresses the linear combination result to (0, 1) so that the margin value increases monotonically with the feature complexity. γ = 0.2 is the basic margin value, which ensures that the similarity gap between positive and negative sample pairs of simple cases is at least 0.2.

[0171] Among them, the pathology experts annotate the complexity level of the training set images (level 1-5, level 5 is the most difficult, such as microinvasive cancer), that is, the pathology complexity label c∈{1,2,3,4,5}, as the supervision signal training weight vector w m ;

[0172] Specifically, when image features include complex pathologies such as cellular atypia suspected of microinvasive carcinoma, As the value increases, the margin can dynamically increase to 0.30.4, forcing the model to more strictly distinguish difficult samples.

[0173] S323, Contrast loss function optimization: force the similarity of positive sample pairs to be higher than that of negative sample pairs by at least margin. The loss function formula is:

[0174] in, is the cosine similarity, with a value range of [1,1]. The larger the value, the stronger the semantic association. is the anchor image feature, is a positive sample (matching text), is a negative sample (does not match the text).

[0175] Optimization goal: force the similarity of positive sample pairs to be at least margin higher than that of negative sample pairs. For example, when the similarity of positive samples is 0.9, the similarity of negative samples needs to be ≤0.90.2=0.7 (simple cases) or ≤0.90.4=0.5 (complex cases).

[0176] The specific steps for training the dynamic margin model are as follows:

[0177] Input feature engineering: image features Combined with the complexity label c, Input the fully connected layer to predict the marginal value.

[0178] The model structure is:

[0179] in,

[0180] The difficult sample mining algorithm is as follows:

[0181] like It is judged as a difficult sample;

[0182] Retraining strategy: Collect all difficult samples every 10 epochs to form a difficult sample set, and train the contrastive learning module separately for 5 epochs.

[0183] Example calculation:

[0184] For microinvasive cancer images with complexity c = 5, the prediction margin = 0.4 requires that the similarity of the positive sample is at least 0.4 higher than that of the negative sample. For example, if the positive sample is 0.85, the negative sample must be ≤ 0.45.

[0185] S33. Introducing Image-Guided Text Attention (IGTA) in Contrastive Learning: Traditional multimodal models treat all terms in a text equally. However, in pathology diagnosis, only certain terms (such as "pessatomies" and "nuclear grooves") play a decisive role. Different terms in a pathology text contribute differently to diagnosis (for example, "pessatomies" have a higher specificity for thyroid cancer diagnosis than "cell hyperplasia"). By using image features to guide attention allocation to text features, semantic matching of key terms is strengthened.

[0186] Specifically, IGTA (Image Guided Text Attention) is a multimodal fusion technology that uses image information to guide attention allocation in text processing, enhancing the relevance of text understanding and images. This mechanism is particularly effective in tasks such as visual question answering (VQA), image description generation, and cross-modal retrieval.

[0187] IGTA uses image features to reversely guide text term weights, implementing a causal reasoning chain of "visual features → key terms → diagnostic conclusions", as described in the following steps:

[0188] S331, text term feature decomposition: the text feature vector f output by BERT T Decompose into sub-vectors of single tokens

[0189] Where N is the number of terms in the text, which is obtained through BERT's Token vector. For example, "papillary arrangement with psammoma" can be split into three term tokens.

[0190] S332, cross-modal attention weight calculation: through image feature f I With each Token sub-vector Calculate the attention weight, the formula is as follows:

[0191]

[0192] in, is the feature of the i-th term in the text, is the attention matrix, which learns the cross-modal association pattern between images and terms through training. For example, the matrix element value of the image feature of “nuclear depression” and the term “nuclear groove” is higher, α i is the attention weight of the i-th term, reflecting the “attention level” of the image to the term. For example, an image with papillary structure will make the α of “papillary arrangement” i Close to 0.8, while the α of "clear background" i Only 0.1;

[0193] S333, weighted text feature generation: W a Decomposed into 8 head matrices Each head captures different types of feature associations, such as the cell nucleus feature head and the cell arrangement feature head:

[0194]

[0195] in, is the output projection matrix.

[0196] Gate vector calculation: g = σ(W g ·fI +b g );

[0197] in, σ is the Sigmoid function, and the output vector element value is ∈(0,1));

[0198] Aggregate term features by attention weights:

[0199] Among them, the gated vector suppresses terms that are not related to the image, such as g corresponding to “inflammatory cell infiltration” when the image has no inflammatory features. i Close to 0.

[0200] The above formula is used to suppress the feature weights of background terms (such as "qualified specimens"), strengthen the semantic expression of diagnosis-related terms (such as "abnormal nuclear-cytoplasmic ratio"), and form a text representation that is highly aligned with the image features, that is, the output is the attention-weighted text feature

[0201] S34. Knowledge distillation: Expert experience injection, using reports generated by pathology experts (through LLM simulation) as "soft labels" to guide ClipRetrieval to learn implicit diagnostic logic and solve expert experience that is difficult to capture with pure data-driven models (such as "the number of nuclear grooves ≥3 indicates HSIL").

[0202] Purely data-driven models struggle to capture the implicit knowledge of pathologists (e.g., "a single nuclear groove is insufficient for diagnosing HSIL and requires nuclear enlargement"). LLM simulates expert report generation and encodes expert logic as "soft labels" to guide ClipRetrieval in learning complex diagnostic rules. The following steps explain this:

[0203] S341, teacher model output soft label: use the fine-tuned LLM (such as GPT4 pathology version) to input image feature f I , generating the term probability distribution p LLM (t|f I ), for example, for “thyroid cell images containing psammoma bodies”, the teacher model outputs the term “psammoma bodies” with a probability of 0.95 and “benign hyperplasia” with a probability of 0.05.

[0204] Pre-training data: GPT-4 was fine-tuned using 1 million PubMed pathology articles and 500,000 clinical reports to generate diagnostic descriptions consistent with the WHO Classification of Tumors.

[0205] Soft label generation: For each image, the teacher model outputs the top 5 high-probability terms and their probability values, such as "psammosome: 0.95, papillary arrangement: 0.88".

[0206] Convert the pathology report into "image feature → term probability" pairs and use Prompt engineering to guide LLM to generate structured outputs, such as:

[0207] Input image features: [dark nuclei, high nuclear-cytoplasmic ratio];

[0208] Output: {"Term":["LSIL","ASC-H","Inflammation"],"Probability":[0.7,0.25,0.05]}.

[0209] The LLM is fine-tuned using the cross entropy loss, as follows:

[0210]

[0211] Among them, y true Probability of terms annotated by experts.

[0212] S342, Student model output distribution:

[0213] Adopt temperature annealing strategy:

[0214] Initial training (0-30 epochs): τ = 1.0, encouraging exploration of multi-term associations;

[0215] Late training (30-100 epochs): τ = 0.1, forcing focus on high-probability terms;

[0216] Clip Retrieval generates term probability distribution p through softmax Student (t|f I ), calculated by softmax formula:

[0217]

[0218] Among them, τ = 0.1 is the temperature parameter. Lowering the temperature makes the probability distribution sharper, forcing the student model to focus on highly correlated terms (e.g., the probability of “sand body” increases from 0.8 to 0.98). i is the i-th term, such as "samme" and "ground glass core".

[0219] S343, knowledge distillation loss function: The KL divergence is used to measure the distribution difference between the student and teacher models. The formula is:

[0220]

[0221] If the teacher model's probability for "psamoid body" is 0.95, while the student model's is only 0.7, the KL divergence increases, driving the model to adjust parameters to increase the matching probability of the term, thereby learning the expert association rule of "psamoid body → thyroid cancer".

[0222] S35. Joint optimization objective function: integrating contrastive learning loss L and knowledge distillation loss L KD , forming an end-to-end optimization goal to ensure that semantic alignment is consistent with data distribution and incorporates expert experience. Contrastive learning ensures that the model learns basic semantic associations from the data, and knowledge distillation injects expert experience. The joint optimization of the two enables the model to have both data fitting capabilities and domain logic interpretability.

[0223] Dynamic adjustment of loss function weights

[0224] Early stage (0-50 epochs): λ = 0.3, focusing on contrastive learning to fit the data distribution;

[0225] Later stage (50-100 epochs): λ = 0.7, strengthening knowledge distillation and incorporating expert experience;

[0226] The dynamic weight function is calculated as follows:

[0227] Linear interpolation formula:

[0228]

[0229] Among them, λ min =0.3,λ max =0.7, total number of training rounds T total =100;

[0230] Specifically, the early stage focuses on data fitting, that is, contrastive learning is dominant, and the later stage focuses on knowledge calibration, that is, distillation loss is dominant, to ensure that the model first learns basic associations and then integrates expert logic.

[0231] Total loss function:

[0232] L total =L+λ(t)·L KD ;

[0233] Where λ(t) is a linear function that changes with the number of training rounds t, and λ = 0.5 is a balance coefficient. Experimental verification shows that this value can achieve the best balance between term accuracy (95%) and expert rule compliance (92%).

[0234] Optimization strategy: Use the AdamW optimizer with an initial learning rate of 1e-4, decaying by 0.9 every 10 epochs to prevent overfitting; set the batch size to 256 pairs, train for 100 epochs, and monitor the loss function until it converges to a stable value (≤0.5) to ensure that the model achieves semantic alignment in a unified space.

[0235] Output generates aligned multimodal feature vector (splicing into 1536 dimensions) can be directly input into LLM to generate a diagnosis report, ensuring the semantic consistency of "image feature terms describing the diagnosis conclusion". For example, the "nuclear groove" feature extracted by ViT triggers LLM to generate the description "irregular nuclear membrane, visible nuclear groove".

[0236] Technical verification: dynamic alignment vs. static alignment is shown in Table 2 below:

[0237] Table 2

[0238]

[0239] Conclusion: Through multi-level technological innovations such as dynamic projection, adaptive contrast, attention enhancement, and knowledge injection, this step has achieved a leap from "coarse-grained matching" to "fine-grained reasoning" in cross-modal semantic alignment, providing a reliable feature foundation for subsequent LLM to generate accurate reports.

[0240] In this embodiment, Figure 5 As shown, S40, LLM vertical classification fine-tuning and diagnosis report generation: According to the aligned multimodal features, the aligned multimodal feature vector (joined into a 1536-dimensional vector) to generate a structured pathology report that complies with the TBS diagnostic criteria;

[0241] Input prompt: "Image features: [ViT feature description], text keywords: [BERT keywords], please generate a standard diagnosis report";

[0242] Output constraints: The report must contain a three-part structure consisting of "microscope findings", "diagnostic opinions", and "suggestions".

[0243] Enhanced explainability: Through knowledge base queries (such as "psammomatous bodies → characteristic structures of papillary thyroid carcinoma"), hyperlinked explanations are inserted into the report, and doctors can view detailed pathological mechanisms after clicking them.

[0244] Misuse of terminology: reduced from 15.7% to 3.2% on a test set of 100,000 examples;

[0245] Report generation time: average 2.8 minutes, compared to 15 minutes for manual reporting;

[0246] Doctor review efficiency: The key feature annotation completeness rate increased from 60% to 92%, and review time was reduced by 60%.

[0247] Example 2

[0248] By combining quantitative data with qualitative cases, we verified whether the Clip-Retrieval model in Example 1 significantly outperformed the traditional CLIP model in semantic alignment of complex pathological features and anti-false positive interference in the thyroid cell pathology scenario. The results were verified by thyroid cell semantic alignment as follows:

[0249] Test samples: 500 thyroid fine needle aspiration (FNA) smear images (including 200 cases of papillary carcinoma and 300 cases of benign hyperplasia);

[0250] Case type:

[0251] Papillary carcinoma (PTC): 200 cases (including features such as psammoma bodies and papillary arrangement)

[0252] Benign hyperplasia (such as nodular goiter): 300 cases (including follicular cell clusters, colloid background, etc.)

[0253] Evaluation Metrics:

[0254] Semantic similarity: the semantic match between the manually annotated image and the text keywords (range: 0-100%, ≥90% is a strong match)

[0255] False positive rate: The proportion of benign samples that are incorrectly matched to malignant keywords (such as "papillary arrangement" and "sand-gravel body"). The model comparison data is shown in Tables 3 and 4 below:

[0256] Table 3 Detailed table of 20 samples

[0257]

[0258]

[0259] Table 4 Comparison of statistical results

[0260]

[0261] In this embodiment, through the above Tables 3 and 4, such as in the positive cases of THY-001 to THY-005, in the papillary carcinoma samples, the similarity of the model of this embodiment for key terms such as "papillary arrangement" and "sandstone body" is ≥91%, which is a significant improvement over the traditional CLIP (72-78%), proving that it can accurately capture the visual-semantic association of malignant features.

[0262] In the combined feature advantages of THY-011 and THY-015, when the image contains "papillary arrangement + psammoma" at the same time, the model similarity of this embodiment reaches 95% (traditional CLIP is only 77.4%), indicating that the dynamic semantic alignment mechanism can effectively handle multi-feature collaborative diagnosis scenarios.

[0263] In mislabeled cases such as THY-009, THY-010, and THY-014, the false positive matching rate of traditional CLIP for malignant terms such as "papillary arrangement" and "pessavirus body" in benign samples is as high as over 80% (such as the similarity of THY-009 is 82.1%). However, the model in this embodiment suppresses irrelevant features through dynamic attention, reducing the similarity to below 45%, directly avoiding benign cases from being misclassified as malignant.

[0264] For benign features such as "follicular cell clusters" and "glial background", the average similarity of the model in this embodiment reached 87.1% (traditional CLIP 71.5%), indicating that while accurately identifying malignant features, the semantic modeling capability of benign features was not sacrificed, ensuring the comprehensiveness of the diagnostic logic.

[0265] The model experimental results of this embodiment are better than traditional methods, and the semantic alignment accuracy is significantly improved. By comparing the data in Table 3 and Table 4, the average similarity of papillary carcinoma samples increased from 76.2% in traditional CLIP to 92.5%, an increase of 21.4% (P < 0.001), which is close to the manual annotation level of pathology experts (95%+).

[0266] Traditional CLIP relies on a static embedding space and cannot dynamically distinguish the subtle morphological differences between "papillary arrangement" and "follicular cell clusters." This embodiment uses dynamic contrast learning and image-guided attention (IGTA) to strengthen the fine-grained association between cell nucleus arrangement patterns and terms, such as the high-weight match between cell cluster arrangement and "papillary arrangement" in THY-001.

[0267] In this embodiment, the false positive rate was significantly reduced, and the proportion of benign samples mismatched to "papillary arrangement" was reduced from 15% (traditional CLIP) to 4%, which means that the number of misjudgments was reduced from 15 to 4 for every 100 benign cases, directly reducing the risk of overdiagnosis by 73.3%.

[0268] Traditional models, lacking a pathological feature enhancement module, can easily misclassify the "clustered arrangement" of benign follicular cell clusters as a malignant "papillary arrangement." This embodiment significantly improves feature discrimination through the ViT pathological feature enhancement module, which strengthens cell nucleus features and knowledge distillation after the last two encoder layers. This involves incorporating the expert rule that "papillary arrangement must be accompanied by a fibrovascular axis."

[0269] In malignant samples containing multiple features such as "nuclear grooves + psammoma bodies," such as THY-015, the model in this embodiment has a similarity of 93.8%, while the traditional CLIP model has a similarity of only 76.7%, demonstrating its robustness in scenarios with complex pathological feature combinations and meeting the requirements of "multi-feature comprehensive judgment" in clinical diagnosis.

[0270] Example 3

[0271] Further illustrate with reference to Example 1, Figure 9 The structure shown, Figure 9 This is a schematic diagram of the structure of a computer device / equipment / system provided in an embodiment of the present application. The computer device / equipment / system includes:

[0272] A processor, memory, a communication bus, and a computer program stored in the memory and executable on the processor.

[0273] The processor can call a computer program in the memory and implement the intelligent cytopathology auxiliary method based on the Clip-Retrieval model provided in the above embodiment when executing the program. The method includes: S10, obtaining data of cervical cell pathology images and pathology texts, and constructing a structured data set of image-text pairs; S20, preprocessing the cervical cell pathology images, and generating corresponding feature vectors according to the preprocessed cervical cell pathology images and pathology texts; S30, constructing a unified medical semantic embedding space, aligning the feature vectors of the image-text pairs through the dynamic projection matrix P, and outputting the alignment features; S40, inputting the alignment features into the LLM model to generate a structured pathology report.

[0274] Furthermore, the computer device / equipment / system also includes:

[0275] Communications Interface: used for communication between memory and processor.

[0276] The memory may include a high-speed RAM memory and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0277] If the memory, processor, and communication interface are implemented independently, the communication interface, memory, and processor can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0278] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0279] A processor may include one or more processing units. For example, a processor may include an application processor (AP), an application-specific integrated circuit (ASIC), a modem processor, a graphics processing unit (CPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may be a neural center or command center. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory may store instructions or data that have just been used or are being recycled by the processor. If the processor needs to use the instruction or data again, it can directly call it from the memory. It avoids repeated access and reduces the waiting time of the processor, thus improving the efficiency of the system.

[0280] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0281] A display device is used to display images, videos, etc. The display device may include a display panel, which may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), MiniLED, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0282] Optionally, in a specific implementation, if the memory, processor, and communication interface are integrated on a chip, the memory, processor, and communication interface can communicate with each other through an internal interface.

[0283] On the other hand, an embodiment of the present application also provides a computer non-transitory readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned intelligent cytopathology assisted method based on the Clip-Retrieval model, the method comprising: S10, acquiring data of cervical cell pathology images and pathology texts, and constructing a structured data set of image-text pairs; S20, preprocessing the cervical cell pathology images, and generating corresponding feature vectors according to the preprocessed cervical cell pathology images and pathology texts; S30, constructing a unified medical semantic embedding space, aligning the feature vectors of the image-text pairs through a dynamic projection matrix P, and outputting alignment features; S40, inputting the alignment features into the LLM model to generate a structured pathology report.

[0284] On the other hand, an embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. The computer program can run computer instructions. When the computer program is executed by a processor, the computer can execute the intelligent cytopathology assisted method based on the Clip-Retrieval model provided by the above methods. The method includes: S10, obtaining data of cervical cell pathology images and pathology texts, and constructing a structured data set of image-text pairs; S20, preprocessing the cervical cell pathology images, and generating corresponding feature vectors according to the preprocessed cervical cell pathology images and pathology texts; S30, constructing a unified medical semantic embedding space, aligning the feature vectors of the image-text pairs through the dynamic projection matrix P, and outputting the alignment features; S40, inputting the alignment features into the LLM model to generate a structured pathology report.

[0285] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0286] For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use with or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, the computer-readable medium can even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting, or otherwise processing it in a suitable manner as needed, and then storing it in a computer memory.

[0287] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0288] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0289] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0290] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0291] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0292] The above specific embodiments do not constitute a limitation on the scope of protection of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limitations of the present application. Those skilled in the art can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A cytopathology intelligent auxiliary method based on Clip-Retrieval model, characterized in that: The following steps are involved: S10, obtaining data of cervical cell pathology images and pathology texts, and constructing a structured dataset of image-text pairs; S20, preprocessing the cervical cell pathology image, and generating corresponding feature vectors according to the preprocessed cervical cell pathology image and pathology text; S30, constructing a unified medical semantic embedding space, aligning the feature vectors of the image-text pair through the dynamic projection matrix P, and outputting the aligned features; S40: Input the alignment features into the LLM model to generate a structured pathology report.

2. The intelligent assisted cytopathology method based on the Clip-Retrieval model according to claim 1, characterized in that: In step S10 , the structured data includes a cervical cell pathology image in a specific format, a pathology text corresponding to the cervical cell pathology image, and an annotation label.

3. The intelligent assisted cytopathology method based on the Clip-Retrieval model according to claim 2, characterized in that: The specific steps of generating the feature vector of the image-text pair in step S20 are as follows: S21. Based on the ViT model, the preprocessed cervical cell pathology image is divided into image blocks of a specific size, mapped into embedding vectors through a linear projection layer, and a learnable position encoding is added. The cell morphology is extracted through a multi-layer Transformer encoder to generate a 256-dimensional image feature vector f. ViT ; S22. Input the key terms in the pathology text into the BERT model pre-trained in the medical field. After the input pathology text is segmented by WordPiece, a word embedding vector is generated. The dynamic representation of words in the context is learned through the Transformer encoder, and the context-aware 768-dimensional text feature vector f is extracted. BERT .

4. The intelligent assisted cytopathology method based on the Clip-Retrieval model according to claim 1, characterized in that: The specific steps in step S30 are as follows: S31, through the learnable matrix and The 256-dimensional image feature vector f output by the ViT model ViT The 768-dimensional text feature vector f output by the BERT model BERT Mapped to 768-dimensional medical semantic space, generating image-text pairs (f I ,f T ); S32, based on the projected image-text pair (f I ,f T ), optimize the semantic space by contrastively learning the triples of the loss function L, so that the similarity of matching pairs is higher than that of non-matching pairs; S33. Introduce IGTA into the contrastive learning of step S32 to guide the attention allocation of text features based on image features and strengthen the semantic matching of key terms. S34, input image feature f I To the fine-tuned LLM model, generate the term probability distribution p LLM (t|f I ); Clip Retrieval generates term probability distribution p through softmax Student (t|f I ); Through the knowledge distillation loss function L KD The divergence measures the probability distribution of terms p Student (t|f I ) and the term probability distribution p LLM (t|f I ) distribution difference; S35. Combine contrastive learning loss L with knowledge distillation loss L KD Integration, make f I With f T Semantically align in a unified space to generate aligned multimodal feature vectors 5. The intelligent assisted cytopathology method based on the Clip-Retrieval model according to claim 4, characterized in that: In step S31, the projection matrix can be learned The 256-dimensional image feature vector f ViT Mapping to the 768-dimensional medical semantic space, the formula is: f I =LayerNorm(P I ·f ViT +b I ); in, is the image projection matrix, and the nonlinear mapping from image features to medical semantic space is learned through back propagation. is the bias vector that compensates for the semantic offset in the projection process. LayerNorm is the layer normalization operation that adjusts the feature distribution to a mean of 0 and a variance of 1. For the 768-dimensional text feature vector f BERT Using residual connection optimization, the formula is: f T =LayerNorm(P T ·f BERT +f BERT )=LayerNorm((P T +I)·f BERT ); in, is the text projection matrix, which is added to the identity matrix I to form a residual connection, preserving the original semantics while optimizing the cross-modal mapping.

6. The intelligent assisted cytopathology method based on the Clip-Retrieval model according to claim 4, characterized in that: The specific steps of step S32 are as follows: S321, let triple and pathological complexity label c∈{1,2,3,4,5}; Among them, the triplet includes anchor samples, positive samples and negative samples. The anchor sample is the image feature vector Positive samples are matching text feature vectors Negative samples are random non-matching text feature vectors S322, based on the anchor image feature vector Dynamically adjust the comparative learning margin value based on the pathological complexity; S323, forcing the similarity of positive sample pairs to be higher than that of negative sample pairs by at least margin, maximizing the similarity between the anchor image and the matching text, and minimizing the similarity with the non-matching text, the contrast loss function formula is: in, is the cosine similarity, with a value range of [1,1]; When the average similarity of positive sample pairs is ≥ 0.9 and that of negative sample pairs is ≤ 0.5, the contrastive learning training is stopped.

7. The intelligent assisted cytopathology method based on the Clip-Retrieval model according to claim 1, characterized in that: The specific steps of step S32 are as follows: S331, the text feature vector f output by BERT T Decompose into sub-vectors of single tokens Where N is the number of terms in the text, obtained through BERT's Token vector; S332, through image feature f I With each Token sub-vector Calculate attention weights; S333. Aggregate term features by attention weight. The formula is: Among them, α i is the attention weight of the i-th term, is the feature of the i-th term in the text, and g is the gating vector.

8. A computer device / apparatus / system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the computer program to implement the steps of the intelligent cytopathology assisted method based on the Clip-Retrieval model according to any one of claims 1 to 5.

9. A computer non-transitory readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the intelligent cytopathology assisted method based on the Clip-Retrieval model according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by one or more processors, the steps of the intelligent assisted method for cytopathology based on the Clip-Retrieval model according to any one of claims 1 to 5 are implemented.

Images