Pathological image cancer cell detection optimization method and system based on retrieval enhancement

Abstract

The invention discloses a pathological image cancer cell detection optimization method and system based on retrieval enhancement, and belongs to the technical field of image detection, and the method comprises the steps: carrying out the preliminary detection of an input histopathology image through a target detection model based on a Transform architecture; and further detecting the preliminary detection result based on a retrieval enhanced optimization mechanism to obtain a final detection result. The precision of the cancer cells in the pathological image is improved.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and more specifically to an optimized method and system for detecting cancer cells in pathological images based on retrieval enhancement. Background Technology

[0002] Histopathological examination is the "gold standard" for cancer diagnosis, and the analysis of pathological images under a microscope is crucial for the identification and localization of cancer cells. With the popularization of digital pathology technology, the application of whole-slide images (WSIs) is becoming increasingly widespread, making AI-based automated analysis technology possible. However, the complexity of pathological images and the heterogeneity of cancer cell manifestations pose significant challenges to automated detection systems.

[0003] Currently, deep learning-based object detection models (such as Faster R-CNN, YOLO series, and Transformer-based detectors) are widely used for cancer cell detection in pathological images. These models, trained on large amounts of labeled data, can initially locate suspicious cancer cell regions from complex tissue backgrounds. However, these methods have a significant drawback: due to the complexity of tissue structures, the variability of cell morphology, and staining differences, the models generate a large number of false positives during inference. That is, the models misclassify many normal cellular structures (such as inflammatory cells and stromal cells) as cancer cells, which seriously affects the reliability and clinical applicability of the detection system.

[0004] Therefore, improving the accuracy and reliability of cancer cell detection in pathological images and reducing false positive results are problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an optimized method and system for detecting cancer cells in pathological images based on retrieval enhancement, which at least solves some of the problems in the background art. To achieve the above objectives, the present invention adopts the following technical solution: This invention discloses an optimized method for detecting cancer cells in pathological images based on retrieval enhancement, comprising the following steps: S1. The target detection model based on the Transformer architecture is used to perform preliminary detection on the input histopathological image to obtain candidate image patches containing false positive data; S2. Obtain a reference dataset containing known true positive image patches and normal image patches, extract features from the reference dataset, and embed the extracted features and corresponding labels to form a retrieval database; S3. Perform clustering operations on the feature embeddings in the retrieval database, and select at least one representative image patch in each cluster category; input the representative image patch into the first visual language model to obtain the corresponding key feature text description, as dynamic domain knowledge; S4. Extract the feature embeddings from the candidate image blocks and compare them with all feature embeddings in the retrieval database to obtain the Top-N true positive data image blocks and the Top-M normal data image blocks that are most similar to the candidate image blocks; S5. Construct a multimodal cue template, including: candidate image patches to be classified, Top-N true positive data image patches, Top-M normal data image patches, and text cue containing dynamic domain knowledge and classification instructions; S6. Input the multimodal cue template constructed in S5 into the second visual language model. Through context learning, analyze the candidate image blocks, N true positive data image blocks and M normal data image blocks. Combine dynamic domain knowledge and classification instructions to output the classification results of the candidate image blocks to be classified.

[0006] Once the classification results are obtained, candidate image blocks that are classified as normal data image blocks can be removed based on the output classification results to obtain the final optimized cancer cell detection results.

[0007] Furthermore, in S1, the object detection model based on the Transformer architecture includes the RF-DETR model.

[0008] Further, in S2, feature extraction is performed on the reference dataset, specifically including: We used a biomedical vision-language pre-trained model based on the CLIP architecture to extract features from known true positive image patches and normal image patches in a reference dataset.

[0009] Furthermore, the biomedical vision-language pre-trained model based on the CLIP architecture includes the BioMedCLIP model.

[0010] Furthermore, in S3, the first visual language model includes the Gemini 2.5 Pro model.

[0011] Furthermore, in S6, the second visual language model includes the InternVL3.5 model.

[0012] Another aspect of the present invention discloses a retrieval-enhanced pathological image cancer cell detection optimization system, comprising: Candidate Detection Module: This module uses a Transformer-based target detection model to perform preliminary detection on the input histopathological images, obtaining candidate images that contain false positive data. Feature extraction module: used to extract feature embeddings from the reference dataset and candidate images; Storage: Used to embed the features of each image patch in the reference dataset and the corresponding labels to form a retrieval database and store them; The retrieval module is used to compare the feature embeddings of candidate image patches with the feature embeddings of all the feature embeddings in the retrieval database to obtain the Top-N most similar true positive data image patches and the Top-M normal data image patches. Dynamic domain knowledge acquisition module: performs clustering operation on the feature embeddings in the retrieval database, and selects at least one representative image patch in each cluster category; and inputs the representative image patch into the first visual language model to obtain the corresponding key feature text description, as dynamic domain knowledge; Multimodal cue template construction module: Constructs a multimodal cue template, including: candidate image patches to be classified, Top-N true positive data image patches, Top-M normal data image patches, and text cue containing dynamic domain knowledge and classification instructions; Visual language classification module: The constructed multimodal cue template is input into the second visual language model. Through context learning, it analyzes candidate image patches, N true positive data image patches and M normal data image patches, and combines dynamic domain knowledge and classification instructions to output the classification results of the candidate image patches to be classified.

[0013] Preferably, the feature extraction module utilizes a biomedical vision-language pre-trained model based on the CLIP architecture to extract features from known true positive image patches and normal image patches in the reference dataset.

[0014] Preferably, in the dynamic domain knowledge acquisition module, the first visual language model includes the Gemini 2.5 Pro model.

[0015] Preferably, in the visual language classification module, the second visual language model includes the InternVL3.5 model.

[0016] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses an optimized method and system for detecting cancer cells in pathological images based on retrieval enhancement, which has the following beneficial effects: 1) Significantly improves detection accuracy and greatly reduces false positives: This invention employs a two-stage optimization framework for the detection of cancer cells in pathological images. In particular, by using retrieval enhancement and intelligent agent (VLM) for contextual analogy reasoning, it can effectively identify and filter out difficult false positive samples that are highly similar in morphology to cancer cells and are difficult to distinguish using existing technologies.

[0017] 2) Provides inherent explainability, enhancing trustworthiness: The decision-making process of the intelligent agent (VLM) of this invention is transparent. Its classification decisions are based on direct comparisons between candidate image patches and retrieved known "cancer" (positive examples) and "normal" (negative examples) samples. This "analogical reasoning" process (supplemented by "thought chain" instructions and "classification rationale" outputs) allows clinicians or researchers to review and understand the model's decision-making basis, overcoming the poor interpretability of existing "black box" models and greatly enhancing the system's reliability in clinical applications.

[0018] 3) Automatic knowledge discovery and dynamic enhancement: Unlike existing technologies that rely on manually defined, static, and unchanging "domain knowledge," this invention includes a knowledge generation module. This module can automatically discover and summarize key morphological features from training data by clustering data and automatically generating "dynamic visual descriptions."

[0019] 4) Framework flexibility and scalability: The modular design of this invention allows for independent upgrades of components such as detection, retrieval, knowledge generation, and Virtual Model (VLM). In the future, a more powerful base model or VLM can be used to replace the corresponding module without retraining the entire detection process, demonstrating excellent technological iterability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the overall process of the method provided in the embodiment of the present invention.

[0022] Figure 2 This is a system module connection diagram provided for an embodiment of the present invention.

[0023] Figure 3 A flowchart illustrating the generation process of dynamic visual descriptions provided in embodiments of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention discloses an optimized method for detecting cancer cells in pathological images based on retrieval enhancement, including dataset construction, first-stage detection, construction of an offline retrieval library and knowledge base, and second-stage online optimization. The overall implementation steps of this method are as follows: Figure 1 The methods disclosed in the embodiments of the present invention can be implemented as follows: Figure 2 The system implementation described above.

[0026] The process described in the embodiments of the present invention will be further explained in detail below.

[0027] First, a dataset was constructed. In this embodiment of the invention, the dataset construction and annotation used a dataset containing 1003 pancreatic cancer histopathological images. In practice, to ensure the reliability of the dataset and the accuracy of the training benchmark, a strict "two-label, one-review" annotation process was adopted: Independent Initial Labeling: First, two pathologists independently examined the digital pathology images and used bounding boxes to select all identifiable cancer cells. Expert Arbitration: Subsequently, a senior pathologist summarized, reviewed, and arbitrated the annotation results of the two doctors, resolving any inconsistencies or disputes.

[0028] The gold standard was established: the final undisputed annotations were designated as the "gold standard". The dataset was randomly divided into a training set (802 images), a validation set (99 images), and a test set (102 images) in an 8:1:1 ratio.

[0029] The candidate detection model for the first stage of detection is the RF-DETR model based on the Transformer architecture (based on the Deformable DETR architecture and integrated with the DINOv2 pre-trained backbone network). The training process uses rf-detr-base pre-trained weights and fine-tunes them on the training set. The input pancreatic cancer histopathological images are resized to 560 x 560 pixels. The AdamW optimizer is used with a learning rate of 1x10. -4 The training process was performed for 150 epochs with an effective batch size of 16. The best-performing model on the validation set was then used as the pre-trained candidate detection model (initial detection model). During testing, a low confidence threshold (e.g., 0.10) was used to obtain candidate cell bounding boxes with high recall, ensuring that no potential cancer cells were missed.

[0030] The construction of the retrieval database, which is the offline preparation stage I, involves building a feature-embedded retrieval database to support subsequent retrieval and knowledge generation. The specific steps are as follows: Sample selection: Select high-quality cell image patches that are clearly labeled as "cancer" and "normal" from the training set data.

[0031] Feature extraction and database construction: Using the BioMedCLIP pre-trained model for the biomedical field based on the CLIP architecture, high-dimensional feature embeddings are extracted for each of the above image patches. These (feature embeddings, category labels, original images) are stored as key-value pairs to construct a retrieval database.

[0032] The generation of dynamic visual descriptions, this process is the offline preparation stage II, see reference. Figure 3 Based on the retrieval database built in the offline preparation phase I, data-driven and precise morphological domain knowledge is further generated. The specific steps are as follows: Feature clustering: BioMedCLIP feature embeddings from the retrieved database were used directly, and the K-Means algorithm was used to cluster "cancer" and "normal" samples separately. To capture morphological diversity, the number of clusters was set to K=10 (i.e., cancer and normal samples were each clustered into 10 classes).

[0033] Representative Sample Selection: Calculate the center of each cluster and select the image patch closest to the center as the "representative sample" for that morphological category. Visual Description Generation: Input the selected representative sample into the first visual language model (in this embodiment, the first visual language model uses the multimodal large model Gemini 2.5 Pro). The model is prompted to describe the cell morphological features in the image patch, such as nuclear size, shape, staining intensity, nucleolar visibility, and nucleoplasm-to-cytoplasm ratio.

[0034] Knowledge base storage: The 20 sets of text descriptions generated by the first visual language model Gemini 2.5 Pro are mapped to the corresponding cluster IDs and stored as a "dynamic visual description library".

[0035] The second phase involves online optimization, specifically machine-based retrieval enhancement optimization (online inference phase). a. Candidate Processing and Retrieval: For each candidate image patch to be classified generated in the first stage, its feature embedding is extracted using the BioMedCLIP model. A k-nearest neighbor search (based on embedding distance) is performed to retrieve the top-3 most similar "cancer" image patches and the top-3 most similar "normal" image patches from the retrieval database constructed in step 3.

[0036] b. Intelligent Agent (VLM) Classification: The large-scale visual language model "InternVL3.5" is used as the intelligent agent. Construction Hints: A multimodal input is created containing seven image patches: a candidate image, the top-3 "cancer" image patches, the top-3 normal image patches, etc., and an augmented text. The input to the second visual language model includes: the target candidate image patch to be classified; three retrieved "cancer" samples (as positive examples); three retrieved "normal" samples (as negative examples); and augmented text hints, including a "dynamic visual description" (corresponding to the cluster category of the retrieved sample) retrieved from the knowledge base in step 4, and a "thought chain" instruction, instructing the model to first progressively compare the morphological differences between the target image and the positive and negative examples, then combine the dynamic description for reasoning, and finally output the classification result.

[0037] Then, based on the multimodal cue template, the second visual language model analyzes the contextual cue and outputs the final classification ("cancer" or "normal") and the reason for the classification.

[0038] Experimental verification shows that this invention significantly improves the accuracy of sample detection through the above method, achieving a recall rate of 0.89 and a precision of 0.36 at a confidence threshold of 0.10. Furthermore, the process provides verifiable decision-making evidence, as the visual language model can provide the retrieved samples and the reasoning behind them, greatly enhancing the clinical reliability of the system.

[0039] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0040] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An optimized method for detecting cancer cells in pathological images based on retrieval enhancement, characterized in that, Includes the following steps: S1. The target detection model based on the Transformer architecture is used to perform preliminary detection on the input histopathological image to obtain candidate image patches containing false positive data; S2. Obtain a reference dataset containing known true positive image patches and normal image patches, extract features from the reference dataset, and embed the extracted features and corresponding labels to form a retrieval database; S3. Perform clustering operations on the feature embeddings in the retrieval database, and select at least one representative image patch in each cluster category; input the representative image patch into the first visual language model to obtain the corresponding key feature text description, as dynamic domain knowledge; S4. Extract the feature embeddings from the candidate image blocks and compare them with all feature embeddings in the retrieval database to obtain the Top-N true positive data image blocks and the Top-M normal data image blocks that are most similar to the candidate image blocks; S5. Construct a multimodal cue template, including: candidate image patches to be classified, Top-N true positive data image patches, Top-M normal data image patches, and text cue containing dynamic domain knowledge and classification instructions; S6. Input the multimodal cue template constructed in S5 into the second visual language model. Through context learning, analyze the candidate image blocks, N true positive data image blocks and M normal data image blocks. Combine dynamic domain knowledge and classification instructions to output the classification results of the candidate image blocks to be classified.

2. The method according to claim 1, characterized in that, In S1, the object detection model based on the Transformer architecture includes the RF-DETR model.

3. The method according to claim 1, characterized in that, In S2, feature extraction is performed on the reference dataset, specifically including: We used a biomedical vision-language pre-trained model based on the CLIP architecture to extract features from known true positive image patches and normal image patches in a reference dataset.

4. The method according to claim 3, characterized in that, The biomedical visual-language pre-trained model based on the CLIP architecture includes the BioMedCLIP model.

5. The method according to claim 1, characterized in that, In S3, the first visual language model includes the Gemini2.5 Pro model.

6. The method according to claim 1, characterized in that, In S6, the second visual language model includes the InternVL3.5 model.

7. A search-enhanced pathological image cancer cell detection optimization system, characterized in that, include: Candidate Detection Module: This module uses a Transformer-based target detection model to perform preliminary detection on the input histopathological images, obtaining candidate images that contain false positive data. Feature extraction module: used to extract feature embeddings from the reference dataset and candidate images; Storage: Used to embed the features of each image patch in the reference dataset and the corresponding labels to form a retrieval database and store them; The retrieval module is used to compare the feature embeddings of candidate image patches with the feature embeddings of all the feature embeddings in the retrieval database to obtain the Top-N most similar true positive data image patches and the Top-M normal data image patches. Dynamic domain knowledge acquisition module: performs clustering operation on the feature embeddings in the retrieval database, and selects at least one representative image patch in each cluster category; and inputs the representative image patch into the first visual language model to obtain the corresponding key feature text description, as dynamic domain knowledge; Multimodal cue template construction module: Constructs a multimodal cue template, including: candidate image patches to be classified, Top-N true positive data image patches, Top-M normal data image patches, and text cue containing dynamic domain knowledge and classification instructions; Visual language classification module: The constructed multimodal cue template is input into the second visual language model. Through context learning, it analyzes candidate image patches, N true positive data image patches and M normal data image patches, and combines dynamic domain knowledge and classification instructions to output the classification results of the candidate image patches to be classified.

8. The system according to claim 7, characterized in that, The feature extraction module utilizes a biomedical vision-language pre-trained model based on the CLIP architecture to extract features from known true positive image patches and normal image patches in the reference dataset.

9. The system according to claim 7, characterized in that, In the dynamic domain knowledge acquisition module, the first visual language model includes the Gemini 2.5 Pro model.

10. The system according to claim 7, characterized in that, In the visual language classification module, the second visual language model includes the InternVL3.5 model.

Images