Cell image recognition method and device based on large language model

Abstract

The application discloses a cell image recognition method and device based on a large language model. When detecting a cervical cytology whole section image to be recognized, if positive cells are detected, the corresponding positive cells are further recognized by a preset recognition model to determine whether the corresponding positive cells are positive; and the recognition result of the cervical cytology whole section image to be recognized is determined according to the recognition result output by the preset recognition model; a preset large language model in the preset recognition model combines image features and text features to generate an explanatory text conforming to pathological logic as a recognition result; the method can improve the recognition accuracy of the cell image, and thus can improve the efficiency and reliability of early screening of cervical cancer.

Description

Technical Field

[0001] This application relates to the field of medical cell image processing technology, specifically to a cell image recognition method and apparatus based on a large language model. Background Technology

[0002] Cervical cancer is a serious disease that severely threatens women's health. Its early screening and diagnosis mainly rely on pathologists' manual interpretation of whole-slide cytology (WSI) images. However, traditional manual interpretation methods have significant limitations: First, they are inefficient, requiring doctors to spend a lot of time examining a large number of cells, resulting in long screening cycles and high costs; second, they are highly subjective, with diagnostic results easily influenced by the doctor's experience and condition, leading to the risk of false negatives and false positives; third, they have poor reproducibility, as different doctors or the same doctor interpreting the same image at different times may produce differences.

[0003] To address these challenges, artificial intelligence-based computer-aided diagnostic (AI-CAD) technology has emerged. While computer-aided automated diagnostic models can provide a powerful supplement to pathologists, their effectiveness is limited by the scarcity of large amounts of detailed labeled data, and their interpretability and robustness are also insufficient. These factors severely hinder their practical application and reliability in clinical settings.

[0004] Therefore, there is an urgent need for a recognition method that can improve the accuracy of cell image recognition, thereby enhancing the efficiency and reliability of early cervical cancer screening. Summary of the Invention

[0005] In view of the above, this application provides a cell image recognition method and apparatus based on a large language model, which can improve the accuracy of cell image recognition and thus improve the efficiency and reliability of early cervical cancer screening.

[0006] To solve the above-mentioned technical problems, the technical solution of this application is implemented as follows:

[0007] In one embodiment, a cell image recognition method based on a large language model is provided, the method comprising:

[0008] Obtain whole-section images of cervical cytology to be identified;

[0009] Cell detection is performed to obtain the location, classification, and confidence information of cells in the whole cervical cytology slide image to be identified;

[0010] In response to the determination of positive cells based on the classification information, a single cell image is cropped from the whole cervical cytology slide image to be identified based on the location information corresponding to the positive cell, and a preset number of single cell images are selected based on the confidence information.

[0011] The user command and the selected single cell image are input into a preset recognition model to obtain the recognition results corresponding to all the single cell images output; wherein, the preset recognition model includes a preset image encoder, a preset text encoder, a preset mapper and a preset large language model; the preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result;

[0012] The identification result corresponding to the whole cervical cytology slide image to be identified is determined based on the identification results corresponding to all the individual cell images.

[0013] The acquisition of the preset image encoder includes:

[0014] Establish the initial image encoder;

[0015] Generate the first training sample set and the second training sample set;

[0016] The initial image encoder is pre-trained using the first training sample set to obtain a pre-trained image encoder.

[0017] Using the pre-trained image encoder as a shared feature extraction layer, a dual-stream architecture containing supervised training stream and weakly supervised training stream is constructed. The dual-stream architecture is then integrated and fine-tuned using the first training sample set and the second training sample set to obtain the preset image encoder.

[0018] in,

[0019] The first training sample set includes a first single cell image and a first classification label;

[0020] The first classification label includes 28 cell categories;

[0021] The first single cell image was obtained by selecting 28 typical single cell images and performing strong enhancement processing;

[0022] The second training sample set includes a second single-cell image and a second classification label;

[0023] The second category label is negative and positive;

[0024] The second single-cell image comprises three parts:

[0025] The first part consists of single-cell images of high-quality positive cells; these high-quality positive cell images include typical cell images and high-confidence atypical cell images.

[0026] The second part consists of individual cell images, which are diverse positive cell images obtained through contextual perturbation.

[0027] The third part consists of individual cell images of diverse negative cells.

[0028] The step of pre-training the initial image encoder using the first training sample set to obtain a pre-trained image encoder includes:

[0029] Connect the initial image encoder to the first classifier;

[0030] When training the initial image encoder using the first single cell image from the first training sample set, a first cross-entropy loss is calculated based on the first predicted cell classification output by the first classifier and the first classification label, and the initial image encoder is updated by backpropagation to obtain the pre-trained image encoder.

[0031] The step of constructing a dual-stream architecture containing supervised training stream and weakly supervised training stream, using the pre-trained image encoder as a shared feature extraction layer, and then performing integrated fine-tuning training on the dual-stream architecture using the first training sample set and the second training sample set to obtain the preset image encoder includes:

[0032] First, connect the pre-trained image encoder to the first classifier; input the first single cell image from the first training sample set into the supervised training stream, and output the first predicted cell classification after processing by the pre-trained encoder and the first classifier; use the first predicted cell classification and the first classification label to calculate the second cross-entropy loss.

[0033] The pre-trained image encoder is then connected to the second classifier; the second single-cell image from the second sample set is input into the weakly supervised training stream, and the following loss is calculated after passing through the pre-trained encoder:

[0034] The second classifier outputs the second predicted cell classification, and the second predicted cell classification and the second classification label are used to calculate the third cross-entropy loss.

[0035] The cell features extracted by the pre-trained encoder are compared with the reference features in the pre-built feature matching library to calculate the feature consistency loss.

[0036] The total loss is calculated based on the second cross-entropy loss, the third cross-entropy loss, and the feature consistency loss. The total loss is then used for backpropagation to update the network parameters of the two-stream training architecture to obtain a preset image encoder.

[0037] in,

[0038] The preset image encoder is used to receive the single cell image and output the high-dimensional feature vector of the single cell image;

[0039] The preset text encoder is used to receive the user instruction and output the text features corresponding to the user instruction;

[0040] The preset mapper is used to receive the high-dimensional feature vector and output a feature vector with the same dimension as the text feature.

[0041] The preset large language model is used to receive the feature vector output by the mapper and the text features output by the text encoder, and output explanatory text that conforms to pathological logic.

[0042] The acquisition of the preset recognition model includes:

[0043] An initial recognition model is established; wherein, the initial recognition model includes a preset image encoder, a preset text encoder, an initial mapper, and an initial large language model;

[0044] A third training sample set is generated; the third training sample set includes multiple individual cell images and their corresponding classification labels.

[0045] First, fix the parameters of the preset image encoder, preset text encoder, and initial large language model in the initial recognition model, then train the preset recognition model using the third training sample set, update the parameters of the initial mapper, and obtain the pre-trained recognition model; the pre-trained recognition model includes the preset image encoder, the preset text encoder, the pre-trained mapper, and the initial large language model;

[0046] Then, fix the parameters of the preset image encoder and the preset text encoder in the pre-trained recognition model, use the third training sample set to train the pre-trained recognition model, update the parameters of the pre-trained mapper and the initial large language model, and obtain the preset recognition model.

[0047] In another embodiment, a cell image recognition device based on a large language model is provided, the device comprising:

[0048] The acquisition unit is used to acquire whole-section images of cervical cytology to be identified;

[0049] The detection unit is used to perform cell detection and obtain the location information, classification information and confidence information of cells in the whole slice image of the cervical cytology to be identified;

[0050] The cropping unit is used to crop out a single cell image from the whole cervical cytology slide image to be identified based on the location information corresponding to the positive cells, in response to determining that positive cells have been detected according to the classification information.

[0051] The selection unit is used to select a preset number of individual cell images based on the confidence information;

[0052] The recognition unit is used to input user instructions and selected single cell images into a preset recognition model to obtain recognition results corresponding to all single cell images output; wherein, the preset recognition model includes a preset image encoder, a preset text encoder, a preset mapper, and a preset large language model; the preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result;

[0053] The determining unit is used to determine the recognition result corresponding to the whole cervical cytology slide image to be identified based on the recognition results corresponding to all the individual cell images.

[0054] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a cell image recognition method based on a large language model.

[0055] In another embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a cell image recognition method based on a large language model.

[0056] As can be seen from the above technical solution, in the above embodiments, when performing cell detection on the whole cervical cytology slide image to be identified, if positive cells are detected, a preset recognition model is used to identify them and determine whether the corresponding positive cells are positive; and the recognition result of the whole cervical cytology slide image to be identified is determined according to the recognition result output by the preset recognition model; the preset large language model in the preset recognition model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result; this method can improve the accuracy of cell image recognition, thereby improving the efficiency and reliability of early cervical cancer screening. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the preset recognition model structure in the embodiments of this application;

[0058] Figure 2 This is a schematic diagram of the process for obtaining the preset image encoder in an embodiment of this application;

[0059] Figure 3This is a schematic diagram of the cell image recognition process based on a large language model in an embodiment of this application;

[0060] Figure 4 This is a schematic diagram of the cell image recognition device based on a large language model in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe the order or sequence of objects. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0064] The technical solution of the present invention will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] Existing technical solutions typically employ deep learning models, such as convolutional neural networks, to directly classify WSI or image patches, achieving automatic lesion detection. However, these methods still have significant drawbacks: the models are often "black boxes," only outputting lesion probabilities or categories, lacking interpretive evidence that aligns with pathological diagnostic logic and can be understood and trusted by clinicians; the workflow is fragmented, with cell detection, feature extraction, and classification diagnosis often performed independently, failing to form an end-to-end intelligent decision-making process; and they are highly dependent on data annotation, with model performance limited by a large amount of accurate cell-level labeled data, the acquisition of which is extremely costly, hindering the widespread application and promotion of the technology.

[0066] Therefore, there is an urgent need for an identification method that can improve the accuracy of cell image recognition, thereby enhancing the efficiency and reliability of early cervical cancer screening.

[0067] To address the aforementioned technical issues, this application provides a cell image recognition method based on a large language model. When performing cell detection on a whole-section cervical cytology image to be identified, if positive cells are detected, a preset recognition model is used to identify them and determine whether the corresponding positive cells are positive. The recognition result of the whole-section cervical cytology image is determined based on the recognition result output by the preset recognition model. The preset large language model in the preset recognition model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result. This method can improve the accuracy of cell image recognition, thereby enhancing the efficiency and reliability of early cervical cancer screening.

[0068] The explanatory text here, which conforms to pathological logic, outputs whether the cells corresponding to the whole cervical cytology slide image to be identified are negative or positive.

[0069] In this embodiment of the application, a preset recognition model needs to be obtained before performing cervical cytology whole-slice image recognition. See also Figure 1 , Figure 1 This is a schematic diagram of the preset recognition model structure in an embodiment of this application. The preset recognition model includes: a preset image encoder, a preset text encoder, a preset mapper, and a preset large language model;

[0070] The preset image encoder is used to receive single cell images and output high-dimensional feature vectors of single cell images.

[0071] A preset text encoder is used to receive user commands and output the text features corresponding to the user commands;

[0072] A preset mapper is used to receive high-dimensional feature vectors and output feature vectors with the same dimensions as the text features;

[0073] A pre-defined large language model is used to receive the feature vectors output by the mapper and the text features output by the text encoder, and output explanatory text that conforms to pathological logic.

[0074] In training the preset recognition model, a preset image encoder needs to be trained first. See [link / reference] Figure 2 , Figure 2 This is a schematic diagram illustrating the process of acquiring the preset image encoder in an embodiment of this application. The specific steps are as follows:

[0075] Step 201: Establish the initial image encoder.

[0076] Step 202: Generate the first training sample set and the second training sample set.

[0077] The first training sample set includes a first single-cell image and a first classification label. The first single-cell image is obtained by selecting typical single-cell images of 28 categories and performing strong enhancement processing. Strong enhancement processing here refers to random flipping, rotation, scaling, color jittering, etc. The first classification label includes 28 cell categories.

[0078] In the field of cervical cancer screening, the 28 categories specifically refer to the standardized cell image classification system used in artificial intelligence (AI)-assisted diagnostic systems. This system refines the traditional Bethesda Reporting System (TBS) into a more machine learning-friendly digital breakdown, enabling AI models to learn, identify, and label various cell morphologies in cervical liquid-based cytology smears (TCT / LBP). Individual cell images from these 28 categories are then subjected to strong enhancement processing and used as the first training sample set; the 28 categories are as follows:

[0079] 1. Normal squamous epithelial cells; 2. Normal cervical endocervical glandular epithelial cells; 3. Squamous metaplastic cells; 4. Atrophic cells; 5. Repair cells; 6. Inflammation-related reactive changes; 7. Radiation therapy-related reactive changes; 8. Intrauterine device (IUD) related changes; 9. Trichomonas vaginalis; 10. Fungal infection (Candida); 11. Actinomyces; 12. Herpes simplex virus (HSV) infection; 13. Bacterial vaginosis (clue cells); 14. Atypical squamous cells of uncertain significance (ASC-US); 15. Atypical Squamous cells, high-grade lesion not excluded (ASC-H), 16. Low-grade squamous intraepithelial lesion (LSIL), 17. High-grade squamous intraepithelial lesion (HSIL), 18. Squamous cell carcinoma (SCC), 19. Atypical glandular epithelial cells (AGC), 20. Atypical glandular epithelial cells, prone to neoplasia, 21. Adenocarcinoma in situ of the endocervical canal (AIS), 22. Adenocarcinoma, 23. Red blood cells / bloody background, 24. Inflammatory cells / exudate, 25. Mucus, 26. Necrotic debris, 27. Sperm, may interfere with diagnosis. 28. Impurities / artificial artifacts, such as staining precipitates, fibers, etc.

[0080] The second training sample set includes a second single-cell image and a second classification label; wherein the second classification label includes negative and positive.

[0081] The second single-cell image consists of three parts:

[0082] The first part consists of single-cell images, which are high-quality positive cell images. High-quality positive cell images include typical cell images and high-confidence atypical cell images. These images can be obtained through the following methods, but are not limited to:

[0083] Using single slices of cells as units, high-value positive samples (i.e., typical cells and high-confidence positive cells) are prioritized for preservation to construct the basic package sequence:

[0084] From the slice instances, select individual cell images corresponding to all positive cells first, and then select cells with a typicality of 1 and corresponding atypical cells with a confidence level ≥ 0.85 from these individual cell images corresponding to positive cells to ensure the reliability of the core information in the package.

[0085] Balanced combination: Each package contains a fixed 4 instances, combined in the ratio of "typical cells + high-confidence atypical cells", such as 1 typical cell + 3 high-confidence atypical cells, which can maintain the consistency between the sequence and the original diagnostic level of the slide.

[0086] The second part consists of individual cell images, which are diverse positive cell images obtained through contextual perturbation.

[0087] To address common clinical misdiagnosis scenarios, such as false negatives and insufficient diversity, we designed targeted enhancement strategies:

[0088] False negative simulation: 1-2 typical positive cells are mixed into the package, and the rest are matched with negative or low confidence cells to simulate the clinical scenario of "typical positive cells being masked by a negative background".

[0089] Enhanced Positive Diversity: Randomly select combinations of positive cells of different categories and confidence levels, such as [Hsil-0.68.png, Lsil-0.94.png, Ascus-0.85.png, Lsil-0.99.png], to cover mixed distributions of high / low confidence and typical / atypical, thereby improving the model's ability to identify diverse positive features.

[0090] The third part consists of individual cell images of diverse negative cells.

[0091] Negative packets are generated by randomly sampling from pure negative slices, such as [N-0.95.png, N-0.94.png, WS-0.95.png, WS-0.94.png], to increase the diversity of negative samples and avoid overfitting of the model to negative features.

[0092] Adding these three cell images to the second training sample not only preserves the core diagnostic value of typical cells, but also simulates complex cell distribution scenarios in clinical practice, such as false negatives and diverse positives, ultimately improving the model's diagnostic robustness to different cell sequences.

[0093] Step 203: Use the first training sample set to pre-train the initial image encoder to obtain the pre-trained image encoder.

[0094] The acquisition of the pre-trained image encoder specifically includes:

[0095] Connect the initial image encoder to the first classifier;

[0096] When training the initial image encoder using the first single cell image in the first training sample set, the first cross-entropy loss is calculated based on the first predicted cell classification and the first classification label output by the first classifier, and the initial image encoder is updated by backpropagation to obtain the pre-trained image encoder.

[0097] Step 204: Using the pre-trained image encoder as a shared feature extraction layer, construct a dual-stream architecture containing supervised training stream and weakly supervised training stream, and use the first training sample set and the second training sample set to perform integrated fine-tuning training on the dual-stream architecture to obtain the preset image encoder.

[0098] The specific process of obtaining a preset image encoder based on a pre-trained image encoder consists of two training phases:

[0099] First training phase: Connect the pre-trained image encoder to the first classifier; the first classifier can output 28 cell classifications.

[0100] The first single cell image in the first training sample set is input into the supervised training stream, and after being processed by the pre-trained encoder and the first classifier, the first predicted cell classification is output; the second cross-entropy loss is calculated using the first predicted cell classification and the first classification label.

[0101] The training in this stage is similar to the training of the initial image encoder, except that the second cross-entropy loss is not used directly to feed back and adjust the parameters. Instead, the second cross-entropy loss is used in conjunction with the subsequent loss to calculate the total loss and feed back and adjust the parameters. The specific implementation is described below.

[0102] Second training phase: Connect the pre-trained image encoder to the second classifier; the second classifier outputs two cell classifications, specifically negative and positive.

[0103] The second single-cell image from the second sample set is input into the weakly supervised training stream, and the following loss is calculated after passing through the pre-trained encoder:

[0104] The second classifier outputs the second predicted cell classification, and the second predicted cell classification and the second classification label are used to calculate the third cross-entropy loss.

[0105] Cellular features extracted by a pre-trained encoder are compared with reference features in a pre-built feature matching library to calculate feature consistency loss. This loss can be calculated as the cosine distance between the extracted cell features and the nearest neighbor features in the library; a larger distance results in a higher loss. This loss constrains the quality of predicted features and avoids noise interference.

[0106] The total loss is calculated based on the second cross-entropy loss, the third cross-entropy loss, and the feature consistency loss. The total loss is then used for backpropagation to update the network parameters of the two-stream training architecture, thereby obtaining the preset image encoder.

[0107] The total loss can be calculated by weighting and summing the second cross-entropy loss, the third cross-entropy loss, and the feature consistency loss.

[0108] After the preset image encoder is obtained, an initial recognition model can be established to obtain the preset recognition model.

[0109] An initial recognition model is established; wherein the initial recognition model includes a preset image encoder, a preset text encoder, an initial mapper, and an initial large language model; wherein the preset text encoder can be a pre-trained one, that is, one used in related technologies, and the preset text encoder is not trained in this embodiment.

[0110] Generate a third training sample set; the third training sample set includes multiple single cell images and corresponding classification labels; the classification labels are positive and negative. The data in the third training sample set can be diverse and can be the same as or different from the second training sample set. For ease of use, this application can directly use the second training sample set for training.

[0111] The training of the initial recognition model is divided into two stages:

[0112] The first stage, also known as the feature alignment stage, involves fixing the parameters of the preset image encoder, preset text encoder, and initial large language model in the initial recognition model, training the preset recognition model using the third training sample set, updating the parameters of the initial mapper, and obtaining the pre-trained recognition model. The pre-trained recognition model includes the preset image encoder, preset text encoder, pre-trained mapper, and initial large language model.

[0113] Here, the parameters of the preset image encoder, preset text encoder, and initial large language model are fixed, and the preset recognition model is trained. That is, during the training process, only the parameters of the initial mapper are updated, while the parameters of the preset image encoder, preset text encoder, and initial large language model remain unchanged. Therefore, the pre-trained recognition model and the pre-trained mapper in the pre-trained recognition model are finally obtained.

[0114] The second stage, also known as the instruction fine-tuning stage, involves fixing the parameters of the preset image encoder and preset text encoder in the pre-trained recognition model, training the pre-trained recognition model using a third training sample set, updating the parameters of the pre-trained mapper and the initial large language model, and obtaining the preset recognition model.

[0115] Here, the parameters of the preset image encoder and preset text encoder are fixed, and the pre-trained recognition model is trained. That is, during the training process, only the parameters of the pre-trained mapper and the initial large language model are jointly trained and updated, while the parameters of the preset image encoder and preset text encoder remain unchanged. Therefore, the preset recognition model is finally obtained, which includes the preset image encoder, preset text encoder, preset mapper and preset large language model.

[0116] The model is trained using complex instructions to understand cross-modal interaction logic, and finally, explanatory text that meets the TBS diagnostic criteria is generated as the recognition result.

[0117] During the training process, the user commands input not only contained problems, but also results, such as...

[0118] Question: Could you tell me whether the test result was negative or positive?

[0119] Result: ASSISTANT: This test result is positive.

[0120] In actual use, this user command only causes problems and produces no results.

[0121] When using a preset recognition model, it can be a preset recognition model that has been trained in real time through the above scheme, or it can be a preset recognition model that has been pre-trained and stored. This application embodiment does not impose any restrictions on this.

[0122] This completes the training of the entire preset recognition model.

[0123] See Figure 3 , Figure 3This is a schematic diagram of the cell image recognition process in an embodiment of this application. The specific steps are as follows:

[0124] Step 301: Obtain the whole slide image of the cervical cytology to be identified.

[0125] To perform cell detection more efficiently and accurately, an open-source library (OpenSlide) can be used to cut the whole cervical cytology slide image to be identified into 1024×1024 pixel blocks with an overlap rate of 10% to avoid cell breakage, and then perform cell detection on the cut pixel blocks.

[0126] Step 302: Perform cell detection to obtain the location information, classification information, and confidence information of cells in the whole slice image of the cervical cytology to be identified.

[0127] When performing cell detection, a pre-trained YOLOv5s model can be used. The confidence threshold is set to 0.7. There are no restrictions on the model used for cell detection; this is just one example.

[0128] Step 303: In response to determining that positive cells have been detected based on classification information, single cell images are cropped from the whole cervical cytology slide image to be identified based on the location information corresponding to the positive cells, and a preset number of single cell images are selected based on confidence information.

[0129] When selecting a single cell image for identification, if no positive cells are detected based on classification information, the identification of the whole cervical cytology slide image to be identified ends. If positive cells are detected based on classification information, the relevant operation in step 303 is performed, specifically:

[0130] In practice, you can crop out individual cell images based on the location information of positive cells, or you can crop out individual cell images based on the location information of all cells.

[0131] If the number of individual cell images corresponding to positive cells is less than a preset number, then all individual cell images corresponding to positive cells are directly input into the preset recognition model. Cell detection in this application is a preliminary identification; after identifying positive cells, a secondary identification is performed using the preset recognition model. The preset number can be set to 32, but is not limited to this.

[0132] Step 304: Input the user command and the selected single cell image into the preset recognition model to obtain the recognition results corresponding to all the output single cell images.

[0133] The preset recognition model includes: a preset image encoder, a preset text encoder, a preset mapper, and a preset large language model;

[0134] A preset image encoder is used to receive multiple single-cell images as the visual input source for a preset recognition model; and outputs a high-dimensional feature vector of the single-cell image.

[0135] A preset text encoder is used to receive user commands and output the text features corresponding to the user commands;

[0136] A preset mapper is used to receive high-dimensional feature vectors and output feature vectors with the same dimensions as the text features. In practice, it can be projected into feature vectors with the same dimensions as the text tokens by a two-layer multilayer perceptron (MLP), so as to achieve dimensional alignment between visual features and text features and prepare for subsequent fusion.

[0137] The pre-defined large language model is the core unit of the pre-defined recognition model. It receives the feature vectors output by the mapper and the text features output by the text encoder. If the sequence length does not meet expectations, it needs to be padded with sequence length or invalid information needs to be marked. This information is not involved in the actual calculation. By processing these feature sequences, the pre-defined recognition model learns the association between image features and text instructions, and finally generates and outputs explanatory text that conforms to pathological logic.

[0138] This architecture enables cross-modal interaction between cell images and natural language through a process of "image encoding → feature projection → text encoding → multimodal fusion".

[0139] The preset image encoder is obtained through two training stages: pre-training and integrated fine-tuning training; the preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result.

[0140] Step 305: Determine the recognition result corresponding to the whole cervical cytology slide image to be identified based on the recognition results corresponding to all individual cell images.

[0141] In practice, a recognition result is output for each input single cell image. Based on all the output recognition results, the recognition result of the whole cervical cytology slide image to be recognized is determined. In practice, as long as there is a positive result in the recognition result corresponding to the single cell image input into the preset recognition model, the recognition result corresponding to the whole cervical cytology slide image to be recognized is determined to be positive.

[0142] In this embodiment, when performing cell detection on a whole cervical cytology slide image to be identified, if positive cells are detected, a preset recognition model is used to identify them and determine whether the corresponding positive cells are positive. The recognition result of the whole cervical cytology slide image to be identified is determined based on the recognition result output by the preset recognition model. The preset large language model in the preset recognition model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result. This method can improve the accuracy of cell image recognition, thereby improving the efficiency and reliability of early cervical cancer screening.

[0143] The hierarchical dynamic data augmentation strategy provided in this application embodiment is as follows: through three levels of augmentation, namely "core preservation - scenario perturbation - negative augmentation", it not only preserves the diagnostic value of typical cells, but also simulates clinical scenarios such as false negatives and diverse positive distributions, thereby solving the problem of loss of diagnostic consistency caused by traditional augmentation and improving the robustness of the model.

[0144] The semi-weakly supervised training mechanism based on the feature matching library innovatively introduces a "feature matching library" that stores labeled cell features. By comparing the quality of cell features predicted by loss constraints, and combining WSI-MIL, a "cell-level-slice-level" dual-layer supervision is achieved. This efficiently mines the value of unlabeled WSI when labeled data is limited, reducing the dependence on labeled data.

[0145] Cross-modal interpretable diagnostic generation: By fusing image features and text features, LLM can generate interpretable text that conforms to pathological logic based on cell images, realizing end-to-end output "from image to diagnostic report" and breaking through the limitations of the "black box output" of traditional AI models.

[0146] This solution can achieve the following technical effects:

[0147] Enhance diagnostic interpretability: Generate explanatory text similar to that of a pathologist, transforming diagnostic results from abstract probabilities into clinically understandable logical descriptions, thereby increasing clinical trust.

[0148] Enhancing model robustness and generalization: Hierarchical data augmentation simulates complex clinical scenarios, and feature matching library constrains the quality of weakly supervised training, enabling the model to adapt to multiple sample distributions even with limited labeled data, resulting in a significant improvement in recognition accuracy compared to related technologies.

[0149] Achieving end-to-end automation: From WSI to clinical needs: Diagnostic texts conform to TBS system standards and are consistent with the pathologist's slide reading logic, lowering the threshold for clinical implementation and providing reliable auxiliary diagnostic tools for primary healthcare institutions.

[0150] Aligned with clinical needs: The diagnostic texts conform to the TBS system standards and are consistent with the reading logic of pathologists, lowering the threshold for clinical implementation and providing reliable auxiliary diagnostic tools for primary healthcare institutions.

[0151] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0152] Based on the same inventive concept, this application also provides a cell image recognition device based on a large language model. See also Figure 4 , Figure 4 This is a schematic diagram of the cell image recognition device based on a large language model, as described in an embodiment of this application. The device includes:

[0153] Acquisition unit 401 is used to acquire whole slice images of cervical cytology to be identified;

[0154] The detection unit 402 is used to perform cell detection and obtain the location information, classification information and confidence information of cells in the whole slice image of cervical cytology to be identified;

[0155] The cropping unit 403 is used to crop out a single cell image from the whole cervical cytology slide image to be identified based on the location information of the positive cells in response to determining that positive cells have been detected according to the classification information.

[0156] Selection unit 404 is used to select a preset number of individual cell images based on confidence information;

[0157] The recognition unit 405 is used to input user instructions and selected single cell images into a preset recognition model and obtain the recognition results corresponding to all single cell images output. The preset recognition model includes a preset image encoder, a preset text encoder, a preset mapper, and a preset large language model. The preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result.

[0158] The determining unit 406 is used to determine the recognition result corresponding to the whole cervical cytology slide image to be identified based on the recognition results corresponding to all individual cell images.

[0159] In another embodiment, obtaining the preset image encoder includes:

[0160] Establish the initial image encoder;

[0161] Generate the first training sample set and the second training sample set;

[0162] The initial image encoder is pre-trained using the first training sample set to obtain the pre-trained image encoder.

[0163] Using a pre-trained image encoder as a shared feature extraction layer, a dual-stream architecture containing supervised training stream and weakly supervised training stream is constructed. The dual-stream architecture is then integrated and fine-tuned using the first and second training sample sets to obtain the preset image encoder.

[0164] In another embodiment,

[0165] The first training sample set includes the first single cell image and the first classification label;

[0166] The first category label includes 28 cell types;

[0167] The first single-cell image was obtained by selecting 28 typical single-cell images and performing strong enhancement processing;

[0168] The second training sample set includes a second single-cell image and a second classification label;

[0169] The second category label is negative and positive;

[0170] The second single-cell image consists of three parts: the images in the second training sample set are composed of three parts of single-cell images.

[0171] The first part consists of single-cell images of high-quality positive cells; high-quality positive cell images include typical cell images and high-confidence atypical cell images.

[0172] The second part consists of individual cell images, which are diverse positive cell images obtained through contextual perturbation.

[0173] The third part consists of individual cell images of diverse negative cells.

[0174] In another embodiment, pre-training the initial image encoder using a first training sample set to obtain a pre-trained image encoder includes:

[0175] Connect the initial image encoder to the first classifier;

[0176] When training the initial image encoder using the first single cell image in the first training sample set, the first cross-entropy loss is calculated based on the first predicted cell classification and the first classification label output by the first classifier, and the initial image encoder is updated by backpropagation to obtain the pre-trained image encoder.

[0177] In another embodiment, a two-stream architecture comprising a supervised training stream and a weakly supervised training stream is constructed using a pre-trained image encoder as a shared feature extraction layer. The two-stream architecture is then integrated and fine-tuned using a first training sample set and a second training sample set to obtain a preset image encoder, including:

[0178] First, connect the pre-trained image encoder to the first classifier; input the first single cell image from the first training sample set into the supervised training stream, and output the first predicted cell classification after processing by the pre-trained encoder and the first classifier; use the first predicted cell classification and the first classification label to calculate the second cross-entropy loss.

[0179] The pre-trained image encoder is then connected to the second classifier; the second single-cell image from the second sample set is input into the weakly supervised training stream, and after passing through the pre-trained encoder, the following loss is calculated:

[0180] The second classifier outputs the second predicted cell classification, and the second predicted cell classification and the second classification label are used to calculate the third cross-entropy loss.

[0181] The cell features extracted by the pre-trained encoder are compared with the reference features in the pre-built feature matching library to calculate the feature consistency loss.

[0182] The total loss is calculated based on the second cross-entropy loss, the third cross-entropy loss, and the feature consistency loss. The total loss is then used for backpropagation to update the network parameters of the two-stream training architecture, thereby obtaining the preset image encoder.

[0183] In another embodiment,

[0184] A preset image encoder is used to receive single-cell images and output high-dimensional feature vectors of single-cell images;

[0185] A preset text encoder is used to receive user commands and output the text features corresponding to the user commands;

[0186] A preset mapper is used to receive high-dimensional feature vectors and output feature vectors with the same dimensions as the text features;

[0187] A pre-defined large language model is used to receive the feature vectors output by the mapper and the text features output by the text encoder, and output explanatory text that conforms to pathological logic.

[0188] In another embodiment, obtaining the preset recognition model includes:

[0189] Establish an initial recognition model; the initial recognition model includes a preset image encoder, a preset text encoder, an initial mapper, and an initial large language model;

[0190] Generate a third training sample set; the third training sample set includes multiple single cell images and their corresponding classification labels;

[0191] First, fix the parameters of the preset image encoder, preset text encoder, and initial large language model in the initial recognition model. Then, train the preset recognition model using the third training sample set, update the parameters of the initial mapper, and obtain the pre-trained recognition model. The pre-trained recognition model includes the preset image encoder, preset text encoder, pre-trained mapper, and initial large language model.

[0192] Then, fix the parameters of the preset image encoder and preset text encoder in the pre-trained recognition model, use the third training sample set to train the pre-trained recognition model, update the parameters of the pre-trained mapper and the initial large language model, and obtain the preset recognition model.

[0193] The units in the above embodiments can be integrated into one unit or deployed separately; they can be merged into one unit or further divided into multiple sub-units.

[0194] In another embodiment, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a cell image recognition method based on a large language model.

[0195] In another embodiment, a computer-readable storage medium is also provided, on which computer instructions are stored, which, when executed by a processor, implement a cell image recognition method based on a large language model.

[0196] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the following methods:

[0197] A cell image recognition method based on a large language model, characterized in that the method includes:

[0198] Obtain whole-section images of cervical cytology to be identified;

[0199] Cell detection is performed to obtain the location, classification, and confidence information of cells in the whole cervical cytology slide image to be identified;

[0200] In response to the detection of positive cells based on classification information, single cell images are cropped from the whole cervical cytology slide image to be identified based on the location information corresponding to the positive cells, and a preset number of single cell images are selected based on confidence information.

[0201] The user's instructions and the selected single cell images are input into the preset recognition model to obtain the recognition results corresponding to all single cell images. The preset recognition model includes a preset image encoder, a preset text encoder, a preset mapper, and a preset large language model. The preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result.

[0202] The identification result corresponding to the whole cervical cytology slide image to be identified is determined based on the identification results corresponding to all individual cell images.

[0203] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connectedly may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0207] Those skilled in the art will understand that the features described in the various embodiments and / or claims disclosed in this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.

[0208] This document uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of these embodiments are merely illustrative of the method and core concepts of the present invention and are not intended to limit this application. Those skilled in the art can make changes to the specific implementation methods and application scope based on the ideas, spirit, and principles of the present invention. Any modifications, equivalent substitutions, or improvements made should be included within the scope of protection of this application.

Claims

1. A cell image recognition method based on a large language model, characterized in that, The method includes: Obtain whole-section images of cervical cytology to be identified; Cell detection is performed to obtain the location, classification, and confidence information of cells in the whole cervical cytology slide image to be identified; In response to the determination of positive cells based on the classification information, a single cell image is cropped from the whole cervical cytology slide image to be identified based on the location information corresponding to the positive cell, and a preset number of single cell images are selected based on the confidence information. The user command and the selected single cell image are input into a preset recognition model to obtain the recognition results corresponding to all the single cell images output; wherein, the preset recognition model includes a preset image encoder, a preset text encoder, a preset mapper and a preset large language model; the preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result; The identification result corresponding to the whole cervical cytology slide image to be identified is determined based on the identification results corresponding to all the individual cell images.

2. The method according to claim 1, characterized in that, The acquisition of the preset image encoder includes: Establish the initial image encoder; Generate the first training sample set and the second training sample set; The initial image encoder is pre-trained using the first training sample set to obtain a pre-trained image encoder. Using the pre-trained image encoder as a shared feature extraction layer, a dual-stream architecture containing supervised training stream and weakly supervised training stream is constructed. The dual-stream architecture is then integrated and fine-tuned using the first training sample set and the second training sample set to obtain the preset image encoder.

3. The method according to claim 2, characterized in that, The first training sample set includes a first single cell image and a first classification label; The first classification label includes 28 cell categories; The first single cell image was obtained by selecting 28 typical single cell images and performing strong enhancement processing; The second training sample set includes a second single-cell image and a second classification label; The second category label is negative and positive; The second single-cell image comprises three parts: The first part consists of single-cell images of high-quality positive cells; these high-quality positive cell images include typical cell images and high-confidence atypical cell images. The second part consists of individual cell images, which are diverse positive cell images obtained through contextual perturbation. The third part consists of individual cell images of diverse negative cells.

4. The method according to claim 3, characterized in that, The step of pre-training the initial image encoder using the first training sample set to obtain a pre-trained image encoder includes: Connect the initial image encoder to the first classifier; When training the initial image encoder using the first single cell image from the first training sample set, a first cross-entropy loss is calculated based on the first predicted cell classification output by the first classifier and the first classification label, and the initial image encoder is updated by backpropagation to obtain the pre-trained image encoder.

5. The method according to claim 3, characterized in that, The step of constructing a dual-stream architecture containing supervised and weakly supervised training streams, using the pre-trained image encoder as a shared feature extraction layer, and then performing integrated fine-tuning training on the dual-stream architecture using the first and second training sample sets to obtain the preset image encoder includes: First, connect the pre-trained image encoder to the first classifier; input the first single cell image from the first training sample set into the supervised training stream, and output the first predicted cell classification after processing by the pre-trained encoder and the first classifier; use the first predicted cell classification and the first classification label to calculate the second cross-entropy loss. The pre-trained image encoder is then connected to the second classifier; the second single-cell image from the second sample set is input into the weakly supervised training stream, and the following loss is calculated after passing through the pre-trained encoder: The second classifier outputs the second predicted cell classification, and the second predicted cell classification and the second classification label are used to calculate the third cross-entropy loss. The cell features extracted by the pre-trained encoder are compared with the reference features in the pre-built feature matching library to calculate the feature consistency loss. The total loss is calculated based on the second cross-entropy loss, the third cross-entropy loss, and the feature consistency loss. The total loss is then used for backpropagation to update the network parameters of the two-stream training architecture to obtain a preset image encoder.

6. The method according to claim 1, characterized in that, The preset image encoder is used to receive the single cell image and output the high-dimensional feature vector of the single cell image; The preset text encoder is used to receive the user instruction and output the text features corresponding to the user instruction; The preset mapper is used to receive the high-dimensional feature vector and output a feature vector with the same dimension as the text feature. The preset large language model is used to receive the feature vector output by the mapper and the text features output by the text encoder, and output explanatory text that conforms to pathological logic.

7. The method according to claim 1, characterized in that, The acquisition of the preset recognition model includes: An initial recognition model is established; wherein, the initial recognition model includes a preset image encoder, a preset text encoder, an initial mapper, and an initial large language model; A third training sample set is generated; the third training sample set includes multiple individual cell images and their corresponding classification labels. First, fix the parameters of the preset image encoder, preset text encoder, and initial large language model in the initial recognition model, then train the preset recognition model using the third training sample set, update the parameters of the initial mapper, and obtain the pre-trained recognition model; the pre-trained recognition model includes the preset image encoder, the preset text encoder, the pre-trained mapper, and the initial large language model; Then, fix the parameters of the preset image encoder and the preset text encoder in the pre-trained recognition model, use the third training sample set to train the pre-trained recognition model, update the parameters of the pre-trained mapper and the initial large language model, and obtain the preset recognition model.

8. A cell image recognition device based on a large language model, characterized in that, The device includes: The acquisition unit is used to acquire whole-section images of cervical cytology to be identified; The detection unit is used to perform cell detection and obtain the location information, classification information and confidence information of cells in the whole slice image of the cervical cytology to be identified; The cropping unit is used to crop out a single cell image from the whole cervical cytology slide image to be identified based on the location information corresponding to the positive cells, in response to determining that positive cells have been detected according to the classification information. The selection unit is used to select a preset number of individual cell images based on the confidence information; The recognition unit is used to input user instructions and selected single cell images into a preset recognition model to obtain recognition results corresponding to all single cell images output; wherein, the preset recognition model includes a preset image encoder, a preset text encoder, a preset mapper, and a preset large language model; the preset large language model combines image features and text features to generate explanatory text that conforms to pathological logic as the recognition result; The determining unit is used to determine the recognition result corresponding to the whole cervical cytology slide image to be identified based on the recognition results corresponding to all the individual cell images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1-7.

Images