Cervical cell pathological image analysis method, device and equipment based on visual language model

Abstract

The application discloses a cervical cell pathological image analysis method, device and equipment based on a visual language model, wherein the method comprises the following steps: acquiring a cervical cell pathological image; inputting the image and a task prompt word into the pathological model to output a multi-task analysis result. The model comprises a region proposal module, a multi-scale region feature extraction network, a hierarchical attention gate encoder and a large language model decoder. The region proposal module is used for generating a candidate cell detection frame; the multi-scale region feature extraction network is used for constructing a multi-scale region representation; the hierarchical attention gate encoder is used for simulating the observation process of a pathologist from coarse to fine and realizing scale selection based on multi-task adaptive gating; and the large language model decoder is used for simultaneously performing cell positioning, region semantic description, lesion and slice source classification and clinical reasoning tasks. The application solves the problems of poor multi-target positioning, missing fine-grained semantics and unbalanced multi-task capability of existing models.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and medical image analysis, and more specifically, to a method, apparatus, and device for analyzing cervical cell pathology images based on a visual language model, belonging to the application of deep learning in computer-aided diagnosis. Background Technology

[0002] Cervical cancer is a malignant tumor that seriously threatens women's health worldwide. Early detection and diagnosis through cervical cytology screening (such as TCT (Thinprep Cytologic Test) / LCT (Liquid-based Cytology Test)) is a recognized effective means of reducing the incidence and mortality of cervical cancer. Traditional screening processes rely heavily on pathologists manually interpreting tens of thousands of cells under a microscope, which is not only labor-intensive but also susceptible to subjective experience, making it prone to missed diagnoses.

[0003] To improve screening efficiency, numerous studies in recent years have utilized deep learning technologies, particularly object detection models (such as YOLO and Faster R-CNN), to assist doctors in identifying abnormal cells. While these specialized detection models can identify and locate atypical cells with high accuracy, their output is limited to bounding box coordinates and predefined category labels, failing to provide detailed explanations of cell morphological characteristics (such as nuclear-cytoplasmic ratio, nuclear membrane roughness, and chromatin distribution). This "black box" approach makes it difficult for doctors to quickly understand the diagnostic criteria provided by the model, limiting the credibility and depth of artificial intelligence's application in clinical decision support.

[0004] To address the interpretability issue, research on visual language models in the field of medical imaging is increasingly emerging. By incorporating natural language processing capabilities, visual language models can generate professional descriptions of medical images, assisting doctors in making logical clinical inferences. However, in the field of cellular pathology analysis, existing visual language models and related technical approaches still face the following challenges: At the data level, existing large-scale medical visual language datasets (such as OmniMedVQA, PubMed, and other publicly available medical image and text datasets) mostly provide global descriptive text for entire slices or images. However, in cervical cytology diagnosis, clinical value often lies in the subtle morphological analysis of specific individual abnormal cells or lesion areas. Due to the lack of precise correspondence annotations between "local cell location and professional region description," existing models cannot truly achieve feature analysis of specific lesion locations, resulting in superficial and vague descriptions lacking spatial direction.

[0005] At the model architecture level, existing visual language models (such as Qwen-VL, InternVL, LLaVA, etc.) typically adopt a "language-centric architecture," treating coordinate information as discrete text markers for prediction. This paradigm, which maps continuous coordinate space to discrete probability distribution, has inherent flaws: on the one hand, language models are essentially probabilistic generative models, making it difficult to accurately and stably output continuous values, resulting in the localization accuracy of VLMs (Visual Language Models) being far lower than that of dedicated detectors; on the other hand, when there are multiple densely distributed abnormal cells in a single slice, the generation of long-sequence coordinate text is prone to cumulative errors and attention diffusion, causing the model to perform poorly in multi-object detection scenarios, often resulting in missed detections, false detections, or "illusion" phenomena of unclear target referencing.

[0006] In summary, existing technologies cannot provide fine-grained pathological diagnostic texts that conform to clinical standards while ensuring high accuracy and multi-target localization. Therefore, there is an urgent need for a multi-task visual language model capable of accurate localization and deep reasoning in complex multi-target scenarios, specifically for cervical cytology. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a multi-task visual language model based on hierarchical attention gating. By using a hierarchical attention gating mechanism, it simulates the multi-scale observation process of a pathologist, from coarse to fine, and achieves multi-task collaboration in localization, description, classification, and reasoning.

[0008] The above-mentioned technical problems of the present invention are mainly solved by the following technical solutions: The first aspect provides a method for analyzing cervical cell pathology images based on a visual language model, including: Obtain pathological images of cervical cells to be analyzed; The cervical cell pathology image to be analyzed and the task prompts are input into the pre-trained cervical cell pathology model, and the multi-task analysis results are output. The cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal module generates detection boxes for candidate regions based on natural language prompts. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features. The hierarchical attention-gated encoder selectively fuses the multi-scale region representations. The large language model decoder performs multi-task analysis in a unified manner to obtain multi-task analysis results.

[0009] In one implementation, the region proposal module generates detection boxes for candidate regions based on natural language prompts, including: The region proposal module uses a target detection model finely tuned on the cervical cell dataset. It takes natural language prompts as input and outputs the bounding box coordinates and confidence scores of candidate cell regions.

[0010] In one implementation, a multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features, including: Using the feature map output by a single-scale ViT as input, feature levels of four scales are constructed through upsampling and lateral connection. For each scale feature map and each candidate box, perform a region of interest alignment operation to obtain a region feature block with a fixed spatial size; Spatial pooling is performed on the regional feature blocks to obtain the regional representation at the corresponding scale.

[0011] In one implementation, the hierarchical attention-gated encoder includes a cross-layer attention module and an adaptive gating module. The hierarchical attention-gated encoder selectively fuses multi-scale region representations, including: The cross-layer attention module is used to aggregate information layer by layer from the regional representations at different scales. By using an adaptive gating module to dynamically weight the contributions of features at each scale based on the Sigmoid function, and fusing task biases, scale preference adjustment can be achieved under multi-task conditions.

[0012] In one implementation, the large language model decoder uniformly performs multi-task analysis to obtain multi-task analysis results, including: The large language model decoder takes a sequence of concatenated image tokens, region tokens, and text tokens as input, unifying all tasks into a next-to-next-to-toe prediction paradigm. This paradigm is used to simultaneously perform abnormal cell localization, region semantic description, lesion and slide origin classification, and clinical reasoning tasks, executing multiple reasoning tasks to obtain multi-task analysis results. Specifically, the image token is obtained by ViT encoding the input image, the region token is obtained by feature extraction and encoding of candidate regions, and the text token is obtained by encoding task text prompts. Unifying all tasks into the next-to-next-to-toe prediction paradigm includes: cell target localization by generating reference tags; region semantic description by generating natural language description text corresponding to the region; cell lesion classification by generating category label text; and clinical reasoning by generating structured thought chain text.

[0013] In one implementation, the pre-trained cervical cell pathology model further includes a global image feature projection layer and a regional image feature projection layer. The pre-trained cervical cell pathology model is obtained using a two-stage training strategy, wherein the two-stage training strategy includes: The first stage focuses on training for perception tasks, optimizing the global image feature projection layer, hierarchical attention-gated encoder, regional image feature projection layer, and attention layer of the large language model decoder through an efficient parameter fine-tuning strategy. The second stage extends to training for semantic understanding and reasoning tasks, freezing the visual encoding module, using LoRA to train the attention layer of the large language model decoder, fine-tuning the hierarchical attention-gated encoder, and introducing some training data from the first stage while injecting the ability of thought chains. The visual encoding module includes ViT, the global image feature projection layer, and the regional image feature projection layer.

[0014] In one implementation, the training process uses a uniform text cross-entropy loss function:

[0015] in, To output the sequence length, For the t-th target token, This represents all text tokens generated up to the current position. This is a visual feature sequence, containing image tokens and region tokens. The image token is obtained by encoding the input image using ViT, and the region token is obtained by extracting and encoding features from candidate regions. The task prompt text is a token sequence. Given visual features, task cues, and all text generated up to the current position, this represents the predicted current position. The logarithm of the probability of a target token at a given location.

[0016] Based on the same inventive concept, a second aspect of the present invention provides a cervical cell pathology image analysis device based on a visual language model, comprising: The image acquisition module is used to acquire pathological images of cervical cells to be analyzed. The multi-task analysis module is used to input the cervical cell pathology image to be analyzed and the task prompt words into the pre-trained cervical cell pathology model and output the multi-task analysis results. The cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal module generates detection boxes for candidate regions based on natural language prompts. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features. The hierarchical attention-gated encoder selectively fuses the multi-scale region representations. The large language model decoder performs multi-task analysis in a unified manner to obtain multi-task analysis results.

[0017] Based on the same inventive concept, a third aspect of the present invention provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cervical cell pathological image analysis method based on a visual language model as described in the first aspect.

[0018] Based on the same inventive concept, a fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cervical cell pathological image analysis method based on the visual language model described in the first aspect.

[0019] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: This invention discloses a cervical cell pathology image analysis method based on a visual language model. After acquiring cervical cell pathology images, the images and task prompts are input into the pathology model, which outputs multi-task analysis results. The model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal generation module generates detection boxes for candidate cell regions based on natural language prompts, realizing cell-level region of interest extraction. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features, providing multi-level features from global context to fine-grained texture. The hierarchical attention-gated encoder simulates the pathologist's progressive observation process "from coarse to fine" through a cross-layer attention mechanism and realizes scale selection based on multi-task adaptive gating. The large language model decoder is used to simultaneously perform cell localization, region semantic description, lesion and slide origin classification, and clinical reasoning tasks, thereby solving the problems of poor multi-target localization, lack of fine-grained semantics, and unbalanced multi-task capabilities in existing models.

[0020] Furthermore, based on the VLM-FO1 paradigm, the coordinate text generation task of traditional VLM in target localization scenarios is reconstructed into a region feature retrieval task, which solves the problem of insufficient multi-target localization capability of existing visual language models in cervical cell analysis.

[0021] Furthermore, by using the cross-layer attention and task-adaptive gating mechanism in the hierarchical attention gating encoder, the progressive multi-scale reasoning ability of pathologists from coarse-grained to fine-grained is simulated. All tasks are unified into an autoregressive next-to-token prediction paradigm for end-to-end optimization. At the same time, a parameter-efficient fine-tuning strategy is adopted to achieve two-stage training, which solves the problems of lack of fine-grained semantic understanding ability and unbalanced multi-task ability in existing methods.

[0022] In summary, the solution of this invention can simultaneously achieve precise localization, regional semantic understanding, and structured reasoning within a unified framework, providing interpretable intelligent auxiliary diagnostic capabilities for cervical cell pathology analysis. This invention is simple and convenient to implement, highly practical, and solves the problems of low practicality and inconvenience in real-world applications of existing technologies. It can improve clinical screening efficiency and diagnostic reliability, and has significant market value. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the architecture of the cervical cell pathological model in an embodiment of the present invention; Figure 2 This is a schematic diagram of the dataset format in an embodiment of the present invention; Figure 3 This is a functional illustration of six tasks in an embodiment of the present invention; Figure 4 This is a schematic diagram of a two-stage training strategy in an embodiment of the present invention, where MLP stands for Multilayer Perceptron, VIT stands for Vision Transformer, HAGE stands for Hierarchical Attention Gated Encoder, Text Encoder stands for Text Encoder, and Self-Attention stands for Self-Attention Mechanism. Detailed Implementation

[0025] Example 1 This embodiment provides a method for analyzing cervical cell pathological images based on a visual language model, including: S1: Obtain the cervical cell pathology image to be analyzed; S2: Input the cervical cell pathology image to be analyzed and the task prompts into the pre-trained cervical cell pathology model, and output the multi-task analysis results; The cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal module generates detection boxes for candidate regions based on natural language prompts. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features. The hierarchical attention-gated encoder selectively fuses the multi-scale region representations. The large language model decoder performs multi-task analysis in a unified manner to obtain multi-task analysis results.

[0026] Specifically, the region proposal module generates detection boxes for candidate cell regions, enabling cell-level region of interest extraction; the multi-scale region feature extraction network is used to construct multi-scale region representations from single-scale ViT features, providing multi-level features from global context to fine-grained texture; the hierarchical attention-gated encoder simulates the pathologist's progressive observation process "from coarse to fine" through a cross-layer attention mechanism, and achieves scale selection based on multi-task adaptive gating; the large language model decoder is used to simultaneously perform cell localization, region semantic description, lesion and slide origin classification, and clinical reasoning tasks.

[0027] As an example, the cervical cytology image is a cervical cell smear prepared using liquid-based thin-layer cytology (LBC) technology and stained with Papanicolaou. Multi-task analysis results include region reference tags for the target cells. <regionx>The model consists of: regional semantic description text, cell lesion classification and slide source classification results, and reasoning diagnosis thought chain text; the cervical cell pathology model is trained using a joint dataset containing fine-grained localization annotations, regional description text, and structured reasoning chains; the cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention gating encoder, and a large language model decoder.

[0028] Please see Figure 1 This is a schematic diagram of the cervical cell pathological model in an embodiment of the present invention.

[0029] The input image first passes through a region proposal module to generate a list of candidate cell regions for detection. Simultaneously, after ViT encoding, the image is used to construct a multi-scale feature map via SimpleFPN (Multi-Scale Region Feature Extraction Network). For each candidate region, ROIAlign (Region of Interest Alignment) and spatial pooling operations are performed on each feature map layer to extract region vectors at each scale. These vectors are then input into a hierarchical attention-gated encoder for progressive cross-layer attention aggregation and task-adaptive gating to obtain a region token. The image token, region token, and task prompt text token are concatenated sequentially and input into a large language model decoder to generate multi-task outputs in an autoregressive manner.

[0030] In one implementation, the region proposal module generates detection boxes for candidate regions based on natural language prompts, including: The region proposal module uses a target detection model finely tuned on the cervical cell dataset. It takes natural language prompts as input and outputs the bounding box coordinates and confidence scores of candidate cell regions.

[0031] Specifically, the target detection model fine-tuned on the cervical cell dataset is Grounding DINO. The region proposal module takes natural language prompts as input and outputs the bounding box coordinates and confidence scores of candidate cell regions, thus achieving cell-level region of interest extraction. For example, taking natural language prompts (such as "positive cervical cells") as input, it outputs the bounding box coordinates and confidence scores of candidate cell regions. In practice, a confidence threshold can be set to filter out low-quality candidate boxes and retain high-confidence regions for subsequent analysis.

[0032] In one implementation, a multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features, including: Using the feature map output by a single-scale ViT as input, feature levels of four scales are constructed through upsampling and lateral connection. For each scale feature map and each candidate box, perform a region of interest alignment operation to obtain a region feature block with a fixed spatial size; Spatial pooling is performed on the regional feature blocks to obtain the regional representation at the corresponding scale.

[0033] Specifically, the multi-scale region feature extraction network takes the feature map output by the single-scale ViT as input, and constructs four feature levels at four scales through upsampling and lateral connections. It simulates the progressive observation granularity from high magnification to low magnification and provides multi-level features from global context to fine-grained texture.

[0034] In practical implementation, the feature map output by single-scale ViT is used. As input, where H represents height, W represents width, and C represents the number of channels, four feature levels are constructed through upsampling and lateral connections. }, for each scale feature map and each candidate box Execute the ROIAlign operation. Let the set of candidate region bounding boxes output by the region proposal module be { },in Let be the coordinates of the top-left and bottom-right corners of the k-th candidate box.

[0035] The ROIAlign operation first scales the candidate bounding box coordinates to... The scale is used to obtain the scaled bounding box coordinates. Then Average score There are 14 grids (s=14 in this example), and m points are sampled within each grid using bilinear interpolation (in this example) ,in (This is the width of the candidate region); finally, the average value of the sampling points within each grid is taken to obtain a value of size . , where for The number of channels. This process can be expressed by the formula:

[0036] in, In order to be in Uniformly divided inside A set of grid center points In feature map The eigenvector obtained by bilinear interpolation on point p.

[0037] Spatial pooling refers to spatial pooling of a region's feature blocks, averaging along the height and width dimensions to obtain a region vector representation at that scale. :

[0038] To unify the feature dimensions across different scales, The image is mapped to a fixed dimension D through a linear projection layer. In this example, D=512, i.e.:

[0039] For all N candidate regions, the region vector matrix at the i-th scale is denoted as... ,in The k-th action Finally, the region vectors of the same candidate region at four scales are { The data is organized hierarchically (from high to low resolution) and then input into a hierarchical attention-gated encoder to achieve multi-scale region vector encoding.

[0040] In one implementation, the hierarchical attention-gated encoder includes a cross-layer attention module and an adaptive gating module. The hierarchical attention-gated encoder selectively fuses multi-scale region representations, including: The cross-layer attention module is used to aggregate information layer by layer from the regional representations at different scales. By using an adaptive gating module to dynamically weight the contributions of features at each scale based on the Sigmoid function, and fusing task biases, scale preference adjustment can be achieved under multi-task conditions.

[0041] Specifically, the hierarchical attention-gated encoder is a first proposed invention. It simulates the progressive observation process of a pathologist from coarse to fine through a cross-layer attention mechanism, and achieves selective fusion of multi-scale features based on task-adaptive gating. The encoder includes a cross-layer attention module and an adaptive gating module.

[0042] The cross-layer attention module is composed of a multi-head attention (MHA) module and a hierarchical aggregation structure, which performs cross-layer information query and fusion in order from fine-grained to coarse-grained.

[0043] The adaptive gating module includes a hierarchical selector, a task bias vector, a sigmoid activation function, and a normalization unit, which are used to implement task adaptive gating.

[0044] The preferred implementation scheme further proposed in the embodiments is as follows: The cross-layer attention module employs a hierarchical encoding and cross-layer attention approach, aggregating information from the finest-grained level to the coarse-grained level sequentially. First, it processes the region vectors at the finest-grained level... Perform a linear transformation to obtain the transformed features. :

[0045] Where N is the number of candidate regions and D is the feature dimension. In this embodiment, the maximum value of N is set to 20, and the preferred value of D is 512.

[0046] Then, cross-layer attention calculations are performed in order from fine to coarse:

[0047]

[0048]

[0049] MHA stands for Multi-Head Attention, and it is calculated as follows:

[0050] in, For normalized exponential functions, This is the learnable parameter matrix.

[0051] The features from each layer are concatenated along the channel dimension to obtain the fused region feature representation S:

[0052] The adaptive gating module dynamically adjusts the contribution of features at different scales based on different task types and input samples. This includes a hierarchical selector, task bias vector, sigmoid activation function, and normalization unit. Before entering the gating network, each sample in the dataset is assigned a task identifier (task_id) during processing. The input to the gating network is the concatenated region feature S and the task identifier (task_id).

[0053] The LevelSelector consists of two fully connected layers and one ReLU activation layer, used to compute adaptive gated logic values.

[0054] Where W1 and W2 are the learnable weight matrices of the two fully connected layers, b1 and b2 are the bias vectors of the two fully connected layers, and the dimension of the hidden layers is set to 128. The i-th row in the middle represents the gating weights of the i-th ROI on the four scales.

[0055] The task bias vector is a predefined, learnable set of vectors for each task type, preferably set as follows:

[0056] The gate logic value GL is calculated as follows:

[0057] in, This is the task bias weight coefficient, with a preferred value of 0.2.

[0058] Subsequently, for the j-th level, the gated logic value is gated using the Sigmoid function. Convert to raw gating weight values And introduce temperature coefficient :

[0059] about and The relationship between the gated logic values ​​is explained below: The dimension is Nx4, where rows represent N samples and columns represent 4 scales. Therefore... This indicates taking the j-th column of the GL matrix, which represents the gated logic value of all N samples at the j-th scale.

[0060] Next, we introduce a lower bound constraint, setting the lower bound threshold f = 0.02, and calculate the gate value with the lower bound constraint:

[0061] Normalizing the gating weights yields:

[0062] The final region features are represented as a weighted sum of features at each scale and their gating weights:

[0063] Will The region feature projection layer (MLP) maps the region tokens to the embedding space of a large language model, generating a sequence of region tokens. The vector obtained from the j-th layer, including the vectors mentioned above. ~ .

[0064] In one implementation, the large language model decoder uniformly performs multi-task analysis to obtain multi-task analysis results, including: The large language model decoder takes a sequence of concatenated image tokens, region tokens, and text tokens as input, unifying all tasks into a next-to-next-to-toe prediction paradigm. This paradigm is used to simultaneously perform abnormal cell localization, region semantic description, lesion and slide origin classification, and clinical reasoning tasks, executing multiple reasoning tasks to obtain multi-task analysis results. Specifically, image tokens are obtained by ViT encoding the input image, region tokens are obtained by feature extraction and encoding of candidate regions, and text tokens are obtained by encoding task text prompts. Unifying all tasks into the next-to-next-to-toe prediction paradigm includes: cell target localization by generating reference tags; region semantic description by generating natural language description text corresponding to the region; cell lesion classification by generating category label text; and clinical reasoning by generating structured thought chain text.

[0065] In the specific implementation, the large language model decoder adopts the VLM-FO1-Qwen2.5-VL pre-trained model. The input sequence is constructed by sequentially concatenating the image token obtained from single-scale ViT encoding, the region token obtained from hierarchical attention-gated encoder encoding, and the task-related text cue token to form a unified multimodal input sequence. The vocabulary of this pre-trained model includes… <region0> , <region1> , …., <regionx>Special index tokens, such as "[index token name]", enable the model to distinguish and reference different regions, where each index token appears in pairs with its corresponding region token. On the output side, the model does not need to output any absolute or relative coordinates; instead, it generates [the output]... <regionx>The form of region reference tags enables precise location of specific candidate regions.

[0066] The aforementioned cervical cell pathology model possesses both precise localization and multi-task synergy. Precise localization refers to the model's reconstruction of target localization from the traditional coordinate regression generation mode to a referencing mode based on regional feature retrieval and symbolic reference, significantly improving the accuracy of cervical cell detection in multi-target scenarios. Multi-task synergy refers to the model's ability to achieve dynamic and adaptive collaborative fusion between multi-scale features and multi-type tasks through a hierarchical attention gating mechanism.

[0067] The aforementioned high localization accuracy stems from the model's reconstruction of the target localization task—which relies on coordinate text generation in traditional visual language models—based on the VLM-FO1 paradigm, into a task oriented towards feature retrieval and symbolic reference for candidate regions. This eliminates the need for the model to output any absolute or relative coordinates, thus avoiding the inherent limitations of language models in continuous numerical prediction. Furthermore, region tokens, image tokens, and text tokens have equal status in the input sequence, enabling the model to simultaneously complete the tasks of "locating which region" and "what morphological features this region possesses" within a unified autoregressive framework, achieving a natural decoupling between localization and semantic description. This characteristic allows the present invention to significantly outperform existing general-purpose visual language models in the multi-abnormal cell detection scenario of cervical cytology images, and is on par with or even better than dedicated detection models.

[0068] The aforementioned multi-task collaboration originates from the hierarchical attention-gated encoder proposed in this invention. This encoder simulates a pathologist's progressive observation process from coarse to fine through a cross-layer attention module, while an adaptive gating module dynamically weights the contributions of features at each scale according to the task type. This allows the model to automatically enhance the weight of fine-grained features when handling tasks requiring detailed morphological analysis, and strengthen the role of coarse-grained features when handling tasks requiring global context. Unlike traditional methods that design separate network branches for each task or rely on fixed feature fusion strategies, the multi-task collaboration mechanism of this invention achieves performance balance and synergistic improvement across multiple tasks under a single model and unified training paradigm, effectively avoiding the performance imbalance problem in multi-task learning.

[0069] Example 2 Based on Example 1, this example further includes a method for training a cervical cell pathological model, comprising: The first stage focuses on training for perception tasks. A parameter-efficient fine-tuning strategy is used to optimize the global image feature projection layer, hierarchical attention-gated encoder, regional image feature projection layer, and the attention layer of the large language model decoder to achieve domain semantic alignment. The second stage extends to training for semantic understanding and reasoning tasks. The visual encoding module is frozen, LoRA is used to train the attention layer of the large language model decoder, and the hierarchical attention-gated encoder is fine-tuned. While injecting cognitive chain capabilities, some training data from the first stage is introduced to mitigate catastrophic forgetting. The visual encoding module includes ViT, the global image feature projection layer, and the regional image feature projection layer.

[0070] It should be noted that the global image feature projection layer corresponds to the leftmost column of the MLP. The entire image passes through the ViT visual encoder and the MLP global image feature projection layer to obtain the image token; the region image feature projection layer corresponds to... Figure 1 The MLP in the middle column, the image, and the bounding box list generated by the region proposal module are input together into the hierarchical attention gating encoder to obtain region image features, which are then input into the region image feature projection layer MLP to obtain region tokens.

[0071] In the specific implementation process, the training and inference process of the model includes the following steps: Step 1: Create a training dataset In this embodiment, the cervical cell pathology images are cervical cell smears prepared using liquid-based thin-layer cytology (LBC) technology and stained with Papanicolaou. To support multi-task learning, a joint dataset containing six tasks is constructed, including: abnormal cell detection, cell index expression understanding, region semantic description, cytopathic effect classification, slide origin classification, and diagnostic reasoning generation.

[0072] The abnormal cell detection task involves the user specifying target labels (e.g., "positive cells" or "negative cells"), and the model outputting the localization results. The cell index expression task involves the user describing the morphological characteristics of target cells (e.g., "enlarged nuclei" or "deeply stained nuclei"), and the model outputting the localization results. The region semantic description task involves the user specifying a region in the image, and the model outputting the cell description for that region. The cell lesion classification task involves the user specifying a cell in the image and asking for the cell type, and the model outputting the cell type label (e.g., "HSIL" or "LSIL"). The slide origin classification task involves the user asking which tissue layer the entire image originates from, and the model outputting the layer label (e.g., "cervical squamous epithelial basal layer" or "cervical squamous epithelial medial layer"). The diagnostic reasoning generation task involves the user asking for the reasoning basis for the diagnosis of a cell in the image, and the model outputting the diagnostic process text in a structured thought chain format.

[0073] The above tasks are numbered 0 to 5 in sequence, and the specific preparation process is as follows.

[0074] The first step is to create a location dataset: The localization dataset is used to train the model to detect and interpret positive / negative cell regions.

[0075] (1) Using the region proposal module, candidate cell detection boxes are generated for each cervical cytopathology image. To ensure recall and avoid missed detections, a confidence threshold of 0.2 is preferably set, and non-maximum suppression (NMS) is used to remove redundant detection boxes. A unique region index token is assigned to each retained detection box, denoted as [token name missing].<region_0> ,<region_1> ,…,<region_N> .

[0076] (2) The pathologist marks the areas of true positive / negative cells on the image. For each Calculate the Intersection over Union (IoU) ratio between the bounding box and all other candidate boxes, and select the candidate box with the highest IoU. Then the region index corresponding to the candidate box<region_i> This is the standard answer for the true positive / negative cell region.

[0077] (3) When constructing dialogue data, if a user asks, "Please box out the location of the positive cervical cells in the image" or "Find the positive cells in the image with enlarged nuclei, high nucleus-cytoplasmic ratio, and irregular nuclei," the assistant will answer in the form of " <ground> Positive cervical cells< / ground> <objects> <region0>< / region0> < / objects> The format enables the binding of natural language descriptions with specific region indexes. Among these, <ground> <objects>and <region0>It is a set of special control tokens that the VLM-FO1 decoder can recognize: <ground>< / ground> "This is a semantic anchor token used to mark key noun phrases in a user query, instructing the model to semantically bind that phrase to subsequent visual regions." <objects> <region0>< / region0> < / objects> "This is a token related to the region reference." <region0>" is the region index token, pointing to the feature vector of the 0th region.

[0078] The second step is to create a region description dataset: The region description dataset is used to train models to generate morphological description text for specific cellular regions.

[0079] (1) The pathologist marks the region of interest on the image and writes the corresponding cell morphology description, which includes the following professional terms: nuclear deep staining, nuclear enlargement, nuclear irregularity, hollowing, high nuclear-cytoplasmic ratio, nuclear crowding, etc.

[0080] (2) When constructing dialogue data, the user's question format is "Describe this area in the diagram".<region_i> The assistant's reply is the description text annotated by the doctor (e.g., "negative cervical cells, cell type Epitheliium Normal, derived from the surface of the cervical squamous epithelium").

[0081] The third step is to create a cell type classification dataset: Cell type classification is a region-level classification task, meaning that for each candidate region...<region_i> Determine the cell type to which it belongs.

[0082] (1) The pathologist marked the cell types in the region of interest in the image. There are five types of labels: "ASCUS" (atypical squamous cells with unclear significance), "ASC_H" (atypical squamous cells do not rule out high-grade lesions), "EpitheliumNormal" (normal epithelial cells), "HSIL" (high-grade squamous intraepithelial lesion), and "LSIL" (low-grade squamous intraepithelial lesion).

[0083] (2) When constructing dialogue data, the user's question format is "<region_i> What type of cell is it? Answer: ASCUS ASC_H EpitheliumNormal HSIL LSIL. The assistant will directly output the corresponding category label.

[0084] Step 4: Create a dataset for classifying the source of the slices. The slide origin classification is an image-level classification task, which is to determine the cell origin type of a whole cervical cell pathology image.

[0085] (1) The pathologist marked the source of the whole image, with four categories of labels: "epithelial cluster", "middle layer of cervical squamous epithelium", "basal layer of cervical squamous epithelium" and "surface layer of cervical squamous epithelium".

[0086] (2) When constructing dialogue data, the user asks in the format of "What is the origin of the cells in the picture? Answer: Epithelial clusters, cervical squamous epithelium, medial layer, basal layer, and superficial layer of cervical squamous epithelium", and the assistant answers and outputs the corresponding source category.

[0087] Step 5: Create a diagnostic reasoning dataset: The diagnostic reasoning dataset is used to train models to generate structured thought chain text.

[0088] (1) The thought chain text was generated by GPT-4 based on the Bethesda System (2014 version) and reviewed and revised by qualified pathology experts to ensure that the reasoning process conforms to clinical norms. Each thought chain strictly follows the following four steps:

Cell origin analysis

[0089] [Pathological Features Observation]: Describe in detail the size of the cell nucleus (whether it is enlarged, and approximately how many times larger it is than a normal cell nucleus), the nuclear-cytoplasmic ratio (whether it is increased), the nuclear chromatin (whether it is increased, whether the distribution is uniform, and whether it is deeply stained), the nuclear membrane (whether it is smooth or irregular), the presence of perinuclear halos (hollowed-out cells), the presence of binucleates or multinucleates, and the presence of nuclear crowding, etc.

[0090] [Differential Diagnosis]: Based on the above characteristics, first exclude normal epithelial cells (Epithelium Normal) and give the reason for exclusion; then, according to the TBS criteria, exclude other possible types in turn (such as ASCUS, ASC_H, HSIL, LSIL), and each exclusion step should cite specific pathological features as the basis.

[0091] [Diagnostic Conclusion]: The final cell type diagnosis is given (ASCUS, ASC_H, Epithelium Normal, HSIL or LSIL).

[0092] (2) When constructing dialogue data, the user's question format is "<region_i> What type of cell is it? Answer: ASCUS ASC_H Epithelium Normal HSIL LSIL. Please provide the diagnostic basis. The assistant's answer will output the above structured thought chain text.

[0093] Taking object detection as an example, the format of the dataset obtained by the above steps can be found in [reference needed]. Figure 2 Each sample contains the following fields in order: task_id, image_id, image_path, regions, and conversations. The regions information includes the region_id, the bounding box corresponding to the index, and the confidence score.

[0094] Based on the dataset above, the multi-task functionality that the final model can achieve can be demonstrated in [link to demonstration]. Figure 3 The "Regional Fine-Grained Perception Task" includes multi-target cell detection, pathological feature representation understanding, and region description generation. These three tasks focus on the model's ability to perceive the morphology of cervical cell regions and are used as the training dataset for the first stage. The "Semantic Understanding and Reasoning Task" includes slide origin classification, cell type classification, and diagnostic reasoning generation. These three tasks focus on the model's ability to understand and generate semantics at multiple scales and are used as the main dataset for the second stage.

[0095] Step 2, First Phase Training: Focus on Perceptual Task Training The training objective in this stage is to achieve domain semantic alignment between visual features and text descriptions, enabling the model to accurately understand the global scene and candidate region content in the image and map them into the embedding space of the large language model.

[0096] This stage employs a parameter-efficient fine-tuning strategy, freezing all parameters of the pre-trained ViT visual encoder and freezing the parameters of the remaining network layers in the large language model decoder, except for the attention layer and the MLP projection layer. Only new or trainable modules are updated with their parameters.

[0097] Step 21: Construct the first-stage training input.

[0098] Based on the joint dataset constructed in step 1, the localization dataset (task_id=0, 1) and the region description dataset (task_id=2) are selected as the training data for this stage. When each data point is input into the model, a region token is obtained through a multi-scale region feature extraction network and a hierarchical attention-gated encoder. The image token, region token, and text prompt token are concatenated in sequence to form a unified multimodal input sequence.

[0099] Step 22: Set up trainable modules and freeze strategy.

[0100] See Figure 4 In part (a), the trainable modules in this stage include: a global image feature projection layer, a region image feature projection layer, a hierarchical attention-gated encoder (including a cross-layer attention module and an adaptive gating module), and the attention layer and MLP layer of the large language model decoder. The frozen modules include: the ViT visual encoder and the remaining network layers in the large language model decoder except for the attention layer and the MLP layer.

[0101] Step 23: Optimize using a unified text cross-entropy loss function. The loss function takes the following form:

[0102] Where T is the length of the output sequence. For the t-th target token, It is a sequence of visual features, containing image tokens and region tokens. The token sequence for task prompt text.

[0103] Step 24: Set training hyperparameters. The preferred settings for training hyperparameters in this stage are shown in the table below. During training, only the trainable modules specified in Step 22 participate in gradient updates; the rest remain frozen.

[0104]

[0105] Step 25: Iterate training until convergence. Repeat the following sub-steps: Step 251: Sample a batch of multimodal input sequences from the training data and input them into the model; Step 252: The model generates an output sequence token by token using an autoregressive approach and calculates the cross-entropy loss; Step 253: Pass the loss function value back to the trainable module; Step 254: Update the network parameters of the global image feature projection layer, the region image feature projection layer, the hierarchical attention gating encoder, the attention layer of the large language model decoder, and the MLP layer according to the loss function value; Step 255: Iterate through steps 251 to 254 until the loss function value stabilizes or reaches the preset number of training rounds.

[0106] Step 3, Second Phase Training: Multi-task Adaptation Training The training objective of this stage is to further adapt the model to all six types of tasks, building upon the perceptual task alignment in the first stage, to achieve performance balance and synergistic improvement across multiple tasks, while preventing catastrophic forgetting.

[0107] Step 31: Construct the second-stage training data.

[0108] We used all six types of task data constructed in step 1, including the perception task data used in the first stage. Each type of data was sampled evenly according to task type to maintain a roughly balanced number of samples for each task in each training batch, avoiding model bias towards tasks with larger sample sizes.

[0109] Step 32: Set up trainable modules and freeze strategy.

[0110] See Figure 4 In part (b), the ViT visual encoder, global image feature projection layer, and region image feature projection layer are frozen in this stage to preserve the visual-text alignment capability learned in the first stage. Only the LoRA method is used to fine-tune the multi-head attention layer of the large language model decoder, and the hierarchical attention-gated encoder is fine-tuned with a small learning rate. The optimal hyperparameter settings are shown in the table below:

[0111] Step 33: The loss function remains the cross-entropy loss function from Step 23, without introducing any additional loss terms. All tasks are optimized under a unified next token prediction paradigm, eliminating the need to design independent loss functions for different tasks.

[0112] Step 34: Iterate through the training until convergence. Repeat the following sub-steps: Step 351: Sample a batch of multimodal input sequences from all six types of task data in an even proportion; Step 352: Input the sampled data into the model, generate the output sequence token by token in an autoregressive manner, and calculate the cross-entropy loss; Step 353: Pass the loss function value back to the trainable module (LoRA parameters and hierarchical attention gating encoder parameters). Step 354: Update the LoRA low-rank matrix parameters and hierarchical attention gating encoder parameters based on the loss function value; Step 355: Iterate through steps 351 to 354 until the model’s validation set performance on all tasks stabilizes or reaches the preset number of training rounds.

[0113] Step 36: After completing the two-stage training, save the model weights as a pre-trained cervical cell pathology model. Within a unified autoregressive framework, this model can flexibly switch output modes based on user queries: outputting region reference labels when the user requests localization or comprehension; outputting natural language descriptive text when a description is requested; outputting category labels when classification is requested; and outputting structured thought chain text when diagnostic criteria are required. All tasks share the same set of model parameters, eliminating the need for task-specific network branches or post-processing logic.

[0114] To facilitate understanding of the technical effects of this invention, a comparative experiment was conducted on a self-built cervical cell pathology image dataset. The dataset contains 8255 cervical cell smear images prepared using LBC technology and stained with Papanicolaou, covering typical lesion types such as ASC-US, LSIL, and HSIL. The experiment consisted of two parts: the first part compared the model with a dedicated target detector to evaluate its localization accuracy; the second part compared the model with mainstream general-purpose visual language models to evaluate its comprehensive performance across multiple tasks.

[0115] Part 1: Comparison with Dedicated Target Detectors YOLOv8, Grounding DINO, and Faster R-CNN were selected as comparison methods. Evaluation metrics included mean average precision (mAP), precision, and recall. Experimental results are shown in the table below:

[0116] As shown in the table above, the method of this invention achieves an mAP of 0.650, significantly higher than YOLOv8 (0.442), Grounding DINO (0.563), and Faster R-CNN (0.437). In terms of precision, the method of this invention achieves 0.910, far exceeding YOLOv8 (0.642) and Faster R-CNN (0.280), indicating that the method of this invention has high accuracy in predicting cell regions and a low false positive rate. In terms of recall, the method of this invention achieves 0.907, basically on par with Faster R-CNN (0.930) and slightly higher than YOLOv8 (0.645), demonstrating that the method of this invention maintains high precision without sacrificing the ability to detect positive cells, achieving a good balance between precision and recall. It is worth noting that although Faster R-CNN has the highest recall (0.930), its precision is only 0.280, which means that a large number of predictions are false positives. In contrast, the method of this invention improves precision by more than 3 times while having a similar recall, which verifies the superiority of this method in multi-object detection scenarios.

[0117] Part Two: Comparison with Visual Language Models Qwen2.5-VL (3.1B), LLaVA-v1.6 (7.0B), InternVL2 (4.0B), and LLaVA-Med (3.0B) were selected as comparison methods. For object detection (OD) and referring expression comprehension (REC) tasks, F1 and mAP metrics were used; for region captioning (RC) tasks, BLEU metrics were used; and for cell type classification and slide origin classification, accuracy metrics were used. Experimental results are shown in Table 2.

[0118] As shown in Table 2, the method of this invention achieves optimal performance across all perception tasks. In the localization task, the F1 and mAP scores of the method of this invention reach 0.912 and 0.651, respectively, significantly higher than the best-performing general VLM, Qwen2.5-VL (0.456 and 0.052, respectively). In the representation expression understanding task (REC), the F1 and mAP scores of the method of this invention also far exceed those of other VLM models, indicating that the model can accurately understand natural language descriptions and locate the corresponding cell regions. In the region description task, the BLEU score of the method of this invention is 0.849, a significant improvement over the second-best, LLaVA-v1.6 (0.701), indicating that the descriptive text generated by the model is closer to the professional expressions of pathologists. In cell type classification, the method of this invention achieves an accuracy of 0.692, outperforming models of similar size (Qwen2.5-VL and LLaVA-Med) and larger size (InternVL and LLaVA-v1.6), demonstrating that the method achieves competitive classification performance with a smaller model size. In slice origin classification, the method achieves an accuracy of 0.790, also outperforming other general-purpose VLMs and medical-specific VLMs (LLaVA-Med), indicating that the method also possesses good capabilities in image-level classification tasks.

[0119] The experimental results above demonstrate that the method of this invention significantly outperforms dedicated detectors in terms of localization accuracy while achieving a balance between precision and recall, thus validating the effectiveness of the "localization accuracy" characteristic of this invention. In terms of multi-task collaboration, the method of this invention, with only 3B parameters, comprehensively surpasses general-purpose VLM models with larger parameter counts, as well as general-purpose and medical-specific VLM models of similar scale, in tasks such as localization, referential interpretation, region description, and cell classification, validating the effectiveness of the "multi-task collaboration" characteristic of this invention. In summary, the cervical cell pathology model proposed in this invention can provide reliable and interpretable technical support for intelligent assisted diagnosis of cervical cytology.

[0120] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0121] The following describes the cervical cell pathology image analysis device, non-transitory computer-readable storage medium, computer program product, and electronic device based on a visual language model provided by the present invention. The cervical cell pathology image analysis device, non-transitory computer-readable storage medium, computer program product, and electronic device based on a visual language model described below can be referred to in correspondence with the cervical cell pathology image analysis method based on a visual language model described above.

[0122] The present invention also provides a cervical cell pathology image analysis device based on a visual language model, comprising: The image acquisition module is used to acquire pathological images of cervical cells to be analyzed. The multi-task analysis module is used to input the cervical cell pathology images to be analyzed and the task prompts into the pre-trained cervical cell pathology model and output the multi-task analysis results. The cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal module generates detection boxes for candidate regions based on natural language prompts. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features. The hierarchical attention-gated encoder selectively fuses the multi-scale region representations. The large language model decoder performs multi-task analysis in a unified manner to obtain multi-task analysis results.

[0123] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the cervical cell pathological image analysis method based on a visual language model as described in Embodiment 1.

[0124] The electronic device may include a processor, a communications interface, memory, and a communication bus. The processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute a cervical cell pathology image analysis method based on a visual language model, primarily including the software processing steps described above.

[0125] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the software processing part of the cervical cell pathology image analysis method based on the visual language model provided by the above methods.

[0127] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the software processing portion of the cervical cell pathology image analysis method based on the visual language model provided by the above methods.

[0128] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations. < / objects> < / ground> < / regionx> < / regionx> < / region1> < / region0> < / regionx>

Claims

1. A method for analyzing cervical cell pathology images based on a visual language model, characterized in that, include: Obtain pathological images of cervical cells to be analyzed; The cervical cell pathology image to be analyzed and the task prompts are input into the pre-trained cervical cell pathology model, and the multi-task analysis results are output. The cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal module generates detection boxes for candidate regions based on natural language prompts. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features. The hierarchical attention-gated encoder selectively fuses the multi-scale region representations. The large language model decoder performs multi-task analysis in a unified manner to obtain multi-task analysis results.

2. The cervical cell pathology image analysis method based on a visual language model as described in claim 1, characterized in that, The region proposal module generates detection boxes for candidate regions based on natural language prompts, including: The region proposal module uses a target detection model finely tuned on the cervical cell dataset. It takes natural language prompts as input and outputs the bounding box coordinates and confidence scores of candidate cell regions.

3. The cervical cell pathology image analysis method based on a visual language model as described in claim 1, characterized in that, Multi-scale region feature extraction networks construct multi-scale region representations from single-scale ViT features, including: Using the feature map output by a single-scale ViT as input, feature levels of four scales are constructed through upsampling and lateral connection. For each scale feature map and each candidate box, perform a region of interest alignment operation to obtain a region feature block with a fixed spatial size; Spatial pooling is performed on the regional feature blocks to obtain the regional representation at the corresponding scale.

4. The cervical cell pathology image analysis method based on a visual language model as described in claim 1, characterized in that, The hierarchical attention-gated encoder includes a cross-layer attention module and an adaptive gating module. It selectively fuses multi-scale region representations, including: The cross-layer attention module is used to aggregate information layer by layer from the regional representations at different scales. By using an adaptive gating module to dynamically weight the contributions of features at each scale based on the Sigmoid function, and fusing task biases, scale preference adjustment can be achieved under multi-task conditions.

5. The cervical cell pathology image analysis method based on a visual language model as described in claim 1, characterized in that, The large language model decoder performs multi-task analysis uniformly, obtaining multi-task analysis results, including: The large language model decoder takes a sequence of concatenated image tokens, region tokens, and text tokens as input, unifying all tasks into a next-to-next-to-toe prediction paradigm. This paradigm is used to simultaneously perform abnormal cell localization, region semantic description, lesion and slide origin classification, and clinical reasoning tasks, executing multiple reasoning tasks to obtain multi-task analysis results. Specifically, the image token is obtained by ViT encoding the input image, the region token is obtained by feature extraction and encoding of candidate regions, and the text token is obtained by encoding task text prompts. Unifying all tasks into the next-to-next-to-toe prediction paradigm includes: cell target localization by generating reference tags; region semantic description by generating natural language description text corresponding to the region; cell lesion classification by generating category label text; and clinical reasoning by generating structured thought chain text.

6. The cervical cell pathology image analysis method based on a visual language model as described in claim 1, characterized in that, The pre-trained cervical cell pathology model also includes a global image feature projection layer and a regional image feature projection layer. The pre-trained cervical cell pathology model is obtained using a two-stage training strategy, which includes: The first stage focuses on training for perception tasks, optimizing the global image feature projection layer, hierarchical attention-gated encoder, regional image feature projection layer, and attention layer of the large language model decoder through an efficient parameter fine-tuning strategy. The second stage extends to training for semantic understanding and reasoning tasks, freezing the visual encoding module, using LoRA to train the attention layer of the large language model decoder, fine-tuning the hierarchical attention-gated encoder, and introducing some training data from the first stage while injecting the ability of thought chains. The visual encoding module includes ViT, the global image feature projection layer, and the regional image feature projection layer.

7. The cervical cell pathology image analysis method based on a visual language model as described in claim 6, characterized in that, The training process uses a uniform text cross-entropy loss function: in, To output the sequence length, For the t-th target token, This represents all text tokens generated up to the current position. This is a visual feature sequence, containing image tokens and region tokens. The image token is obtained by encoding the input image using ViT, and the region token is obtained by extracting and encoding features from candidate regions. The task prompt text is a token sequence. Given visual features, task cues, and all text generated up to the current position, this represents the predicted current position. The logarithm of the probability of a target token at a given location.

8. A cervical cell pathology image analysis device based on a visual language model, characterized in that, include: The image acquisition module is used to acquire pathological images of cervical cells to be analyzed. The multi-task analysis module is used to input the cervical cell pathology image to be analyzed and the task prompt words into the pre-trained cervical cell pathology model and output the multi-task analysis results. The cervical cell pathology model includes a region proposal module, a multi-scale region feature extraction network, a hierarchical attention-gated encoder, and a large language model decoder. The region proposal module generates detection boxes for candidate regions based on natural language prompts. The multi-scale region feature extraction network constructs multi-scale region representations from single-scale ViT features. The hierarchical attention-gated encoder selectively fuses the multi-scale region representations. The large language model decoder performs multi-task analysis in a unified manner to obtain multi-task analysis results.

9. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the cervical cell pathology image analysis method based on a visual language model as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cervical cell pathology image analysis method based on a visual language model as described in any one of claims 1 to 7.

Images