A pathological section image analysis method and device based on a large language model

Abstract

The application provides a pathological section image analysis method and device based on a large language model, and belongs to the fields of computer vision, deep learning and medical image analysis. The method comprises the following steps: constructing an image coding large model for pathological sections composed of a pre-training model and an additionally added task head; freezing the pre-training model, training the task head by using a preset training set and testing by using a preset verification set to obtain a preliminarily trained task head; unfreezing the pre-training model, fine-tuning training the large model by using the training set and testing by using the verification set to obtain a finally trained large model; inputting a pathological section image to be analyzed into the large model, calculating corresponding features of the image based on the result output by the large model to obtain a final analysis result. The application can automatically extract features in the pathological section image and generate a diagnosis report based on the features, so that the accuracy and efficiency of pathological section image diagnosis are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and medical image analysis, and particularly to a method and apparatus for pathological slide image analysis based on a large language model. Background Technology

[0002] Pathology is an indispensable branch of medicine. It provides crucial information for disease diagnosis, classification, and treatment by studying the morphological changes of tissue sections and cell samples under a microscope. Pathological slide image analysis, as an important component of pathology, reveals the microscopic characteristics of diseases through meticulous observation and analysis of pathological slides, providing a scientific basis for clinical decision-making. However, traditional pathological slide image analysis relies heavily on the professional knowledge and experience of pathologists. This method is not only time-consuming and laborious but also prone to misdiagnosis and missed diagnosis due to human subjectivity.

[0003] With the rapid development of computer vision and artificial intelligence technologies, especially breakthroughs in deep learning algorithms, new possibilities have emerged for the automated analysis of pathological slide images. Deep learning models, particularly Convolutional Neural Networks (CNNs), have demonstrated unprecedented performance in image recognition and classification tasks. CNNs can automatically learn complex feature representations from raw images, which play a crucial role in identifying and distinguishing different pathological structures. However, traditional CNNs often require a large amount of labeled data for training when dealing with large-scale and high-resolution pathological images, and their ability to recognize fine structures and complex patterns in images is limited.

[0004] To address these challenges, Large Language Models (LLMs) have been introduced into pathological slide image analysis. LLMs are deep learning models based on the Transformer architecture, and they have achieved significant success in natural language processing. LLMs capture long-range dependencies through a self-attention mechanism, which is crucial for understanding the complex structures and contextual information in pathological images. Furthermore, LLMs possess strong generalization capabilities, can be trained on relatively small amounts of labeled data, and can handle images of varying scales and resolutions.

[0005] Despite the immense potential of LLMs in pathological slide image analysis, effectively utilizing them still faces several challenges. First, the sheer volume of pathological images makes efficient data preprocessing and augmentation a critical issue. Second, annotating pathological images typically requires specialized knowledge, and acquiring high-quality annotated data is costly. Furthermore, designing suitable network structures and training strategies to enable rapid model convergence with limited computational resources is also a current research focus. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a method and apparatus for pathological slide image analysis based on a large language model. This invention can automatically extract features from pathological slide images, and further analyze these images to automatically generate diagnostic reports, thereby improving the accuracy and efficiency of pathological slide image diagnosis.

[0007] A first aspect of this invention proposes a method for analyzing pathological slide images based on a large language model, comprising:

[0008] A large image coding model for pathological slides is constructed, which consists of a pre-trained model and additional task heads.

[0009] Freeze the pre-trained model, train the task head using a preset training set and verify it using a preset validation set to obtain the task head that has been initially trained.

[0010] Unfreeze the pre-trained model, fine-tune the large model using the training set, and verify it using the validation set to obtain the final trained large model.

[0011] The pathological slide image to be analyzed is input into the trained large model, and the corresponding features of the pathological slide image are calculated based on the output of the large model to obtain the analysis result of the pathological slide image.

[0012] In one specific embodiment of the present invention, the image coding large model adopts a pre-trained DinoV2 model, and a MaskFormer task header is added to the output layer of the DinoV2 model.

[0013] In one specific embodiment of the present invention, it further includes:

[0014] 1) Obtain pathological slide images to form a pathological slide image dataset;

[0015] The pathological slide image dataset contains pathological slide images corresponding to various pathological types, wherein each pathological slide image is of the same size.

[0016] 2) Label each pathological slide image in the pathological slide image dataset of step 1); wherein, the labeling content of each pathological slide image includes the polygonal boundary of the lesion area and the corresponding lesion type label.

[0017] 3) Preprocess the images in the pathological slide image dataset after step 2) annotation, and normalize the pixel values ​​of each image to the range of [0,1].

[0018] 4) The pathological slide dataset that has been preprocessed in step 3) is randomly divided into a training set and a validation set according to a preset ratio.

[0019] In one specific embodiment of the present invention, it further includes:

[0020] The performance of the trained image coding model is evaluated using a preset test set. Based on the evaluation results, the image coding model is adjusted to obtain an optimized image coding model. The pathological slide images in the test set are different from those in the training set and the validation set.

[0021] In one specific embodiment of the present invention, it further includes:

[0022] The pathological slide image to be analyzed is input into the trained large model, and the large model outputs a lesion distribution map corresponding to the input image;

[0023] The lesion distribution map is processed to obtain the features of the pathological slide image, specifically including:

[0024] Each cell or lesion region in the distribution map is marked, and its area and perimeter are calculated to estimate the diameter of the cell or the size of the lesion region. For marked cells, the number of marked cells is counted and divided by the area of ​​the corresponding region to obtain the cell density. For marked lesion regions, the proportion of the region to the distribution map or its size relative to a set reference standard is calculated to quantify the size of the lesion.

[0025] In one specific embodiment of the present invention, it further includes:

[0026] 1) Based on the output of the image coding large model, generate features of multiple pathological slide images and obtain the diagnostic report corresponding to each pathological slide image; wherein, each diagnostic report is preprocessed to obtain the corresponding text vector;

[0027] 2) The text vector and the features of the corresponding pathological slide image are concatenated to obtain the multimodal input. This multimodal input and the corresponding diagnostic report form the training sample of a large language model.

[0028] 3) Select a large language model;

[0029] 4) Use the training samples obtained in step 2) to train the large language model in step 3) to obtain the trained large language model.

[0030] In one specific embodiment of the present invention, it further includes:

[0031] The features of the pathological slide image to be analyzed, generated based on the output of the large image encoding model, are combined with text encoding to form a multimodal input. Then, the multimodal input is fed into the trained large language model to generate the final diagnostic report.

[0032] A second aspect of the present invention provides a pathological slide image analysis device based on a large language model, comprising:

[0033] The image coding large model construction module is used to build a large image coding model for pathological slides. The large model consists of a pre-trained model and additional task heads.

[0034] The task head training module is used to freeze the pre-trained model, train the task head using a preset training set and verify it using a preset validation set, so as to obtain the task head that has been initially trained.

[0035] The image coding large model training module is used to unfreeze the pre-trained model, fine-tune the large model using the training set, and verify it using the validation set to obtain the finally trained large model.

[0036] The image analysis module is used to input the pathological slide image to be analyzed into the trained large model, and calculate the corresponding features of the pathological slide image based on the output of the large model to obtain the analysis result of the pathological slide image.

[0037] A third aspect of the present invention provides an electronic device comprising:

[0038] At least one processor; and a memory communicatively connected to said at least one processor;

[0039] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to perform the above-described method for analyzing pathological slide images based on a large language model.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described method for pathological slide image analysis based on a large language model.

[0041] The features and beneficial effects of this invention are as follows:

[0042] This invention extracts features from pathological slide images using a large image coding model, and then segments, classifies, or detects these images. The processed image features are then trained using a large language model along with the analysis report and diagnostic results of the pathological image. This invention leverages the powerful feature extraction and representation capabilities of LLMs to achieve automatic feature extraction and analysis of pathological slide images. In this way, the invention aims to improve the accuracy and efficiency of pathological slide image analysis, providing doctors with a reliable auxiliary diagnostic tool. Attached Figure Description

[0043] Figure 1 This is an overall flowchart of a pathological slide image analysis method based on a large language model in an embodiment of the present invention. Detailed Implementation

[0044] This invention proposes a method and apparatus for pathological slide image analysis based on a large language model, which will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] The first aspect of this invention proposes a method for analyzing pathological slide images based on a large language model, the overall process of which is as follows: Figure 1 As shown, it includes:

[0046] A large image coding model for pathological slides is constructed, which consists of a pre-trained model and additional task heads.

[0047] Freeze the pre-trained model, train the task head using a preset training set and verify it using a preset validation set to obtain the task head that has been initially trained.

[0048] Unfreeze the pre-trained model, fine-tune the large model using the training set, and verify it using the validation set to obtain the final trained large model.

[0049] The pathological slide image to be analyzed is input into the trained large model, and the corresponding features of the pathological slide image are calculated based on the output of the large model to obtain the analysis result of the pathological slide image.

[0050] In one specific embodiment of the present invention, the pathological slide image analysis method based on a large language model includes the following steps:

[0051] 1) Construct a training set and a validation set of pathological slide images; the specific steps are as follows:

[0052] 1-1) Obtain pathological slide images to form a pathological slide image dataset.

[0053] In this embodiment, the pathological slide image dataset contains pathological slide images corresponding to various pathological types, including but not limited to cancer, inflammation, and benign lesions. The number of pathological slide images for each type should be greater than 100. All pathological slide images in the dataset have the same size; in one specific embodiment of this invention, they are all 512×512 pixels, to ensure consistency and standardization when inputting into the model.

[0054] In one specific embodiment, a pathological slide image dataset was constructed by collecting a total of 20,000 pathological slide images from multiple pathology laboratories and medical institutions. To ensure the diversity and representativeness of the dataset, these images cover different pathological features, patient backgrounds, and sample sources.

[0055] Furthermore, in this embodiment, the following points are particularly emphasized during the process of collecting slice images:

[0056] The dataset includes a wide variety of pathological types, including cancers (such as breast cancer, lung cancer, and gastric cancer), with segmentation primarily focusing on the cancerous region and its boundaries; inflammations (such as hepatitis and gastritis), with segmentation primarily focusing on the inflammatory response region and its boundaries; and benign lesions (such as benign tumors and cysts), with segmentation primarily focusing on the lesion region and its boundaries.

[0057] The sample sources are diverse. This embodiment collects samples from different geographical regions and medical institutions, covering different patient groups and pathological characteristics, to ensure the broad representativeness of the dataset.

[0058] In addition, this embodiment selects high-quality pathological slide images to ensure image clarity and detail, so that the model can learn more useful information.

[0059] 1-2) Label each pathological slide image in the pathological slide image dataset obtained in step 1-1).

[0060] In this embodiment, the annotation content of each pathological slide image includes the polygonal boundary of the lesion area and the corresponding lesion type label. This annotation data will be used for subsequent model training and validation.

[0061] In one specific embodiment of the present invention, the annotation process includes: preliminary examination, where pathology experts first conduct a preliminary examination of each pathological slide image to determine the presence and approximate location of the lesion area; detailed annotation, where professional annotation tools are used to perform detailed polygon annotation on each lesion area to accurately delineate the lesion boundary; type annotation, where each annotated lesion area is assigned a corresponding lesion type label, such as cancer, inflammation, or benign lesion; and verification, where all annotation results are verified multiple times to ensure the accuracy and consistency of the annotation and avoid mis-annotation or omission.

[0062] 1-3) Preprocess the images in the pathological slide image dataset after annotation in step 1-2).

[0063] In this embodiment of the invention, the preprocessing includes: normalizing the image pixel values ​​to the range of [0,1] to improve the training effect and convergence speed of the model.

[0064] 1-4) Divide the dataset that has been preprocessed in steps 1-3) into a training set and a validation set.

[0065] If the present invention must retain the validation set, what is the ratio between the training set and the validation set?

[0066] In one specific embodiment of the present invention, 15,000 images are randomly selected from a dataset of 20,000 pathological slide images to form a training set, and the remaining 5,000 images form a validation set. Generally, the ratio of the training set to the validation set can be 3:1.

[0067] Optionally, this embodiment can also perform data augmentation on the pathological slide image dataset in the training set, such as random cropping, rotation, and flipping, to increase the diversity of the training data and improve the robustness of the model. These preprocessing steps ensure that the pathological slide image data has a consistent format and quality when input into the model, thereby improving the training effect and performance of the model.

[0068] 2) Construct a large-scale image coding model for pathological slides.

[0069] In this embodiment, the existing pre-trained DinoV2 model is used as the main body of the large image coding model. A MaskFormer task header is added to the output layer of the DinoV2 model, and the large image coding model is completed.

[0070] It's worth noting that DinoV2 is an advanced self-supervised visual representation learning model that, through training on large-scale image datasets, can extract rich image features. DinoV2 primarily utilizes Vision Transformers (ViT), also known as visual attention methods, as its architectural foundation. Through a mechanism called "self-attention," it efficiently processes and understands the global content of an image. During the self-supervised learning process, DinoV2 employs various data augmentation techniques, such as cropping, color perturbation, and Gaussian blur, to generate multiple views of the image and allow the model to learn the relationships between these views, thereby obtaining a deeper and more robust image representation.

[0071] In this embodiment, in the large-scale image encoding model, the input to DinoV2 is a preprocessed 512×512 pixel pathological slide image, and the output is a high-dimensional image feature vector, typically a 16×16×768 feature map. This feature map is generated by DinoV2 through its deep visual feature extraction mechanism and contains rich information from the input image.

[0072] MaskFormer is an advanced image segmentation model specifically designed to handle complex image segmentation tasks. In this embodiment, the input to MaskFormer is the high-dimensional image feature vector output by DinoV2, and the output is a segmentation mask for each pixel in the pathological slide image. Specifically, the MaskFormer task head consists of three convolutional layers, each with a 3×3 kernel, and is equipped with eight self-attention heads. Each self-attention head can capture different feature dimensions of the image, enhancing the model's segmentation ability. The classification probability of each pixel is generated through the last softmax operation in the MaskFormer task head, ensuring that the output segmentation mask has accurate classification results.

[0073] 3) Train the large image coding model constructed in step 2) using the training set obtained in step 1); the specific steps are as follows:

[0074] 3-1) Freeze the pre-trained model in the large image coding model and train the task head.

[0075] In this embodiment, the pre-training parameters of the DinoV2 image encoding model are frozen to ensure that these parameters remain unchanged during this training phase. Freezing the pre-trained model's parameters preserves its feature extraction capabilities learned on large-scale datasets, preventing the degradation of these features during initial training. Training only the task head's parameters allows the task head to focus on learning specific features of pathological slide images. This allows the model to leverage the rich features learned during pre-training to enhance the task head's segmentation capabilities. The advantage of this method is that it fully utilizes the powerful feature extraction capabilities of the pre-trained model while focusing on task head optimization, thus improving the overall model's segmentation performance.

[0076] Then, the MaskFormer task head is trained using the pathological slide image training set obtained in step 1). During training, the same or similar loss function and optimization algorithm as the pre-trained model are used to optimize the parameters of the task head through backpropagation. Appropriate training parameters are set, including learning rate, training period, batch size, etc., to ensure that the task head converges stably during training.

[0077] In a specific embodiment of the present invention, the specific steps for training the MaskFormer task header are as follows:

[0078] First, the parameters of the MaskFormer task head are randomly initialized, setting the parameters required for training, including the learning rate, training period, and batch size. In one specific embodiment of this invention, the learning rate is set to 0.001, the training period is set to 50 epochs, and the batch size is set to 32. The cross-entropy loss function and the Adam optimizer are selected, and the parameters of the MaskFormer task head are optimized using the backpropagation algorithm. The cross-entropy loss function is suitable for classification tasks and can effectively measure the difference between the predicted result and the actual label, while the Adam optimizer ensures stability and fast convergence during training through adaptive learning rate adjustment.

[0079] Then, the MaskFormer task head is trained using a training set of pathological slide images. During training, the parameters of the task head are continuously adjusted to enable it to accurately segment lesion regions. Through multiple rounds of iterative training, the task head gradually learns how to identify and segment lesion regions on high-dimensional feature maps, thereby improving segmentation accuracy.

[0080] Finally, monitor the loss value and validation set performance in real time during training to ensure the stability and effectiveness of the training. When the loss value on the validation set no longer decreases significantly, consider ending the training. Monitoring the training process helps to identify overfitting in a timely manner and optimize model performance by adjusting training parameters. For example, if the initial learning rate is 0.001, it can be adjusted to 0.002 if the training process is too slow, and to 0.0005 if the training fails to converge.

[0081] Through the above steps, the MaskFormer task head can fully learn the specific features of pathological slide images based on the frozen DinoV2 pre-trained model, laying a solid foundation for subsequent full model fine-tuning. This process not only improves the model's segmentation ability but also ensures the model's accuracy and stability when processing complex pathological slide images.

[0082] Once the MaskFormer task header has been trained, proceed to step 3-2.

[0083] 3-2) Unfreeze the pre-trained model, and then fine-tune the large image encoding model; specifically including:

[0084] Unfreeze the pre-trained parameters of the DinoV2 model so that the parameters of all layers of the DinoV2 model can be updated during subsequent training.

[0085] Set the parameters needed for fine-tuning the training, including the learning rate, training epochs, and batch size. The learning rate should be set low during the fine-tuning phase to avoid large perturbations to the learned features. A commonly used learning rate is 0.0001, the training epochs can be set to 20, and the batch size can be kept at 32. Choose the cross-entropy loss function and the Adam optimizer, and optimize the parameters of the entire model using the backpropagation algorithm.

[0086] It's important to note that the learning rate setting is crucial during the fine-tuning training phase. A common approach is to gradually decrease the learning rate, starting with a higher rate to quickly adjust model parameters; as training progresses, the learning rate is gradually reduced to refine the adjustment of large model parameters. This method ensures stable convergence of the large image encoding model during fine-tuning, avoiding training oscillations or overfitting.

[0087] Through the above steps, the pre-trained model and the task head can be jointly optimized in pathological slide image segmentation tasks, improving the overall model performance. The fine-tuned image encoding model not only has powerful feature extraction capabilities but also possesses high-precision segmentation capabilities for pathological slide images, thus meeting practical application needs. The method described in this embodiment, by fully utilizing the existing knowledge of the pre-trained model and combining it with specific optimizations of the task head, can significantly improve the overall performance and adaptability of the large model, providing reliable technical support for practical pathological slide image analysis.

[0088] After fine-tuning, the final trained image coding model for pathological slides is obtained.

[0089] 3-3) Further, optionally, this embodiment can also further optimize the model by evaluating the trained image coding model for pathological slides; the specific steps are as follows:

[0090] 3-3-1) Evaluate model performance;

[0091] After fine-tuning the large model, the next step is to comprehensively evaluate it to ensure its performance in the pathological slide image segmentation task meets expectations. The evaluation process includes measuring the model's segmentation accuracy and robustness using validation and test sets. Key evaluation metrics include Intersection over Union (IoU), Pixel Accuracy (PA), and Dice coefficient, which comprehensively reflect the model's performance in the segmentation task. Specifically, these include:

[0092] First, a separate test set is prepared to ensure that the images in the test set are not used for training and validation of the image coding model, thus guaranteeing the objectivity of the evaluation results. Then, the pathological slide images from the test set are input into the trained image coding model to generate segmentation results for each pathological slide image in the test set. Based on these segmentation results, key evaluation metrics of the image coding model on the test set are calculated, including Intersection over Union (IoU), Pixel Accuracy (PA), and Dice coefficient.

[0093] Intersection over Union (IoU) measures the overlap between the predicted and actual segments, Pixel Accuracy (PA) measures the proportion of correctly classified pixels in the prediction, and the Dice coefficient evaluates the consistency between the predicted and actual segments. By combining these evaluation metrics, a comprehensive understanding of the model's segmentation performance can be obtained.

[0094] 3-3-2) Optimize and adjust the model;

[0095] Based on the evaluation results of step 3-3-1), the model is further optimized and adjusted to improve its performance in pathological slide image segmentation tasks. The optimization process includes adjusting model parameters, improving training strategies, and redesigning data augmentation methods. The goal is to gradually improve the model's segmentation accuracy and robustness through continuous iteration. Specifically, this includes:

[0096] First, analyze the evaluation results to identify the model's strengths and weaknesses. For example, if the model performs poorly on certain types of lesions, consider increasing the amount of training data for those types of lesions or performing specific parameter tuning for those types of lesions.

[0097] Then, adjust the model's hyperparameters, including learning rate, batch size, and regularization coefficient, and conduct multiple experiments to find the optimal parameter combination. Methods such as grid search or random search can be used to systematically explore the effects of different parameter combinations. Furthermore, consider improving training strategies, such as using more advanced optimization algorithms, introducing new loss functions, or increasing the diversity of training data.

[0098] Next, data augmentation methods can be improved to increase the diversity of training data and the robustness of the model. For example, more types of image transformation techniques can be introduced, such as color jitter, blurring, and noise, so that the model can maintain high performance when faced with different image changes.

[0099] Finally, further fine-tuning training is performed, gradually improving the model's segmentation performance by incorporating new data and optimization strategies. After each training iteration, the model's performance is re-evaluated, and the next round of optimization is conducted based on the evaluation results.

[0100] For example, in this embodiment, the final image coding model obtained in step 3-2) is optimized. First, independent test set pathological slice data is input into the model for lesion identification and segmentation. The lesion segmentation results are compared with the actual labeled results to calculate the pixel-level classification accuracy. Assuming that the average classification accuracy of different categories is 80%, if we want the model to better complete the recognition task and achieve a higher accuracy, such as above 90%, then the model needs to be adjusted. First, we need to find out which type of lesion has a low recognition accuracy for the large model and increase the weight of the corresponding category in the loss function so that the large model can better learn the features of that type of lesion. For example, if the classification rate of cell edema is low, the weight corresponding to edema in the loss function can be doubled. In addition, the number of different types of lesions can be counted. If the number of a certain type of lesion is too small, the large model will not be able to learn the characteristics of that type of lesion, resulting in poor classification performance. For example, if there are too few samples of cell proliferation lesions, data augmentation is needed for that type of sample. For example, the images of that type of sample can be distorted, rotated, or noise added to increase the number of samples of that type. Then, the large model can be trained. After the training is completed, the recognition accuracy is retested to continue to find parts that can be optimized.

[0101] In this embodiment, the large-scale image coding model, after fine-tuning and optimization, has good performance in extracting and coding pathological slide features, and can be applied to practical tasks such as pathological image analysis and diagnosis.

[0102] 4) Use the image coding model obtained in step 3) to process the pathological slide image to be analyzed, and calculate the corresponding features of the pathological slide image based on the output of the model; the specific steps are as follows:

[0103] 4-1) Obtain an image of a pathological slide to be analyzed.

[0104] In this embodiment, the image is included in the training set or validation set. The image size is the same as the training set image, which is 512×512 pixels. The lesion categories contained in the image should be included in the lesion categories corresponding to the training set images.

[0105] 4-2) Input the pathological slide image obtained in step 4-1) into the large image encoding model trained in step 3). The large model outputs the lesion distribution map corresponding to the input image.

[0106] 4-3) Perform image analysis processing on the lesion distribution map obtained in step 4-2) to obtain the features corresponding to the pathological slice image to be analyzed.

[0107] In this embodiment, each cell or lesion region in the distribution map is marked, and its area and perimeter are calculated. These parameters allow for the estimation of the cell diameter or the size of the lesion region. Simultaneously, for marked cells, the number of marked cells is counted and divided by the area of ​​the corresponding region to obtain the cell density; for marked lesion regions, their proportion of the entire lesion distribution map or their size relative to a certain reference standard is calculated to quantify the size of the lesion.

[0108] In this process, the Connected Component Labeling (CCL) algorithm is used to label each cell or lesion region in the distribution map. For each labeled region R... i Let i = 1, 2, ..., N, where N is the total number of labeled cells or the total number of lesion areas, and calculate its area S. i and perimeter C i The area can be obtained by calculating the number of pixels within the region, and the perimeter can be obtained by calculating the number of pixels at the region's edges. The diameter D of the cell or tissue region... i The area S of the region can be used as a reference. i The estimate is that, assuming it is circular, then... The size of the lesion region can be quantified by calculating the ratio of its area to the total image area. Let the area of ​​the lesion region be S. lesion The total area of ​​the image is S total The proportion of the lesion area is then... Cell density μ can be calculated by statistically labeling the number of cells N and dividing by the area S of the corresponding region. region To obtain, that is

[0109] Morphological operations, such as opening, closing, dilation, and erosion, are used to remove noise and pinholes from the distribution map and separate tightly connected cells. Then, cell size is represented by region area or equivalent diameter, cell shape is described by shape factors such as roundness and rectangularity, and texture features, such as contrast, energy, and entropy, are extracted using the gray-level co-occurrence matrix. Finally, the extracted feature vectors are standardized, and the most discriminative features are selected to provide a feature representation that is of significant value to pathologists' diagnostic decisions.

[0110] Correlation analysis is used to assess the relationships between features. To evaluate the relationships between features and avoid multicollinearity, this embodiment uses correlation analysis to identify highly correlated features. Correlation analysis can use the Pearson correlation coefficient r or the Spearman rank correlation coefficient ρ to measure the linear relationship between features. The Pearson correlation coefficient is defined as:

[0111]

[0112] Among them, X i and Y i Let μx and μy be the observed values ​​of two features, and μx and μy be the average values ​​of the two features. The Spearman rank correlation coefficient is defined as:

[0113]

[0114] Where d i Features X and Y i The grade difference, where n is the number of observations.

[0115] This embodiment identifies highly correlated feature pairs by calculating these correlation coefficients. For highly correlated features, one feature can be removed, or they can be combined into a new feature to reduce the impact of multicollinearity on model performance. In this way, this embodiment can extract the most valuable and discriminative information from a large number of features, providing strong support for pathologists' diagnoses.

[0116] Further, optionally, the method described in this embodiment also includes:

[0117] 5) Train a large language model.

[0118] In this embodiment, large language models such as BERT and GPT are used to encode the doctor's diagnostic report into text. The features of the pathological slide image obtained in step 4) are combined with the text encoding to form a multimodal input. A multimodal Transformer model, such as CLIP, is used for feature fusion. Then, the fused features and the doctor's diagnostic report are used as training data to train the large language model. The specific steps are as follows:

[0119] 5-1) Prepare training data for the large language model.

[0120] This approach integrates features extracted from pathological slide images from doctors' diagnostic reports and the output of a large image-coding model. For text data, preprocessing operations such as word segmentation and stop word removal are performed to convert the processed text into word vectors, using tools like Word2Vec or GloVe, enabling the model to learn word semantics and contextual relationships. For image features, normalization or standardization is applied to ensure data consistency and comparability. Then, the text vectors and image features are concatenated to obtain a multimodal input. This multimodal input, along with the corresponding diagnostic report, constitutes the training samples for a large language model.

[0121] In this embodiment, to ensure that the dimensions of the concatenated vector match the input requirements of the large language model, it may be necessary to adjust the dimensions of the text data embedding vector and the image feature vector. If the image feature vector is longer than the text data embedding vector, zero padding or pooling can be considered to reduce the dimensionality of the image features. The concatenated input vector will contain information from both text and images, better representing the diversity of the data.

[0122] Through the preprocessing steps described above, this embodiment effectively integrates the features of doctors' diagnostic reports and pathological slide images, providing high-quality training data for subsequent large language model training. In this embodiment, the number of samples in the integrated training data should be as large as possible; the larger the number, the richer the doctors' diagnostic experience and logic that the model can learn.

[0123] 5-2) Select a suitable large language model as the base model, such as BERT, GPT, etc.

[0124] In this embodiment of the invention, the GPT large model is selected as the base model. The GPT model adopts the encoder-decoder architecture of Transformer and consists of multiple Transformer modules. The Transformer module includes a self-attention layer and a feed-forward neural network layer.

[0125] 5-3) Train the large language model using the training data obtained in step 5-1). During training, the model parameters are adjusted by optimizing the loss function. Commonly used loss functions include cross-entropy loss and mean squared error. Optimization algorithms such as batch gradient descent and Adam can be used to optimize the model parameters during training.

[0126] 5-4) Further, during training, the performance of the large language model can be evaluated using a validation set, and the model can be tuned based on the evaluation results. Tuning may include adjusting the learning rate, changing training strategies, etc., to improve the model's accuracy and generalization ability. After training is complete, the model is evaluated using a test set of the large language model. Evaluation metrics may include accuracy, recall, F1 score, etc., to comprehensively evaluate the model's performance. In addition, visualization tools can be used to display the comparison between the model's prediction results and actual results, in order to further analyze and improve the model.

[0127] For example, the samples obtained in step 5-1) can be shuffled and randomly divided into a training set and a validation set in an 8:1 ratio, with no overlap between the two sets. After each round of training, the data from the validation set can be input into the model for performance evaluation. The accuracy and recall of the model on the validation set can be calculated to analyze the model's training progress. If there is no significant improvement after multiple rounds of validation set testing, the large language model can be considered to have basically converged, and training can be stopped early. If the accuracy of the large language model on the validation set remains low and improves slowly, the learning rate can be increased, for example, by doubling it. Furthermore, the accuracy obtained from the validation set evaluation after each round of training can be plotted to analyze the model's performance during the training process.

[0128] Ultimately, the trained large language model can learn the correlation between pathological slide features and diagnostic reports, and can be applied to practical tasks such as pathological image analysis and diagnosis.

[0129] 6) Generate a diagnostic report.

[0130] In this embodiment, the features of the pathological slide image to be analyzed obtained in step 4) are combined with text encoding to form a multimodal input. Then, this multimodal input is fed into the trained large language model to generate the final diagnostic report.

[0131] To implement the above embodiments, a second aspect of the present invention proposes a pathological slide image analysis device based on a large language model, comprising:

[0132] The image coding large model construction module is used to build a large image coding model for pathological slides. The large model consists of a pre-trained model and additional task heads.

[0133] The task head training module is used to freeze the pre-trained model, train the task head using a preset training set and verify it using a preset validation set, so as to obtain the task head that has been initially trained.

[0134] The image coding large model training module is used to unfreeze the pre-trained model, fine-tune the large model using the training set, and verify it using the validation set to obtain the finally trained large model.

[0135] The image analysis module is used to input the pathological slide image to be analyzed into the trained large model, and calculate the corresponding features of the pathological slide image based on the output of the large model to obtain the analysis result of the pathological slide image.

[0136] It should be noted that the foregoing explanation of an embodiment of a pathological slide image analysis method based on a large language model also applies to a pathological slide image analysis device based on a large language model in this embodiment, and will not be repeated here. According to an embodiment of the present invention, a pathological slide image analysis device based on a large language model constructs a large image coding model for pathological slides. This large model consists of a pre-trained model and additional task heads. The pre-trained model is frozen, and the task heads are trained using a preset training set and validated using a preset validation set to obtain a pre-trained task head. The pre-trained model is unfrozen, and the large model is fine-tuned using the training set and validated using the validation set to obtain a finally trained large model. The pathological slide image to be analyzed is input into the trained large model, and the corresponding features of the pathological slide image are calculated based on the output of the large model to obtain the analysis result of the pathological slide image. This enables automated extraction of features from pathological slide images, and further allows for analysis of pathological slide images and automatic generation of diagnostic reports, thereby improving the accuracy and efficiency of pathological slide image diagnosis.

[0137] To implement the above embodiments, a third aspect of the present invention provides an electronic device, comprising:

[0138] At least one processor; and a memory communicatively connected to said at least one processor;

[0139] The memory stores instructions that can be executed by the at least one processor, and the instructions are configured to perform the above-described method for analyzing pathological slide images based on a large language model.

[0140] To implement the above embodiments, a fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the above-described method for pathological slide image analysis based on a large language model.

[0141] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform a pathological slide image analysis method based on a large language model according to the above embodiments.

[0143] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0146] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0148] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0149] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for analyzing pathological slide images based on a large language model, characterized in that, include: A large image coding model for pathological slides is constructed, which consists of a pre-trained model and additional task heads. Freeze the pre-trained model, train the task head using a preset training set and verify it using a preset validation set to obtain the task head that has been initially trained. Unfreeze the pre-trained model, fine-tune the large model using the training set, and verify it using the validation set to obtain the final trained large model. The pathological slide image to be analyzed is input into the trained large model, and the large model outputs a lesion distribution map corresponding to the input image; The lesion distribution map is processed to obtain the features of the pathological slide image, and then the analysis results of the pathological slide image are obtained. Specifically, the features of obtaining the pathological slide image include: Each cell or lesion region in the distribution map is marked, and its area and perimeter are calculated to estimate the diameter of the cell or the size of the lesion region. For marked cells, the number of marked cells is counted and divided by the area of ​​the corresponding region to obtain the cell density. For marked lesion regions, the proportion of the region to the distribution map or its size relative to a set reference standard is calculated to quantify the size of the lesion.

2. The method according to claim 1, characterized in that, The image encoding large model uses a pre-trained DinoV2 model, and a MaskFormer task header is added to the output layer of the DinoV2 model.

3. The method according to claim 1, characterized in that, Also includes: 1) Obtain pathological slide images to form a pathological slide image dataset; The pathological slide image dataset contains pathological slide images corresponding to various pathological types, wherein each pathological slide image is of the same size. 2) Label each pathological slide image in the pathological slide image dataset of step 1); wherein, the labeling content of each pathological slide image includes the polygonal boundary of the lesion area and the corresponding lesion type label; 3) Preprocess the images in the pathological slide image dataset after step 2) annotation, normalizing the pixel values ​​of each image to the range of [0, 1]; 4) The pathological slide dataset that has been preprocessed in step 3) is randomly divided into a training set and a validation set according to a preset ratio.

4. The method according to claim 1, characterized in that, Also includes: The performance of the trained image coding model is evaluated using a preset test set. Based on the evaluation results, the image coding model is adjusted to obtain an optimized image coding model. The pathological slide images in the test set are different from those in the training set and the validation set.

5. The method according to claim 1, characterized in that, Also includes: 1) Based on the output of the image coding large model, generate features of multiple pathological slide images and obtain the diagnostic report corresponding to each pathological slide image; wherein, each diagnostic report is preprocessed to obtain the corresponding text vector; 2) The text vector and the features of the corresponding pathological slide image are concatenated to obtain the multimodal input. This multimodal input and the corresponding diagnostic report form the training sample of a large language model. 3) Select a large language model; 4) Use the training samples obtained in step 2) to train the large language model in step 3) to obtain the trained large language model.

6. The method according to claim 5, characterized in that, Also includes: The features of the pathological slide image to be analyzed, generated based on the output of the image coding large model, are combined with text coding to form a multimodal input; Then, the multimodal input is fed into the trained large language model to generate the final diagnostic report.

7. A pathological slide image analysis device based on a large language model, characterized in that, include: The image coding large model construction module is used to build a large image coding model for pathological slides. The large model consists of a pre-trained model and additional task heads. The task head training module is used to freeze the pre-trained model, train the task head using a preset training set and verify it using a preset validation set, so as to obtain the task head that has been initially trained. The image coding large model training module is used to unfreeze the pre-trained model, fine-tune the large model using the training set, and verify it using the validation set to obtain the finally trained large model. The image analysis module is used to input the pathological slide image to be analyzed into the trained large model, and the large model outputs the lesion distribution map corresponding to the input image; The lesion distribution map is processed to obtain the features of the pathological slide image, and then the analysis results of the pathological slide image are obtained. Specifically, the features of obtaining the pathological slide image include: Each cell or lesion region in the distribution map is marked, and its area and perimeter are calculated to estimate the diameter of the cell or the size of the lesion region. For marked cells, the number of marked cells is counted and divided by the area of ​​the corresponding region to obtain the cell density. For marked lesion regions, the proportion of the region to the distribution map or its size relative to a set reference standard is calculated to quantify the size of the lesion.

8. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-6.

Images