Pathological section image analysis method and device based on large language model

Abstract

The invention provides a pathological section image analysis method and device based on a large language model, and belongs to the field of computer vision, deep learning and medical image analysis. The method comprises the following steps: constructing an image coding large model which is composed of a pre-training model and an additionally-added task head and aims at a pathological section; freezing the pre-training model, training a task head by using a preset training set, and checking by using a preset verification set to obtain a task head after preliminary training; unfreezing the pre-training model, performing fine tuning training on the large model by using the training set, and performing inspection by using the verification set so as to obtain a finally trained large model; and inputting a to-be-analyzed pathological section image into the large model, and calculating corresponding features of the image based on a result output by the large model to obtain a final analysis result. According to the method, the features in the pathological section image can be automatically extracted, and the diagnosis report is generated based on the features, so that the accuracy and efficiency of pathological section image diagnosis are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning and medical image analysis, and in particular to a pathological slice image analysis method and device based on a large language model. Background Art

[0002] Pathology is an indispensable branch of medicine. It provides key information for the diagnosis, classification and treatment of diseases by studying the morphological changes of tissue sections and cell samples under a microscope. Pathology section image analysis, as an important part of pathology, can reveal the microscopic characteristics of diseases through careful observation and analysis of pathology sections, providing a scientific basis for clinical decision-making. However, traditional pathology section image analysis mainly relies on the professional knowledge and experience of pathologists. This method is not only time-consuming and laborious, but also prone to misdiagnosis and missed diagnosis due to human subjective factors.

[0003] The rapid development of computer vision and artificial intelligence technologies, especially the breakthrough of deep learning algorithms, has provided new possibilities for the automatic analysis of pathological slice images. Deep learning models, especially convolutional neural networks (CNNs), have demonstrated unprecedented performance in image recognition and classification tasks. CNNs can automatically learn complex feature representations from raw images, which play a key role in identifying and distinguishing different pathological structures. However, traditional CNNs often require a large amount of labeled data to train when facing large-scale and high-resolution pathological images, and have limited ability to recognize small structures and complex patterns in images.

[0004] To solve these problems, Large Language Models (LLMs) have been introduced into pathology image analysis. LLMs are deep learning models based on the Transformer architecture, and they have achieved great success in the field of natural language processing. LLMs can capture long-range dependencies through the self-attention mechanism, which is crucial for understanding the complex structure and contextual information in pathology images. In addition, LLMs have strong generalization capabilities, can be trained on less labeled data, and can process images of different scales and resolutions.

[0005] Although LLMs have great potential in pathological image analysis, how to effectively use LLMs still faces a series of challenges. First, the amount of pathological image data is huge, and how to efficiently preprocess and enhance the data is a key issue. Second, the annotation of pathological images usually requires professional knowledge, and it is costly to obtain high-quality annotated data. In addition, how to design a suitable network structure and training strategy so that the model can converge quickly under limited computing resources is also a hot topic in current research. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a pathological slice image analysis method and device based on a large language model. The present invention can automatically extract features from pathological slice images, further analyze pathological slice images, and automatically generate diagnostic reports to improve the accuracy and efficiency of pathological slice image diagnosis.

[0007] The first embodiment of the present invention proposes a pathological slice image analysis method based on a large language model, comprising:

[0008] Constructing a large image encoding model for pathological sections, wherein the large model consists of a pre-trained model and an additional task head;

[0009] Freeze the pre-trained model, train the task head using a preset training set and test it using a preset validation set to obtain the task head that has been preliminarily trained;

[0010] Unfreeze the pre-trained model, use the training set to fine-tune the large model and use the validation set to test it, so as to obtain the finally trained large model;

[0011] The pathological slice image to be analyzed is input into the trained large model, and the corresponding features of the pathological slice image are calculated based on the results output by the large model to obtain the analysis results of the pathological slice image.

[0012] In a specific implementation of the present invention, the image coding model adopts a pre-trained DinoV2 model, and a MaskFormer task header is added to the output layer of the DinoV2 model.

[0013] In a specific implementation of the present invention, it also includes:

[0014] 1) Acquire pathological slice images to form a pathological slice image dataset;

[0015] The pathological slice image dataset includes pathological slice images corresponding to a plurality of pathological types, wherein each pathological slice image has a consistent size;

[0016] 2) annotating each pathological slice image of the pathological slice image data set of step 1); wherein the annotation content of each pathological slice image includes a polygonal boundary of the lesion area and a corresponding lesion type label;

[0017] 3) Preprocessing the images of the pathological slice image dataset annotated in step 2) to normalize the pixel value of each image to the range of [0, 1];

[0018] 4) The pathological section data set preprocessed in step 3) is randomly divided into a training set and a validation set according to a preset ratio.

[0019] In a specific implementation of the present invention, it also includes:

[0020] The performance of the trained image coding large model is evaluated using a preset test set, and the image coding large model is adjusted according to the evaluation result to obtain the optimized image coding large model; wherein the pathological slice images in the test set are different from the pathological slice images in the training set and the validation set.

[0021] In a specific implementation of the present invention, it also includes:

[0022] Inputting the pathological slice image to be analyzed into the trained large model, and the large model outputs a lesion distribution map corresponding to the input image;

[0023] Processing the lesion distribution map to obtain the characteristics of the pathological slice image specifically includes:

[0024] Each cell or lesion area in the distribution map is marked, and its area and perimeter are calculated to estimate the diameter of the cell or the size of the lesion area; for the marked cells, the number of marked cells is counted and divided by the area of ​​the corresponding area to obtain the cell density; for the marked lesion area, the proportion of the area in the distribution map or the size relative to the set reference standard is calculated to quantify the size of the lesion.

[0025] In a specific implementation of the present invention, it also includes:

[0026] 1) generating features of a plurality of pathological slice images based on the output result of the image coding large model and obtaining a diagnosis report corresponding to each pathological slice image; wherein each diagnosis report is preprocessed to obtain a corresponding text vector;

[0027] 2) The text vector and the features of the corresponding pathological slice image are concatenated to obtain a multimodal input, which together with the corresponding diagnosis report constitutes a training sample for a large language model;

[0028] 3) Select a large language model;

[0029] 4) Using the training samples obtained in step 2), the large language model obtained in step 3) is trained to obtain a trained large language model.

[0030] In a specific implementation of the present invention, it also includes:

[0031] The features of the pathological slice image to be analyzed generated based on the output result of the image encoding large model are combined with the text encoding to form a multimodal input; then, the multimodal input is sent to the trained large language model to generate a final diagnosis report.

[0032] The second embodiment of the present invention provides a pathological slice image analysis device based on a large language model, comprising:

[0033] An image coding large model construction module is used to construct an image coding large model for pathological sections, wherein the large model is composed of a pre-trained model and an additional task head;

[0034] A task head training module, used to freeze the pre-trained model, train the task head using a preset training set and test it using a preset validation set to obtain the task head that has been preliminarily trained;

[0035] An image coding large model training module is used to unfreeze the pre-trained model, perform fine-tuning training on the large model using the training set and perform verification using the verification set to obtain the finally trained large model;

[0036] The image analysis module is used to input the pathological slice image to be analyzed into the trained large model, and calculate the corresponding features of the pathological slice image based on the results output by the large model to obtain the analysis results of the pathological slice image.

[0037] A third aspect of the present invention provides an electronic device, including:

[0038] at least one processor; and a memory communicatively coupled to the at least one processor;

[0039] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned pathological section image analysis method based on a large language model.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned pathological section image analysis method based on a large language model.

[0041] The characteristics and beneficial effects of the present invention are:

[0042] The present invention extracts the features of pathological slice images through an image coding macromodel, and performs segmentation, classification or detection on the pathological images, and then trains the processed image features and the analysis report and diagnosis results of the pathological images with a language macromodel. The present invention realizes automatic feature extraction and analysis of pathological slice images by utilizing the powerful feature extraction and representation capabilities of LLMs. In this way, the present invention aims to improve the accuracy and efficiency of pathological slice image analysis and provide doctors with a reliable auxiliary diagnosis tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The figure is an overall flow chart of a pathological slice image analysis method based on a large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention proposes a pathological slice image analysis method and device based on a large language model, which is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0045] The first embodiment of the present invention proposes a pathological slice image analysis method based on a large language model. The overall process is as follows: Figure 1 As shown, including:

[0046] Constructing a large image encoding model for pathological sections, wherein the large model consists of a pre-trained model and an additional task head;

[0047] Freeze the pre-trained model, train the task head using a preset training set and test it using a preset validation set to obtain the task head that has been preliminarily trained;

[0048] Unfreeze the pre-trained model, use the training set to fine-tune the large model and use the validation set to test it, so as to obtain the finally trained large model;

[0049] The pathological slice image to be analyzed is input into the trained large model, and the corresponding features of the pathological slice image are calculated based on the results output by the large model to obtain the analysis results of the pathological slice image.

[0050] In a specific implementation of the present invention, the pathological section image analysis method based on a large language model comprises the following steps:

[0051] 1) Construct pathological slice image training set and validation set; the specific steps are as follows:

[0052] 1-1) Acquire pathological slice images to form a pathological slice image dataset.

[0053] In this embodiment of the present invention, the pathological slice image dataset contains pathological slice images corresponding to multiple pathological types, including but not limited to cancer, inflammation, benign lesions, etc., and the number of pathological slice images of each type should be greater than 100. The size of all pathological slice images in the dataset is consistent, and in a specific embodiment of the present invention, they are all 512×512 pixels to ensure consistency and standardization when inputting the model.

[0054] In a specific embodiment of the invention, a total of 20,000 pathology slide images were collected from multiple pathology laboratories and medical institutions to form a pathology slide image dataset. In order to ensure the diversity and representativeness of the dataset, these images cover different pathological features, patient backgrounds, and sample sources.

[0055] Furthermore, in the process of collecting slice images in this embodiment, special attention is paid to the following points:

[0056] Diversity of pathological types to ensure that the dataset contains a wide range of pathological types, including cancer (such as breast cancer, lung cancer, gastric cancer, etc.), where the segmentation content is mainly the cancerous area and its boundaries; inflammation (such as hepatitis, gastritis, etc.), where the segmentation content is mainly the inflammatory reaction area and its boundaries; benign lesions (such as benign tumors, cysts, etc.), where the segmentation content is mainly the lesion area and its boundaries.

[0057] Diversity of sample sources,This embodiment collects samples from different geographical regions and medical institutions,,covering different patient groups and pathological characteristics to ensure,the broad representativeness of the dataset.

[0058] In addition, this embodiment selects high-quality pathological slice images to ensure the clarity and details of the images so that the model can learn more useful information.

[0059] 1-2) Annotate each pathological slice image in the pathological slice image dataset obtained in step 1-1).

[0060] In this embodiment, the annotation content of each pathological slice image includes the polygonal boundary of the lesion area and the corresponding lesion type label, and these annotation data will be used for subsequent model training and verification.

[0061] In a specific embodiment of the present invention, the labeling process includes: preliminary inspection, pathology experts first perform preliminary inspection on each pathology slice image to determine the existence and approximate location of the lesion area; detailed labeling, using professional labeling tools to perform detailed polygonal labeling on each lesion area and accurately outline the lesion boundary; type labeling, assigning a corresponding lesion type label to each labeled lesion area, such as cancer, inflammation or benign lesions; review, all labeling results are reviewed multiple times to ensure the accuracy and consistency of the labeling and avoid mislabeling or missing labels.

[0062] 1-3) Preprocessing the images of the pathological slice image dataset annotated in step 1-2).

[0063] In an embodiment of the present invention, the preprocessing includes: normalizing the image pixel values ​​to a range of [0, 1] to improve the training effect and convergence speed of the model.

[0064] 1-4) Divide the data set preprocessed in step 1-3) into a training set and a validation set.

[0065] If the present invention must retain the validation set, what is the ratio between the training set and the validation set?

[0066] In a specific implementation of the present invention, 15,000 images are randomly selected from a pathology slice image data set consisting of 20,000 images to form a training set, and the remaining 5,000 images form a validation set. Generally speaking, the ratio of the training set to the validation set can be 3:1.

[0067] Optionally, this embodiment can also perform data enhancement on the pathological slice image data set in the training set, such as random cropping, rotation, flipping, etc., to increase the diversity of the training data and improve the robustness of the model. Through these preprocessing steps, it is ensured that the pathological slice image data has a consistent format and quality when input into the model, thereby improving the training effect and performance of the model.

[0068] 2) Build a large image coding model for pathological sections.

[0069] In this embodiment, the existing pre-trained DinoV2 model is used as the main body of the image coding model, and a MaskFormer task head is added to the output layer of the DinoV2 model, and the image coding model is constructed.

[0070] It should be noted that DinoV2 is an advanced self-supervised visual representation learning model that can extract rich image features by training on large-scale image datasets. DinoV2 mainly uses Vision Transformers (ViT), also known as visual attention methods, as the architectural basis, and can efficiently process and understand the global content of images through a mechanism called "self-attention". In the process of self-supervised learning, DinoV2 uses a variety of data enhancement techniques, such as cropping, color perturbation, and Gaussian blur, to generate multiple views of the image and let the model learn the relationship between these views, so as to obtain a more in-depth and robust image representation.

[0071] In this embodiment, in the image coding model, the input of DinoV2 is a preprocessed 512×512 pixel pathological slice image, and the output is a high-dimensional image feature vector, usually a feature map of 16×16×768. This feature map is generated by DinoV2 through its deep visual feature extraction mechanism and contains rich information of the input image.

[0072] MaskFormer is an advanced image segmentation model specially designed to handle complex image segmentation tasks. In this embodiment, the input of MaskFormer is the high-dimensional image feature vector output by DinoV2, and the output is the segmentation mask of each pixel in the pathological slice image. Specifically, the MaskFormer task head consists of three convolutional layers, each with a convolution kernel size of 3×3 and equipped with 8 self-attention heads. Each self-attention head can capture different feature dimensions of the image and enhance the segmentation ability of the model. The classification probability of each pixel is generated by the last softmax in the MaskFormer task head to ensure that the output segmentation mask has accurate classification results.

[0073] 3) Using the training set obtained in step 1), the image coding model constructed in step 2) is trained; the specific steps are as follows:

[0074] 3-1) Freeze the pre-trained model in the image encoding large model and train the task head.

[0075] In this embodiment, the pre-trained parameters of DinoV2 in the large image encoding model are frozen to ensure that these parameters remain unchanged during the training process at this stage. Among them, the parameters of the pre-trained model are frozen in order to maintain the feature extraction capabilities learned by the pre-trained model on a large-scale data set and avoid the destruction of these features during the initial training process; and only training the parameters of the task head allows the task head to focus on learning the specific features of the pathological slice image. In this way, the model can utilize the rich features learned in the pre-training stage to enhance the segmentation ability of the task head. The advantage of this method is that it can fully utilize the powerful feature extraction capabilities of the pre-trained model, while focusing on the optimization of the task head to improve the segmentation performance of the overall model.

[0076] Then, the MaskFormer task head is trained using the pathological slice image training set obtained in step 1). During the training process, the parameters of the task head are optimized by the back propagation algorithm using the same or similar loss function and optimization algorithm as the pre-trained model. Appropriate training parameters, including learning rate, training cycle, batch size, etc., are set to ensure that the task head converges stably during the training process.

[0077] In a specific embodiment of the present invention, the specific steps of training the MaskFormer task head are as follows:

[0078] First, the parameters of the MaskFormer task head are randomly initialized, and the parameters required for training are set, including the learning rate, training cycle, and batch size. In a specific embodiment of the present invention, the learning rate is set to 0.001, the training cycle is set to 50 epochs, and the batch size is set to 32. The cross entropy loss function and the Adam optimizer are selected, and the parameters of the MaskFormer task head are optimized by the back propagation algorithm. The cross entropy loss function is suitable for classification tasks and can effectively measure the difference between the predicted results and the actual labels, while the Adam optimizer ensures stability and rapid convergence during training through adaptive learning rate adjustment.

[0079] Then, the MaskFormer task head is trained using the pathological slice image training set. During the training process, the parameters of the task head are continuously adjusted so that it can accurately segment the lesion area. Through multiple rounds of iterative training, the task head gradually learns how to identify and segment the lesion area on the high-dimensional feature map, thereby improving the segmentation accuracy.

[0080] Finally, monitor the loss value and validation set performance during the training process in real time to ensure the stability and effectiveness of the training. When the loss value on the validation set no longer decreases significantly, you can consider ending the training. Monitoring the training process can help detect overfitting in a timely manner and optimize model performance by adjusting training parameters. For example, the original initial learning rate is 0.001. If the training process is too slow, it can be adjusted to 0.002. If the training process fails to converge, it can be adjusted to 0.0005.

[0081] Through the above steps, the MaskFormer task head can fully learn the specific features of pathological slice images based on the frozen DinoV2 pre-trained model, laying a solid foundation for the subsequent fine-tuning of the full model. This process not only improves the segmentation ability of the model, but also ensures the accuracy and stability of the model when processing complex pathological slice images.

[0082] When the MaskFormer task head is trained, proceed to step 3-2).

[0083] 3-2) Unfreeze the pre-trained model, and then fine-tune the large image encoding model; specifically, it includes:

[0084] Unfreeze the pre-trained parameters of the DinoV2 model so that the parameters of all layers of the DinoV2 model can be updated in the subsequent training process.

[0085] Set the parameters required for fine-tuning training, including learning rate, training cycle, and batch size. The learning rate in the fine-tuning phase should be set low to avoid large perturbations to the learned features. A commonly used learning rate is 0.0001, the training cycle can be set to 20 epochs, and the batch size can be kept at 32. Select the cross entropy loss function and the Adam optimizer to optimize the parameters of the entire model through the back-propagation algorithm.

[0086] It should be noted that the setting of the learning rate is crucial during the fine-tuning training stage. The learning rate is usually reduced in stages. The initial learning rate is high so that the model parameters can be adjusted quickly. As the training progresses, the learning rate is gradually reduced to refine the adjustment of the large model parameters. This method can ensure that the large image coding model converges stably during the fine-tuning process and avoid training oscillation or overfitting.

[0087] Through the above steps, the pre-trained model and the task head can be jointly optimized in the pathological slice image segmentation task to improve the performance of the overall model. The fine-tuned image encoding large model not only has a powerful feature extraction capability, but also has a high-precision segmentation capability for pathological slice images, thereby meeting the needs of practical applications. The method described in this embodiment can significantly improve the overall performance and adaptability of the large model by making full use of the existing knowledge of the pre-trained model and combining the specific optimization of the task head, providing reliable technical support for actual pathological slice image analysis.

[0088] After fine-tuning is completed, the final trained image encoding model for pathological sections is obtained.

[0089] 3-3) Further, optionally, this embodiment can also further optimize the model by evaluating the trained large image encoding model for pathological sections; the specific steps are as follows:

[0090] 3-3-1) Evaluate model performance;

[0091] After completing the fine-tuning training of the large model, the next step is to conduct a comprehensive evaluation of the large model to ensure that its performance in the pathology image segmentation task meets expectations. The evaluation process includes using a validation set and a test set to measure the segmentation accuracy and robustness of the model. Key evaluation indicators include intersection over union (IoU), pixel accuracy (PA), and Dice coefficient, which can fully reflect the performance of the model in the segmentation task. Specifically, they include:

[0092] First, prepare an independent test set to ensure that the images in the test set have not been used for training and verification of the large image coding model to ensure the objectivity of the evaluation results. Then, input the pathological slice images of the test set into the trained large image coding model to generate the segmentation results of each pathological slice image in the test set. Based on the segmentation results, calculate the key evaluation indicators of the large image coding model on the test set, including intersection over union (IoU), pixel accuracy (PA), and Dice coefficient.

[0093] The intersection over union (IoU) is used to measure the overlap between the predicted segmentation and the actual segmentation, the pixel accuracy (PA) is used to measure the proportion of correctly classified pixels in the predicted results, and the Dice coefficient is used to evaluate the consistency between the predicted segmentation and the actual segmentation. By combining these evaluation indicators, we can fully understand the segmentation performance of the model.

[0094] 3-3-2) Optimize and adjust the model;

[0095] According to the evaluation results of step 3-3-1), the model is further optimized and adjusted to improve its performance in the pathological slice image segmentation task. The optimization process includes adjusting model parameters, improving training strategies, and redesigning data enhancement methods. The goal is to gradually improve the segmentation accuracy and robustness of the model through continuous iteration. Specifically, it includes:

[0096] First, analyze the evaluation results to identify the strengths and weaknesses of the model. For example, if the model performs poorly on certain types of lesions, consider increasing the amount of training data for these types of lesions or performing specialized parameter tuning for these types of lesions.

[0097] Then, adjust the model's hyperparameters, including learning rate, batch size, regularization coefficient, etc., and conduct multiple experiments to find the best parameter combination. You can use methods such as grid search or random search to systematically explore the effects of different parameter combinations. In addition, you can consider improving the training strategy, such as using more advanced optimization algorithms, introducing new loss functions, or increasing the diversity of training data.

[0098] Next, improve the data augmentation method to increase the diversity of training data and the robustness of the model. For example, more types of image transformation techniques can be introduced, such as color jitter, blur, noise, etc., so that the model can still maintain high performance when facing different image changes.

[0099] Finally, further fine-tuning training is performed to gradually improve the segmentation performance of the model by combining new data and optimization strategies. After each training, the performance of the model is re-evaluated and the next round of optimization is performed based on the evaluation results.

[0100] For example, this embodiment optimizes the image encoding large model finally obtained in step 3-2). First, an independent test set of pathological slice data is input into the model for lesion recognition and segmentation, and the results of lesion segmentation are compared with the actual results of actual annotation, and the classification accuracy at the pixel level is calculated. Assuming that the average classification accuracy of different categories finally calculated is 80%, if it is hoped that the model can better complete the recognition task and achieve a higher accuracy, for example, more than 90%, then the model needs to be adjusted. First, find out which type of lesion recognition accuracy of the large model is low, and increase the weight of the corresponding category in the loss function so that the large model can better learn the characteristics of this type of lesion. For example, if the classification rate of cell edema is low, the weight corresponding to the original edema can be doubled in the loss function. In addition, the number of different types of lesions can be counted. If the number of a certain type of lesions is too small, the large model will not be able to learn the characteristics of this type of lesions, resulting in poor classification results. For example, if there are too few samples of cell proliferation lesions, it is necessary to enhance the data of this type of samples, such as distorting, rotating, adding noise and other operations to increase the number of samples of this type, and then train the large model. After the training is completed, retest the recognition accuracy and continue to look for parts that can be optimized.

[0101] In this embodiment, the image coding model after fine-tuning and optimization has better pathological slice feature extraction and coding effects, and can be applied to practical tasks such as pathological image analysis and diagnosis.

[0102] 4) Using the image coding large model obtained in step 3) to process the pathological slice image to be analyzed, and calculating the corresponding features of the pathological slice image based on the results output by the large model; the specific steps are as follows:

[0103] 4-1) Obtain a pathological slice image to be analyzed.

[0104] In this embodiment, the image is included in the training set or the validation set, the image size is the same as the training set image, which is 512×512 pixels, and the lesion category included in the image should be included in the lesion category corresponding to the training set image.

[0105] 4-2) The pathological slice image obtained in step 4-1) is input into the image coding large model trained in step 3), and the large model outputs a lesion distribution map corresponding to the input image.

[0106] 4-3) Performing image analysis on the lesion distribution map obtained in step 4-2) to obtain features corresponding to the pathological slice image to be analyzed.

[0107] In this embodiment, each cell or lesion area in the distribution map is marked, and its area and perimeter are calculated. Through these parameters, the diameter of the cell or the size of the lesion area can be estimated. At the same time, for the marked cells, the number of marked cells is counted and divided by the area of ​​the corresponding area to obtain the cell density; for the marked lesion area, its proportion of the entire lesion distribution map or its size relative to a reference standard is calculated to quantify the size of the lesion.

[0108] The connected component labeling (CCL) algorithm is used to label each cell or lesion area in the distribution map. i , i = 1, 2, ..., N, where N is the total number of labeled cells or the total number of lesion areas, and its area S is calculated. i and perimeter C i The area can be obtained by counting the number of pixels within the region, and the perimeter can be obtained by counting the number of pixels at the edge of the region. The diameter D of the cell or tissue region i The area S of the region i It is estimated that, assuming it is a circle, The size of the lesion region can be quantified by calculating the ratio of its area to the entire image area. Let the area of ​​the lesion region be S lesion , the entire image area is S total , then the lesion area size ratio is The cell density μ can be calculated by counting the number of labeled cells N and dividing it by the area S of the corresponding region. region To get, that is

[0109] Through morphological operations such as opening, closing, dilation and erosion, noise and small holes in the distribution map are removed, and tightly connected cells are separated. Then, the area or equivalent diameter of the region is used to represent the size of the cell, the shape factors such as circularity and rectangularity are used to describe the shape of the cell, and the gray-level co-occurrence matrix is ​​used to extract texture features such as contrast, energy, entropy, etc. Finally, the extracted feature vectors are normalized and the most discriminative features are selected to provide feature representations that are of great reference value for pathologists' diagnostic decisions.

[0110] Correlation analysis is used to evaluate the relationship between features. In order to evaluate the relationship between features and avoid multicollinearity problems, this embodiment uses correlation analysis to identify highly correlated features. Correlation analysis can use the Pearson correlation coefficient r or the Spearman rank correlation coefficient ρ to measure the linear relationship between features. The Pearson correlation coefficient is defined as:

[0111]

[0112] Among them, X i and Y i are the observed values ​​of the two features, μx and μy are the average values ​​of the two features. The Spearman rank correlation coefficient is defined as:

[0113]

[0114] where d i are features X and Y i The rank difference is n, where n is the number of observations

[0115] By calculating these correlation coefficients, this embodiment can identify highly correlated feature pairs. For highly correlated features, one of the features can be removed or combined into a new feature to reduce the impact of multicollinearity on model performance. In this way, this embodiment can extract the most valuable and discriminative information from a large number of features, providing strong support for pathologists' diagnosis.

[0116] Further, optionally, the method described in this embodiment also includes:

[0117] 5) Train large language models.

[0118] In this embodiment, a large language model such as BERT and GPT is used to encode the doctor's diagnosis report; the features of the pathological slice image obtained in step 4) are combined with the text encoding to form a multimodal input. A multimodal Transformer model, such as CLIP, is used for feature fusion. Then, the fused features and the doctor's diagnosis report are used as training data to train the large language model. The specific steps are as follows:

[0119] 5-1) Prepare training data for the large language model.

[0120] The doctor's diagnosis report and the features extracted from the pathological slice image based on the output of the large image coding model are integrated. For text data, preprocessing operations such as word segmentation and stop word removal are performed, and the processed text is converted into word vectors, such as using Word2Vec or GloVe, so that the model can learn the semantics and contextual relationships of words. For image features, normalization or standardization is performed to ensure the consistency and comparability of the data. Then, a multimodal input is obtained by splicing the text vector and the image feature. The multimodal input and the corresponding diagnosis report constitute a training sample of a large language model.

[0121] In this embodiment, in order to ensure that the concatenated vector dimension matches the input requirements of the large language model, it may be necessary to adjust the dimensions of the text data embedding vector and the image feature vector. If the image feature vector is longer than the text data embedding vector, you can consider adding zero padding or using pooling to reduce the dimension of the image feature. The concatenated input vector will contain information about both text and image, which can better represent the diversity of the data.

[0122] Through the above preprocessing steps, this embodiment can effectively integrate the features of the doctor's diagnosis report and the pathological slice image, providing high-quality training data for the subsequent large language model training. In this embodiment, the number of samples of the integrated training data should be as large as possible. The more samples, the richer the doctor's diagnosis experience and logic that the model can learn.

[0123] 5-2) Select a suitable large language model as the base model, such as BERT, GPT, etc.

[0124] The embodiment of the present invention uses the GPT large model as the basic model. The GPT model adopts the Transformer encoder-decoder architecture and is composed of multiple layers of Transformer modules. The Transformer module includes a self-attention layer and a feed-forward neural network layer.

[0125] 5-3) Use the training data obtained in step 5-1) to train the large language model. During the training process, the parameters of the model are adjusted by optimizing the loss function. Common loss functions include cross entropy loss, mean square error, etc. The training process can use batch gradient descent, Adam and other optimization algorithms to optimize the model parameters.

[0126] 5-4) Further, during the training process, the validation set of the large language model can be used to evaluate the performance of the large language model, and the model can be tuned according to the evaluation results. Tuning can include adjusting the learning rate, changing the training strategy, etc. to improve the accuracy and generalization ability of the model. After the training is completed, the model is evaluated using the test set of the large language model. Evaluation indicators can include accuracy, recall rate, F1 value, etc. to comprehensively evaluate the performance of the model. In addition, visualization tools can be used to show the comparison between the predicted results and the actual results of the model, so as to further analyze and improve the model.

[0127] For example, the samples obtained in step 5-1) can be shuffled and randomly divided into a training set and a validation set at a ratio of 8:1, with no duplication between the two sets. Then, after each round of training, the data of the validation set is input into the model for performance evaluation, and the accuracy and recall of the model in the validation set are calculated to analyze the model training situation. If the results of adding multiple rounds of validation sets do not improve significantly, then it can be considered that the large language model has basically converged and the training can be stopped in advance. If the accuracy of the large language model in the validation set is always low and improves slowly, then you can consider increasing the learning rate, such as increasing it to twice the previous rate. In addition, the accuracy obtained by evaluating the validation set after each round of training can be plotted to analyze the situation of the model during the training process.

[0128] Ultimately, the trained large language model can learn the association between pathological slice features and diagnostic reports, and can be applied to practical tasks such as pathological image analysis and diagnosis.

[0129] 6) Generate a diagnostic report.

[0130] In this embodiment, the features of the pathological slice image to be analyzed obtained in step 4) are combined with the text encoding to form a multimodal input. Then, the multimodal input is sent to the trained large language model to generate a final diagnosis report.

[0131] To implement the above embodiment, a second aspect of the present invention provides a pathological slice image analysis device based on a large language model, comprising:

[0132] An image coding large model construction module is used to construct an image coding large model for pathological sections, wherein the large model is composed of a pre-trained model and an additional task head;

[0133] A task head training module, used to freeze the pre-trained model, train the task head using a preset training set and test it using a preset validation set to obtain the task head that has been preliminarily trained;

[0134] An image coding large model training module is used to unfreeze the pre-trained model, perform fine-tuning training on the large model using the training set and perform verification using the verification set to obtain the finally trained large model;

[0135] The image analysis module is used to input the pathological slice image to be analyzed into the trained large model, and calculate the corresponding features of the pathological slice image based on the results output by the large model to obtain the analysis results of the pathological slice image.

[0136] It should be noted that the above explanation of the embodiment of a pathological slice image analysis method based on a large language model is also applicable to a pathological slice image analysis device based on a large language model in this embodiment, which will not be repeated here. According to a pathological slice image analysis device based on a large language model proposed in an embodiment of the present invention, by constructing a large model of image encoding for pathological slices, the large model consists of a pre-trained model and an additional task head; freezing the pre-trained model, training the task head with a preset training set and testing it with a preset validation set to obtain the task head that has been preliminarily trained; unfreezing the pre-trained model, fine-tuning the large model with the training set and testing it with the validation set to obtain the large model that has been finally trained; inputting the pathological slice image to be analyzed into the trained large model, and calculating the corresponding features of the pathological slice image based on the results output by the large model to obtain the analysis results of the pathological slice image. In this way, the features in the pathological slice image can be automatically extracted, and the pathological slice image can be further analyzed to automatically generate a diagnosis report to improve the accuracy and efficiency of the diagnosis of the pathological slice image.

[0137] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0138] at least one processor; and a memory communicatively coupled to the at least one processor;

[0139] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned pathological section image analysis method based on a large language model.

[0140] To implement the above-mentioned embodiment, the fourth aspect of the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned pathological section image analysis method based on a large language model.

[0141] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0142] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes a pathological slice image analysis method based on a large language model in the above embodiment.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0148] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A pathological slice image analysis method based on a large language model, characterized in that: include: Constructing a large image encoding model for pathological sections, wherein the large model consists of a pre-trained model and an additional task head; Freeze the pre-trained model, train the task head using a preset training set and test it using a preset validation set to obtain the task head that has been preliminarily trained; Unfreeze the pre-trained model, use the training set to fine-tune the large model and use the validation set to test it, so as to obtain the finally trained large model; The pathological slice image to be analyzed is input into the trained large model, and the corresponding features of the pathological slice image are calculated based on the results output by the large model to obtain the analysis results of the pathological slice image.

2. The method according to claim 1, characterized in that: The image coding model adopts the pre-trained DinoV2 model, and adds a MaskFormer task head to the output layer of the DinoV2 model.

3. The method according to claim 1, characterized in that Also includes: 1) Acquire pathological slice images to form a pathological slice image dataset; The pathological slice image dataset includes pathological slice images corresponding to a plurality of pathological types, wherein each pathological slice image has a consistent size; 2) annotating each pathological slice image of the pathological slice image data set of step 1); wherein the annotation content of each pathological slice image includes a polygonal boundary of the lesion area and a corresponding lesion type label; 3) Preprocessing the images of the pathological slice image dataset annotated in step 2) to normalize the pixel value of each image to the range of [0, 1]; 4) The pathological section data set preprocessed in step 3) is randomly divided into a training set and a validation set according to a preset ratio.

4. The method according to claim 1, characterized in that Also includes: The performance of the trained image coding large model is evaluated using a preset test set, and the image coding large model is adjusted according to the evaluation result to obtain the optimized image coding large model; wherein the pathological slice images in the test set are different from the pathological slice images in the training set and the validation set.

5. The method according to claim 1, characterized in that Also includes: Inputting the pathological slice image to be analyzed into the trained large model, and the large model outputs a lesion distribution map corresponding to the input image; Processing the lesion distribution map to obtain the characteristics of the pathological slice image specifically includes: Marking each cell or lesion area in the distribution map, and calculating its area and perimeter, thereby estimating the diameter of the cell or the size of the lesion area; For the labeled cells, the number of labeled cells is counted and divided by the area of ​​the corresponding region to obtain the cell density; for the labeled lesion area, the proportion of the area in the distribution map or the size relative to the set reference standard is calculated to quantify the size of the lesion.

6. The method according to claim 5, characterized in that Also includes: 1) generating features of a plurality of pathological slice images based on the output result of the image coding large model and obtaining a diagnosis report corresponding to each pathological slice image; wherein each diagnosis report is preprocessed to obtain a corresponding text vector; 2) The text vector and the features of the corresponding pathological slice image are concatenated to obtain a multimodal input, which together with the corresponding diagnosis report constitutes a training sample for a large language model; 3) Select a large language model; 4) Using the training samples obtained in step 2), the large language model obtained in step 3) is trained to obtain a trained large language model.

7. The method according to claim 6, characterized in that Also includes: Combining the features of the pathological slice image to be analyzed generated based on the output result of the image coding large model with the text coding to form a multimodal input; This multimodal input is then fed into a trained large language model to generate a final diagnosis report.

8. A pathological slice image analysis device based on a large language model, characterized in that: include: An image coding large model construction module is used to construct an image coding large model for pathological sections, wherein the large model is composed of a pre-trained model and an additional task head; A task head training module, used to freeze the pre-trained model, train the task head using a preset training set and test it using a preset validation set to obtain the task head that has been preliminarily trained; An image coding large model training module is used to unfreeze the pre-trained model, perform fine-tuning training on the large model using the training set and perform verification using the verification set to obtain the finally trained large model; The image analysis module is used to input the pathological slice image to be analyzed into the trained large model, and calculate the corresponding features of the pathological slice image based on the results output by the large model to obtain the analysis results of the pathological slice image.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

Images