Grape tire focus segmentation method and device based on image large model

Abstract

The invention discloses a grape tire lesion segmentation method and device based on a large image model. The method comprises the following steps: A, constructing a lesion segmentation neural network model HmNet for segmenting hyperplasia and edema lesions; b, obtaining a grape tire slice scanning graph; c, obtaining a plurality of grape embryo slice scanning graph blocks; d, obtaining a lesion segmentation image corresponding to each block through the lesion segmentation neural network model HmNet, and obtaining a preliminary complete slice lesion segmentation image after fusion; and E, performing image post-processing on the preliminary complete section focus segmentation image, and finally obtaining a post-processed section focus segmentation image reflecting the actual focus form. According to the method, the hyperplasia and edema lesions in the pathological section can be accurately identified and segmented, so that clinical doctors can be assisted in more efficient case screening.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and particularly to a hydatidiform mole lesion segmentation method and device based on an image large model. Background Art

[0002] Hydatidiform mole (HM) refers to a vesicular mole in the shape of a grape cluster formed by the placenta after pregnancy. Most hydatidiform mole babies die or form teratomas, and very few full-term infants are born. Generally, 10% to 20% of hydatidiform moles will develop into malignant hydatidiform moles and choriocarcinoma, and such cancers will metastasize through blood clots. If the treatment is not timely, it will pose a life threat to the patient. Therefore, the early pathological diagnosis of hydatidiform mole is of great significance to each pregnant woman suffering from the disease.

[0003] In the prior art, there are mainly two ways to detect and screen hydatidiform moles. The first is to observe the slices manually through a microscope, and the second is to detect the genes related to hydatidiform moles.

[0004] In the first method, generally, a pathologist uses a microscope with magnifications of 5*10 times and 10*10 times to observe multiple slices of a patient, and then makes a comprehensive diagnosis based on experience and the morphological characteristics of the tissue cells in the slices. The diagnosis of hydatidiform mole is mainly made by observing the villus characteristics in the slices, and the main pathological characteristics of the slices are villous trophoblast hyperplasia and interstitial edema inside the villi.

[0005] Pathologists in gynecology hospitals need to spend a large amount of time every day diagnosing diseases such as hydatidiform mole with a relatively low risk coefficient compared to tumors. Most of these patients are not ill, but this will take up a large amount of the working time of pathologists.

[0006] However, at present, the number of pathologists in China is about 15,000, and there is a large talent gap, and the detection efficiency is low. In addition, since the clinical diagnosis of hydatidiform mole mainly relies on pathologists to manually screen slices, it is difficult to guarantee the accuracy. Especially for hydatidiform moles before 12 weeks, since the hydatidiform mole has not reached maturity, the lesions are not fully developed, and the tissue morphology is similar to that of normal hydatidiform mole slices, making it difficult to distinguish, resulting in a very low clinical diagnosis accuracy of less than 50%.

[0007] In the second method, the invention patent with the application number 201310027715.1 and the title of "Gene Chip, Detection Reagent and Kit for Detecting NLRP7 Gene" discloses that detecting the NLRP7 gene SNP related to hydatidiform mole is of great significance for the clinical diagnosis of hydatidiform mole, early screening of high-risk populations and early preventive intervention, and can be widely used for the efficient clinical screening of high-risk hydatidiform mole populations. This invention patent constructs a gene chip detection system for screening high-risk populations with NLRP7 gene polymorphisms related to hydatidiform mole. The gene chip includes a solid-phase carrier and oligonucleotide probes synthesized on this carrier. The detection reagent includes the gene chip and 18 pairs of PCR primers for amplifying each SNP in the sample. The kit includes the detection reagent, a negative control sample and a positive control sample. This invention patent can quickly and accurately detect each related SNP locus of the NLRP7 gene in clinical samples, which is of great significance for the clinical diagnosis of hydatidiform mole, early screening of high-risk populations and early preventive intervention.

[0008] Although it is necessary to screen for hydatidiform mole by detecting genes, screening for hydatidiform mole by detecting the NLRP7 gene has two problems. One is that it increases the detection steps of the kit, and the entire detection cycle will be relatively long. The other is that it involves the production of chips, reagents and kits, resulting in a substantial increase in screening costs. Its application scope in the clinical diagnosis of hydatidiform mole is very limited and it is not easy to promote and apply.

[0009] Based on the current situation of the small number of pathologists, the low efficiency and low accuracy of manual screening of slices by physicians, as well as the high cost and long cycle of gene detection screening, it is necessary to develop a complete set of methods and devices for automatically obtaining images from microscopes to generating distribution maps of pathological features such as edema and hyperplasia, so as to assist clinicians in screening cases more efficiently.

[0010] Clinical pathologists mainly comprehensively judge whether it is a hydatidiform mole disease through pathological features such as villous stromal edema and diffuse hyperplasia of trophoblasts at the edge of the villi, as well as information such as the length of menopause and pregnancy history of the patient. Among them, the pathological feature of villous stromal edema is a relatively key diagnostic basis. Summary of the Invention

[0011] The purpose of the present invention is to provide a method and device for segmenting hydatidiform mole lesions based on an image large model, which can accurately identify and segment hyperplasia and edema lesions in pathological sections, thereby assisting clinicians in screening cases more efficiently.

[0012] The present invention adopts the following technical solutions:

[0013] A method for segmenting hydatidiform mole lesions based on an image large model includes the following steps:

[0014] A: Build a lesion segmentation neural network model based on an image encoding large model; after connecting a segmentation head MaskFormer for distinguishing background, hyperplasia, and edema lesions in a pre-trained pathological image encoding large model, a lesion segmentation neural network model HmNet for segmenting hyperplasia and edema lesions is finally formed after training;

[0015] B: Perform he staining on hydatidiform mole sections and obtain scanned images of hydatidiform mole sections;

[0016] C: Cut the scanned images of hydatidiform mole sections to obtain several cut pieces of scanned images of hydatidiform mole sections;

[0017] D: Input several cut pieces of scanned images of hydatidiform mole sections into the lesion segmentation neural network model HmNet to obtain a lesion segmentation map corresponding to each cut piece, and then fuse the cut piece lesion segmentation maps corresponding to all cut pieces of scanned images of hydatidiform mole sections to obtain a preliminary complete sliced lesion segmentation map;

[0018] E: Perform image post-processing on the preliminary complete sliced lesion segmentation map to finally obtain a post-processed sliced lesion segmentation map reflecting the actual lesion morphology.

[0019] The pre-trained pathological image encoding large model is based on the DinoV2 image encoding large model, and the DinoV2 image encoding large model is deeply trained through a segmentation task using a preset pathological slice dataset to obtain the pre-trained pathological image encoding large model.

[0020] The pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the segmentation head MaskFormer; after the segmentation head MaskFormer processes the feature map, it outputs three probability maps for background, hyperplasia, and edema lesions.

[0021] When the output background probability is less than or equal to the environmental probability threshold θ, this area is considered a lesion area; when the output hyperplasia lesion probability is greater than or equal to the edema lesion probability, this area is considered a hyperplasia lesion; when the hyperplasia lesion probability is less than the edema lesion probability, this area is considered an edema lesion.

[0022] In the probability maps output by the segmentation head MaskFormer, the sum of the probabilities of the pixel points for background, hyperplasia lesions, and edema lesions is 1.

[0023] In step E, first remove the noise in the sliced lesion segmentation map, then optimize the boundary of the lesion segmentation area, then remove the too small areas in the sliced lesion segmentation map with an area less than the set threshold caused by errors in the segmentation process, and finally connect the adjacent segmentation areas to form a continuous lesion area; finally, obtain an optimized sliced lesion segmentation map reflecting the actual lesion morphology.

[0024] The training method of the lesion segmentation neural network model HmNet is as follows:

[0025] In the first step, the DinoV2 image encoding large model is deeply trained through a segmentation task using a preset pathological slice dataset to obtain a trained pathological image encoding large model;

[0026] In the second step, the pictures in the labeled dataset are cut into blocks according to the set size to construct the training dataset required for training the lesion segmentation neural network model HmNet;

[0027] In the third step, for the lesion segmentation neural network model composed of the pathological image encoding large model and the segmentation head MaskFormer, the parameters of the pre-trained pathological image encoding large model are frozen to keep them unchanged in the initial training, and only the segmentation head MaskFormer is trained through the training dataset to optimize and update the parameters of the segmentation head MaskFormer;

[0028] In the fourth step, after the segmentation head MaskFormer has been preliminarily trained and verified to be effective, all layers of the pre-trained DinoV2 image encoding large model are unfrozen, so that the parameters of the entire network can be updated together, thereby integrating the learning results of the pre-trained layer and the newly trained segmentation head MaskFormer, and optimizing the parameters of the entire lesion segmentation neural network model HmNet through global fine-tuning;

[0029] In the fifth step, the performance of the lesion segmentation neural network model HmNet is evaluated, and based on the evaluation results, the lesion segmentation neural network model HmNet is further optimized and adjusted.

[0030] The segmentation head MaskFormer contains several layers of Transformer encoders and decoders for further processing the feature maps output by the pathological image encoding large model; multiple self-attention heads in each encoder of the segmentation head MaskFormer are used to capture different feature dimensions of the image.

[0031] A hydatidiform mole lesion segmentation device based on the above-mentioned lesion segmentation method includes:

[0032] A slice image extraction module for magnifying the microscopic structure of a hydatidiform mole slice and obtaining a scanned image of the hydatidiform mole slice containing the microscopic structure of the hydatidiform mole slice;

[0033] A lesion segmentation map generation module for cutting the scanned image of the hydatidiform mole slice according to the set requirements, inputting the obtained several scanned image blocks into the lesion segmentation neural network model HmNet to obtain corresponding lesion segmentation maps, and then fusing all the lesion segmentation maps to obtain a preliminary complete slice lesion segmentation map;

[0034] The sliced lesion segmentation map post - processing module is used to perform image post - processing on the initially complete sliced lesion segmentation map, and finally obtain the post - processed sliced lesion segmentation map that reflects the actual lesion morphology.

[0035] The lesion segmentation neural network model HmNet includes a pre - trained large - scale pathological image encoding model and a segmentation head MaskFormer; the large - scale pathological image encoding model is used to extract features from the input pathological slice image and output the feature map to the segmentation head MaskFormer; the segmentation head MaskFormer is used to perform segmentation prediction on the pathological slice image and output three probability maps of background, hyperplasia, and edema lesions.

[0036] The present invention combines large - model technology and hydatidiform mole slice pathological recognition technology. After cutting the scanned hydatidiform mole slice into pieces and sending them into the lesion segmentation neural network model HmNet, integrating and post - processing the cut scanned maps, the sliced lesion segmentation map of the scanned hydatidiform mole slice can be obtained. The sliced lesion segmentation map can be directly output to clinicians to provide help. Clinicians can intuitively obtain the distribution of hyperplasia and edema according to the sliced lesion segmentation map, reducing the excessive dependence on the subjective analysis of pathologists during the detection process.

[0037] In addition, the large - scale image encoding model adopted in the present invention has been deeply trained on the pathological slice data set, so it has a powerful pathological slice feature extraction ability. At the same time, the segmentation head MaskFormer has been deeply trained, and the network structure is optimized through a specific training method, thereby improving the accuracy of the sliced lesion segmentation map. Through this comprehensive training, the model can more accurately identify and segment hyperplasia and edema lesions in pathological slices, improving the overall segmentation effect. Brief Description of the Drawings

[0038] Figure 1 It is a schematic flow chart of the hydatidiform mole lesion segmentation method based on the large - scale image model in the present invention;

[0039] Figure 2 It is the scanned hydatidiform mole slice in the present invention;

[0040] Figure 3 It is a schematic diagram of a hyperplasia annotation example of the scanned hydatidiform mole slice in the present invention;

[0041] Figure 4 It is a schematic diagram of an edema annotation example of the scanned hydatidiform mole slice in the present invention;

[0042] Figure 5 It is a flow chart for obtaining the annotation training data set of the scanned hydatidiform mole slice in the present invention. Detailed Embodiments

[0043] The present invention will be described in detail below in conjunction with the accompanying drawings and embodiments:

[0044] As Figures 1 to 5 shown, the hydatidiform mole lesion segmentation method based on the large image model of the present invention includes the following steps:

[0045] A: Build a lesion segmentation neural network model based on the large image encoding model; by connecting a segmentation head MaskFormer for distinguishing the background, hyperplasia, and hydropic lesions in the pathological section after the pre-trained pathological image encoding large model, and finally forming a lesion segmentation neural network model HmNet for segmenting hyperplasia and hydropic lesions after training;

[0046] In the present invention, the pre-trained pathological image encoding large model is based on the DinoV2 image encoding large model, and the DinoV2 image encoding large model is deeply trained through a segmentation task using a preset pathological section dataset to obtain the pre-trained pathological image encoding large model;

[0047] In this embodiment, a segmentation head MaskFormer is connected after the pre-trained pathological image encoding large model to perform segmentation prediction on the pathological section image to distinguish the background, hyperplasia, and hydropic lesions in the pathological section; since the segmentation task is to distinguish the background, hyperplasia, and hydropic lesions, the output category of the segmentation head MaskFormer is set to 3.

[0048] In this embodiment, the pathological section dataset includes various pathological types, including cancer, inflammation, and benign lesions; among them, cancer includes breast cancer, lung cancer, gastric cancer, etc., and the segmentation content is mainly the cancerous area and its boundary; inflammation includes hepatitis, gastritis, etc., and the segmentation content is mainly the inflammatory reaction area and its boundary; benign lesions include benign tumors and cysts, etc., and the segmentation content is mainly the lesion area and its boundary.

[0049] In this embodiment, the size of the pathological section image input into the pathological image encoding large model is 512×512 pixels; the pathological image encoding large model extracts features from the input pathological section image and outputs a feature map with a dimension of 16×16×768, and then outputs the feature map to the segmentation head MaskFormer;

[0050] In the present invention, the segmentation head MaskFormer includes several layers of Transformer encoders and decoders for further processing the feature map output by the pathological image encoding large model. The number of Transformer encoder layers is 6 layers, each layer contains 8 self-attention heads, and the feature embedding dimension is 768 dimensions; each self-attention head can capture different feature dimensions of the image and enhance the segmentation ability of the model.

[0051] After the segmentation head MaskFormer processes the feature map, it outputs probability maps for three categories: background, hyperplasia, and edema lesions, which are also the lesion segmentation maps. The dimension of the probability map is 512×512×3, and each pixel corresponds to three probability values, namely the probabilities of belonging to the background, hyperplasia, and edema lesions.

[0052] In the present invention, in the probability map output by the segmentation head MaskFormer, the probability that a pixel is the background is denoted as P background , the probability that a pixel is a hyperplastic lesion is denoted as P hyperplasia , and the probability that a pixel is an edema lesion is denoted as P hydropic . These three probabilities satisfy P background +P hyperplasia +P hydropic =1.

[0053] To determine the final classification, an environmental probability threshold θ is also defined in the present invention. When the output background probability P background is less than or equal to this threshold, then this region is considered a lesion region; when the probability of hyperplastic lesions is greater than or equal to the probability of edema lesions, then this region is considered a hyperplastic lesion; when the probability of hyperplastic lesions is less than the probability of edema lesions, then this region is considered an edema lesion, that is:

[0054]

[0055] B: Perform he staining on the hydatidiform mole section and obtain a scanned image of the hydatidiform mole section;

[0056] In step B, first perform he staining on the hydatidiform mole section, and then place the he-stained hydatidiform mole section on the digital microscope stage. A digital microscope is selected for the microscope; then use the autofocus module to perform automatic focusing of the digital microscope so that a clear hydatidiform mole section can be seen in the field of view of the microscope; finally, use the section scanning module to obtain a scanned image of the hydatidiform mole section under the microscope, as Figure 2 shown.

[0057] C: Cut the scanned image of the hydatidiform mole section to obtain several cut pieces of the scanned image of the hydatidiform mole section;

[0058] Since the resolution of the scanned image of the hydatidiform mole section obtained in step B is relatively high, the pixel size is usually 65536×65536; while the input image pixel size requirement of the lesion segmentation neural network model HmNet is 512×512, so the section needs to be cut; therefore, in this step, the high-resolution scanned image of the hydatidiform mole section is cut to obtain multiple scanned image cut pieces with a pixel size of 512×512.

[0059] In this embodiment, for the scanned image of a hydatidiform mole section with a pixel size of 65536×65536, a total of 128×128 scanned image blocks with a size of 512×512 pixels are finally segmented;

[0060] D: Input several scanned image blocks of hydatidiform mole sections in step C into the lesion segmentation neural network model HmNet to obtain the lesion segmentation map corresponding to each block. Then, fuse the block lesion segmentation maps corresponding to all scanned image blocks of hydatidiform mole sections to obtain a preliminary complete sliced lesion segmentation map;

[0061] E: Perform image post - processing on the preliminary complete sliced lesion segmentation map to finally obtain a post - processed sliced lesion segmentation map that reflects the actual lesion morphology;

[0062] In the present invention, an adaptive morphological operation or filtering technique can be first used to remove the noise in the segmentation map to ensure the clarity and accuracy of the segmentation result; then, through multi - level edge detection and smoothing algorithms, the boundary of the lesion segmentation area is optimized to make the segmentation result more accurate and coherent; on this basis, small areas with an area less than a set threshold caused by errors in the segmentation process are removed to reduce meaningless noise interference; finally, an adjacent segmentation area connection algorithm is used to effectively merge adjacent lesion areas to ensure the integrity and consistency of the hyperplasia and edema lesion areas. Through the above steps, a post - processed sliced lesion segmentation map is finally obtained, which can more truly reflect the actual morphology of the lesion, significantly improving the accuracy, reliability and clinical application value of the segmentation result.

[0063] In this embodiment, an adaptive morphological operation or filtering technique is used to remove the noise in the segmentation map to ensure the clarity and accuracy of the segmentation result. Morphological operations include erosion, dilation, opening operation, and closing operation, etc., to remove isolated small noise points and fill small holes. Filtering techniques include Gaussian filtering and median filtering, which can effectively smooth the noise in the image while retaining the main structural features. For the characteristics of different regions, the filtering parameters can be dynamically adjusted through an adaptive filtering method to further improve the noise removal effect.

[0064] Through a multi - level edge detection method, combining Canny edge detection and Sobel operator, the boundary positions are detected in the image to avoid inaccurate segmentation caused by noise or weak edges. Then, bilateral filtering or other edge - preserving smoothing algorithms are applied to optimize the boundary to keep the clear boundary of the lesion area while smoothing the image, ensuring that the segmentation result is more accurate and coherent.

[0065] For the too small regions generated during the segmentation process with an area smaller than the set threshold (e.g., 10 pixel points), by calculating the area of each segmented region and performing screening, the noise regions without clinical significance are removed. On this basis, for some important small regions, they can be supplemented through region repair algorithms (such as neighborhood-based interpolation repair) to avoid incorrect deletion or region missing, thereby further improving the reliability of the segmentation result.

[0066] For adjacent segmented regions, the connected component labeling algorithm is used to connect the pixel points belonging to the same lesion category to form a complete region. To prevent incorrect merging, combined with the graph cut algorithm or graph segmentation method, intelligent judgment is made according to the texture similarity and edge connectivity between regions to decide whether to merge adjacent regions. This process ensures the integrity of the hyperplasia and edema lesion regions and can accurately reflect the true morphology of the lesions.

[0067] Through the above optimization processing steps, not only the accuracy and reliability of the segmentation result are effectively improved, but also the clinical usability of the segmentation result is ensured, which can provide a clearer and more accurate diagnostic basis for doctors.

[0068] In the present invention, the lesion segmentation neural network model HmNet needs to be trained on the hydatidiform mole slice dataset. This embodiment also provides a training method for the lesion segmentation neural network model HmNet.

[0069] For network training, training pictures for inputting into the network and training are required.

[0070] This embodiment combines the morphological characteristics of hydatidiform mole required for the actual diagnosis of hydatidiform mole. Through the annotation and annotation review of hydatidiform mole slices, the annotation results of 1000 typical hydatidiform mole slice scanning images are obtained. Each slice needs to be detailedly annotated with the hyperplasia and edema lesions therein. The specific annotation examples can be seen in Figure 3 and Figure 4 .

[0071] Currently, the bottleneck of medical imaging lies in the extremely lack of high-quality labeled datasets, and there is no good labeled dataset for hydatidiform mole disease worldwide. The recognition rate of deep networks is based on the premise of a good dataset. Therefore, it is necessary to first perform standardized, reasonable, and strict annotation to obtain a good dataset. In the present invention, as Figure 5 shown, the training dataset is obtained through the following method:

[0072] Step 101: Develop a reliable annotation scheme

[0073] After several proposed annotation schemes are reviewed and confirmed by multiple professional clinicians, finally an annotation scheme is determined, that is, hyperplasia and edema annotation. The specific annotation examples can be seen in Figure 3 andFigure 4 For the hyperplasia and edema annotation, the trophoblast regions with diffuse hyperplasia and edema of the chorionic villi edge elements are circled and indicated.

[0074] Step 102: Training of annotators

[0075] Select multiple annotators with relevant medical knowledge for training.

[0076] Step 103: Preliminary annotation by annotators

[0077] Each annotator is responsible for approximately 80 scanned slices according to the annotation regulations, and annotates the hyperplasia and edema annotations for network training.

[0078] Step 104: Review by pathologists

[0079] The preliminary annotations obtained in Step 103 are strictly reviewed by clinicians with long-term clinical experience and fed back to the annotators.

[0080] Step 105: Detailed annotation by annotators

[0081] Analyze and modify the annotation review results, unify the annotation standards of the annotators, and obtain the final annotation dataset.

[0082] Step 106: Network automatic annotation

[0083] Based on the annotation dataset obtained in Step 105, train the lesion segmentation neural network model HmNet; and use the trained lesion segmentation neural network model HmNet to perform image semantic segmentation on the subsequent newly added slice scan images to obtain a new annotation dataset, that is, the network automatically annotates the new slice scan images, so as to expand the dataset and further train a network with better robustness.

[0084] In the present invention, the training method of the lesion segmentation neural network model HmNet is as follows:

[0085] First step, use a preset pathological slice dataset to deeply train the DinoV2 image encoding large model through a segmentation task to obtain a trained pathological image encoding large model;

[0086] Second step, cut the pictures in the annotation dataset into blocks according to the size of 512×512 to construct the training dataset required for training the lesion segmentation neural network model HmNet;

[0087] In the third step, for the lesion segmentation neural network model composed of the pathological image encoding large model and the segmentation head MaskFormer, freeze the parameters of the pre-trained pathological image encoding large model to keep them unchanged during the initial training. Only train the segmentation head MaskFormer using the training dataset to optimize and update the parameters of the segmentation head MaskFormer.

[0088] The above steps can avoid the damage to the feature extraction ability of the pathological image encoding large model during the initial training, and at the same time can make full use of the excellent performance of the pre-trained pathological image encoding large model in pathological slice feature extraction, so that the segmentation head can effectively learn the segmentation task.

[0089] When training the segmentation head MaskFormer, select the same or similar loss function and optimization algorithm as the pathological image encoding large model, and optimize the parameters of the task head through the backpropagation algorithm, so that the segmentation head MaskFormer can adapt to the data distribution of hydatidiform mole pathological slice images. By setting training parameters such as the learning rate, training cycle, and batch size, ensure that the segmentation head MaskFormer converges stably during the training process. Monitor the training loss and validation set performance of the model in real time to ensure the stability and effectiveness of the training process.

[0090] In the fourth step, after the segmentation head MaskFormer has been preliminarily trained and verified to be effective, unfreeze all layers of the pre-trained DinoV2 image encoding large model, so that the parameters of the entire network can be updated together, thereby integrating the learning results of the pre-trained layer and the newly trained segmentation head MaskFormer, and optimizing the parameters of the entire lesion segmentation neural network model HmNet through global fine-tuning.

[0091] Through the joint training of the lesion segmentation neural network model HmNet, the feature extraction and segmentation capabilities of the model can be further optimized, and the accuracy and reliability of the segmentation results can be improved.

[0092] The initial learning rate of the lesion segmentation neural network model HmNet is set to be low (usually 0.0001) to finely adjust the weights of the model without losing or damaging the important features learned in the pre-training stage. The cross-entropy loss function and the Adam optimizer are used in this stage. The adaptive learning rate feature of the optimizer is particularly suitable for this scenario that requires fine adjustment.

[0093] During the fine-tuning process of the lesion segmentation neural network model HmNet, continuously optimize and adjust the lesion segmentation neural network model HmNet to ensure the efficiency of the training process and the performance of the model. Monitor the performance metrics of the model in real time, including loss and accuracy, and adjust the learning rate and regularization parameters based on these metrics. In addition, experiment with different optimization algorithms and data augmentation strategies to enhance the model's generalization ability for unseen data. The continuous optimization aims to respond to challenges found during the training process, such as overfitting or inappropriate learning rates, and gradually find the optimal parameter configuration and network settings through fine-tuning.

[0094] In the fifth step, evaluate the performance of the lesion segmentation neural network model HmNet, and further optimize and adjust the lesion segmentation neural network model HmNet according to the evaluation results.

[0095] By evaluating the performance of the trained model on the validation set, ensure that its performance in encoding hydatidiform mole pathological section images meets the expected goals. Select appropriate evaluation metrics, such as the intersection over union (IoU) for the segmentation task. Subsequently, further optimize and adjust the model according to the evaluation results. Adjust the parameters of the segmentation head MaskFormer or the training parameters to optimize the model performance. If necessary, be able to perform more fine-tuning training cycles to ensure continuous improvement of the model's performance on the validation set.

[0096] The hydatidiform mole lesion segmentation device based on the large image model described in the present invention includes:

[0097] A slice image extraction module for magnifying the microscopic structure of the hydatidiform mole slice and obtaining a scanned image of the hydatidiform mole slice containing the microscopic structure of the hydatidiform mole slice;

[0098] In this embodiment, the slice image extraction module can use an existing digital microscope in combination with a slice scanning module, or directly use an existing fully automatic digital slide scanner; finally, obtain a scanned image of the hydatidiform mole slice containing the microscopic structure of the hydatidiform mole slice through the slice image extraction module.

[0099] A lesion segmentation map generation module for cutting the scanned image of the hydatidiform mole slice according to set requirements, inputting a number of cut scanned image blocks into the lesion segmentation neural network model HmNet to obtain corresponding lesion segmentation maps, and then fusing all the lesion segmentation maps to obtain a preliminary complete slice lesion segmentation map;

[0100] Among them, the lesion segmentation neural network model HmNet includes a pre-trained large pathological image encoding model and a segmentation head MaskFormer; the large pathological image encoding model is used to extract features from the input pathological section image and output the feature map to the segmentation head MaskFormer; the segmentation head MaskFormer is used to perform segmentation prediction on the pathological section image to distinguish the background, hyperplasia, and hydropic lesions in the pathological section.

[0101] In this embodiment, the structure of the segmentation head MaskFormer and the process of distinguishing the background, hyperplasia, and hydropic lesions in the pathological section have been described in detail above and will not be elaborated here.

[0102] The slice lesion segmentation map post-processing module is used to perform image post-processing on the preliminarily complete slice lesion segmentation map to finally obtain the post-processed slice lesion segmentation map reflecting the actual lesion morphology.

[0103] The slice lesion segmentation map post-processing module first uses adaptive morphological operations or filtering techniques to remove the noise in the segmentation map to ensure the clarity and accuracy of the segmentation result; then, through multi-level edge detection and smoothing algorithms, it optimizes the boundaries of the lesion segmentation area to make the segmentation result more accurate and coherent; on this basis, it removes the too small areas with an area smaller than the set threshold caused by errors in the segmentation process to reduce meaningless noise interference; finally, it uses the adjacent segmentation area connection algorithm to effectively merge adjacent lesion areas to ensure the integrity and consistency of the hyperplasia and hydropic lesion areas. Through the above steps, the finally obtained post-processed slice lesion segmentation map can more truly reflect the actual morphology of the lesion, significantly improving the accuracy, reliability, and clinical application value of the segmentation result.

[0104] The specific structure and processing steps of the above-mentioned hydatidiform mole lesion segmentation device based on the large image model have been described in detail in the above-mentioned hydatidiform mole lesion segmentation method based on the large image model and will not be elaborated here.

Claims

1. A method for segmenting hydatidiform mole lesions based on an image large model, characterized in that, Including the following steps: A: Build a lesion segmentation neural network model based on an image encoding large model; by connecting a segmentation head MaskFormer for distinguishing background, hyperplasia, and edema lesions after a pre-trained pathological image encoding large model, and finally forming a lesion segmentation neural network model HmNet for segmenting hyperplasia and edema lesions after training; B: Perform he staining on the hydatidiform mole sections and obtain scanned images of the hydatidiform mole sections; C: Cut the scanned images of the hydatidiform mole sections to obtain several cut blocks of the scanned images of the hydatidiform mole sections; D: Input several cut blocks of the scanned images of the hydatidiform mole sections into the lesion segmentation neural network model HmNet to obtain a lesion segmentation map corresponding to each cut block, and then fuse the cut lesion segmentation maps corresponding to all cut blocks of the scanned images of the hydatidiform mole sections to obtain a preliminary complete sliced lesion segmentation map; E: Perform image post-processing on the preliminary complete sliced lesion segmentation map to finally obtain a post-processed sliced lesion segmentation map reflecting the actual lesion morphology.

2. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: The pre-trained pathological image encoding large model is based on the DinoV2 image encoding large model, and the DinoV2 image encoding large model is deeply trained through a segmentation task using a preset pathological slice dataset to obtain the pre-trained pathological image encoding large model.

3. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: The pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the segmentation head MaskFormer; after the segmentation head MaskFormer processes the feature map, it outputs three probability maps of background, hyperplasia, and edema lesions.

4. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: When the output background probability is less than or equal to the environmental probability threshold, this area is considered a lesion area; when the output hyperplasia lesion probability is greater than or equal to the edema lesion probability, this area is considered a hyperplasia lesion; when the hyperplasia lesion probability is less than the edema lesion probability, this area is considered an edema lesion.

5. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: In the probability maps output by the segmentation head MaskFormer, the sum of the probabilities of the pixel points being background, hyperplasia lesions, and edema lesions is 1.

6. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: In step E, first remove the noise in the sliced lesion segmentation map, then optimize the boundary of the lesion segmentation area, then remove the too small areas with an area less than the set threshold caused by errors during the segmentation process in the sliced lesion segmentation map, and finally connect the adjacent segmentation areas to form a continuous lesion area; finally, obtain an optimized sliced lesion segmentation map reflecting the actual lesion morphology.

7. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: The training method of the lesion segmentation neural network model HmNet is as follows: The first step is to deeply train the DinoV2 image encoding large model through a segmentation task using a preset pathological slice dataset to obtain the trained pathological image encoding large model; The second step is to cut the pictures in the labeled dataset according to the set size to construct a training dataset required for training the lesion segmentation neural network model HmNet; In the third step, for the lesion segmentation neural network model composed of the pathological image encoding large model and the segmentation head MaskFormer, freeze the parameters of the pre-trained pathological image encoding large model to keep them unchanged during the initial training. Only train the segmentation head MaskFormer with the training dataset to optimize and update the parameters of the segmentation head MaskFormer. In the fourth step, after the segmentation head MaskFormer has been preliminarily trained and verified to be effective, unfreeze all layers of the pre-trained DinoV2 image encoding large model so that the parameters of the entire network can be updated together, thereby integrating the learning results of the pre-trained layer and the newly trained segmentation head MaskFormer, and optimizing the parameters of the entire lesion segmentation neural network model HmNet through global fine-tuning. In the fifth step, perform performance evaluation on the lesion segmentation neural network model HmNet, and further optimize and adjust the lesion segmentation neural network model HmNet according to the evaluation results.

8. The method for segmenting hydatidiform mole lesions based on an image large model according to claim 1, wherein: The segmentation head MaskFormer includes several layers of Transformer encoders and decoders for further processing the feature map output by the pathological image encoding large model; multiple self-attention heads in each encoder of the segmentation head MaskFormer are used to capture different feature dimensions of the image.

9. A hydatidiform mole lesion segmentation device based on the lesion segmentation method according to any one of claims 1 to 8, characterized in that, It includes: A slice image extraction module for magnifying the structure of the hydatidiform mole slice and obtaining a hydatidiform mole slice scan image containing the structure of the hydatidiform mole slice. A lesion segmentation map generation module for cutting the hydatidiform mole slice scan image according to set requirements, inputting the obtained several scan image cuts into the lesion segmentation neural network model HmNet to obtain corresponding lesion segmentation maps, and then fusing all the lesion segmentation maps to obtain a preliminary complete slice lesion segmentation map. A slice lesion segmentation map post-processing module for performing image post-processing on the preliminary complete slice lesion segmentation map to finally obtain a post-processed slice lesion segmentation map reflecting the actual lesion morphology.

10. The method for a hydatidiform mole lesion segmentation device according to claim 9, wherein: The lesion segmentation neural network model HmNet includes a pre-trained pathological image encoding large model and a segmentation head MaskFormer; the pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the segmentation head MaskFormer; the segmentation head MaskFormer is used to perform segmentation prediction on the pathological slice image and output three probability maps of background, hyperplasia, and edema lesions.

Images