Grape tire edema lesion segmentation method and device based on image large model

Abstract

The invention discloses a grape tire edema lesion segmentation method and device based on a large image model, and the method comprises the steps: A, constructing a plurality of edema image segmentation neural network models for segmenting pathological sections of different scales, and obtaining an edema network; b, obtaining a grape tire slice scanning graph; c, acquiring a plurality of scanning graph blocks; d, inputting the scan image blocks into the edema network to obtain corresponding edema segmentation maps, and fusing the block edema segmentation maps corresponding to all the scan image blocks to obtain a preliminary complete slice edema segmentation map; and E, optimizing the preliminary complete slice edema segmentation image, and finally obtaining an optimized slice edema segmentation image reflecting the actual focus form. According to the method, accurate edema focus area identification can be carried out on the grape embryo slices, the edema distribution condition of the grape embryos is obtained, and therefore clinical doctors are assisted to carry out case screening more efficiently.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a method and a device for segmenting hydatidiform mole edema lesions based on a large image model. Background Art

[0002] Hydatidiform mole (HM) refers to a grape-shaped blister-like fetal mass formed by the placenta after pregnancy. Babies with hydatidiform mole often die or develop malformations, and very few full-term babies are born. Under normal circumstances, 10% to 20% of hydatidiform moles will develop into malignant hydatidiform mole and choriocarcinoma. These cancers can metastasize through blood groups and pose a life-threatening threat to the patient if not treated in time. Therefore, early pathological diagnosis of hydatidiform mole is of great significance to every pregnant woman with the disease.

[0003] In the prior art, there are two main ways to detect and screen hydatidiform mole. The first is to manually observe the sections under a microscope, and the second is to detect genes related to hydatidiform mole.

[0004] In the first method, pathologists generally use 5*10x and 10*10x microscopes to observe multiple sections of the patient, and then make a comprehensive diagnosis based on experience and the morphology of the tissue cells in the sections. The diagnosis of hydatidiform mole is mainly made by observing the characteristics of the villi in the sections. The pathological characteristics of the sections are mainly villous trophoblastic hyperplasia and interstitial edema inside the villi.

[0005] Pathologists in gynecological hospitals need to spend a lot of time every day diagnosing diseases such as hydatidiform mole, which have a lower risk factor than tumors. Most of these patients are not sick, but this takes up a lot of the pathologists' working time.

[0006] However, since clinical diagnosis of hydatidiform mole is mainly done by doctors through manual screening of slices, it is difficult to guarantee accuracy, especially for hydatidiform mole before 12 weeks. Since the hydatidiform mole has not reached maturity, the lesion is not fully developed, and the tissue morphology is similar to that of normal hydatidiform mole slices, making it difficult to distinguish. This results in an extremely low clinical diagnostic accuracy of less than 50%.

[0007] In the second method, the invention patent with application number 201310027715.1 and titled Gene chip, detection reagent and kit for detecting NLRP7 gene discloses that by detecting NLRP7 gene SNPs related to hydatidiform mole, it is of great significance to achieve clinical diagnosis of hydatidiform mole and early screening of high-risk populations and early preventive intervention, and can be widely used for clinical efficient screening of high-risk populations for hydatidiform mole. This invention patent constructs a gene chip detection system for screening high-risk populations for NLRP7 gene polymorphisms related to hydatidiform mole. The gene chip includes a solid phase carrier and an oligonucleotide probe synthesized on the carrier. The detection reagent includes a gene chip and 18 pairs of PCR primers for amplifying each SNPs in the sample. The kit includes a detection reagent, a negative control sample and a positive control sample. This invention patent can quickly and accurately detect each related SNPs site of NLRP7 gene in clinical samples, which is of great significance for clinical diagnosis of hydatidiform mole and early screening of high-risk populations and early preventive intervention.

[0008] Although screening for hydatidiform mole by testing genes is necessary, testing the NLRP7 gene for hydatidiform mole increases the number of test kit testing steps, which makes the entire testing cycle longer. It also involves the production of chips, reagents, and test kits, which significantly increases the screening cost. Its application scope in the clinical diagnosis of hydatidiform mole is very limited, and it is not easy to promote and apply.

[0009] Based on the above-mentioned current situation of small number of pathologists, low efficiency and low accuracy of manual screening of slices by physicians, high cost and long cycle of genetic testing screening, it is necessary to develop a complete set of methods and devices from automatically acquiring images with a microscope to generating distribution maps of pathological characteristics such as edema and hyperplasia, so as to assist clinicians in screening cases more efficiently.

[0010] Clinical pathologists mainly judge whether it is a hydatidiform mole based on pathological features such as villous stromal edema, diffuse hyperplasia of trophoblastic cells at the villous margins, and information such as the patient's menopausal duration and pregnancy history. Among them, the pathological feature of villous stromal edema is the more critical basis for diagnosis. Summary of the invention

[0011] The purpose of the present invention is to provide a method and device for segmenting edema lesions of hydatidiform mole based on a large image model, which can accurately identify the edema lesion area of ​​hydatidiform mole slices and obtain the edema distribution of hydatidiform mole, thereby assisting clinicians to perform case screening more efficiently.

[0012] The present invention adopts the following technical solutions:

[0013] A method for segmenting hydatidiform mole edema lesions based on a large image model comprises the following steps:

[0014] A: Construct multiple edema image segmentation neural network models for segmenting pathological slices of different scales. Each edema image segmentation neural network model consists of a pre-trained pathological image encoding large model and a segmentation head MaskFormer connected thereto for distinguishing the background and edema lesions in the pathological slices. After training, an edema network HydropicNet containing multiple edema image segmentation neural network models is obtained. Multiple edema image segmentation neural network models are used to segment edema lesions for pathological slices of different scales. The edema network HydropicNet finally averages the segmentation results of multiple different scales by scaling them to a unified scale to obtain the final edema segmentation map.

[0015] B: The hydatidiform mole section was stained with he stain and a scan of the hydatidiform mole section was obtained;

[0016] C: Slice the hydatidiform mole slice scan image to obtain a number of scan image slices;

[0017] D: Input the blocks of several scanned images in step C into the edema network HydropicNet to obtain the corresponding edema segmentation map, and then fuse the block edema segmentation maps corresponding to all the scanned image blocks to obtain a preliminary complete slice edema segmentation map;

[0018] E: The initial complete slice edema segmentation map image is optimized, and finally an optimized slice edema segmentation map reflecting the actual lesion morphology is obtained.

[0019] In the step A, the pathological image coding model may adopt the DinoV2 image coding model, and the DinoV2 image coding model is deeply trained by the segmentation task using the preset pathological slice data set to obtain the trained pathological image coding model;

[0020] In the step A, the segmentation head MaskFormer includes several layers of Transformer encoders and decoders, which are used to further process the feature map output by the edema image segmentation neural network model. The multiple attention heads of each layer in the Transformer encoder are used to capture different feature dimensions of the image.

[0021] In the step A, for edema segmentation maps of different scales output by multiple edema image segmentation neural network models, the edema network HydropicNet first uses a bilinear interpolation method to scale the probability results of different scales to the same scale to ensure that each pixel of the output probability map represents the same position, and then averages the multiple edema segmentation maps output by the multiple edema image segmentation neural network models to obtain the final edema segmentation map.

[0022] In the step A, a probability threshold θ is also set. When the output edema lesion probability is greater than or equal to the probability threshold θ, the area is considered to be an edema lesion area.

[0023] The pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the corresponding segmentation head MaskFormer; the segmentation head MaskFormer is used to perform segmentation prediction on the pathological slice image, distinguish the background and edema lesions in the pathological slice, and finally output two types of probability maps of background and edema lesions. Each pixel point in the probability map corresponds to two probability values, namely the probability of belonging to the background and the probability of belonging to the edema lesion.

[0024] In the step E, the noise in the edema lesion segmentation map is first removed, and then the boundary of the lesion segmentation area is optimized. Then, the small area in the edema lesion segmentation map whose area is less than the set threshold due to errors in the segmentation process is removed, and finally, the adjacent segmented areas are connected to form a continuous edema lesion area; finally, an optimized slice edema segmentation map reflecting the actual lesion morphology is obtained.

[0025] The training method of the edema network HydropicNet is:

[0026] The first step is to use the preset pathology slice data set to perform deep training on the pathology slice image coding model through segmentation tasks to obtain the trained pathology slice image coding model;

[0027] The second step is to cut the images in the labeled dataset into blocks according to the set size, and then scale each block into a set standardized image to construct the training dataset required for the HydropicNet training of various sizes.

[0028] The third step is to freeze the parameters of the pre-trained pathological slice image encoding model so that it remains unchanged in the initial training. The segmentation head MaskFormer is trained only through the training data set to optimize and update the parameters of the segmentation head MaskFormer.

[0029] The fourth step is to unfreeze all layers of the pre-trained pathological image encoding model after the segmentation head MaskFormer has been initially trained and verified to be effective, so that the parameters of the entire edema network HydropicNet can be updated together, thereby integrating the learning results of the pre-trained pathological image encoding model and the newly trained task head segmentation head MaskFormer, and optimizing the parameters of the entire edema network HydropicNet through global fine-tuning;

[0030] The fifth step is to evaluate the performance of the HydropicNet and further optimize and adjust the HydropicNet based on the evaluation results.

[0031] The device for segmenting hydatidiform mole edema lesions based on a large image model includes:

[0032] A slice image extraction module is used to amplify the structure of the hydatidiform mole slice and obtain a hydatidiform mole slice scan image containing the hydatidiform mole slice structure;

[0033] The slice edema segmentation map generation module is used to slice the hydatidiform mole slice scan image according to the set requirements, and input the obtained scan image slices into the edema network HydropicNet to obtain the corresponding slice edema segmentation map, and then fuse all the slice edema segmentation maps to obtain a preliminary and complete slice edema segmentation map;

[0034] The slice edema segmentation map post-processing module is used to perform image optimization processing on the preliminary and complete slice edema segmentation map, and finally obtain an optimized slice edema segmentation map reflecting the actual lesion morphology.

[0035] The edema network HydropicNet includes multiple edema image segmentation neural network models, each of which includes a pathology image encoding large model and a segmentation head MaskFormer; the multiple pathology image encoding large models are respectively used to extract features from input pathology slices of different scales, and output feature maps to the corresponding segmentation head MaskFormer; the segmentation head MaskFormer is used to distinguish the background and edema lesions in the pathology slices through segmentation prediction, and output the segmentation results; the edema network HydropicNet finally scales the segmentation results of multiple different scales to a unified scale and then averages them to obtain the final edema segmentation map.

[0036] The present invention combines the image coding large model technology with the hydatidiform mole slice pathology recognition technology, cuts the multi-scale hydatidiform mole slice scan image into blocks and sends it to the edema network HydropicNet. After integration and optimization, the final slice edema segmentation map of the hydatidiform mole slice scan image is obtained. The slice edema segmentation map can directly provide data support for clinicians, making it convenient for doctors to intuitively obtain the distribution of edema, thereby assisting clinicians to screen cases more efficiently.

[0037] In addition, the image coding model used in the present invention has been deeply trained on the pathological slice data set, and has a strong pathological slice feature extraction capability; at the same time, the segmentation head MaskFormer has been deeply trained separately, and the network structure has been optimized through a specific training method, which effectively improves the accuracy of the slice edema segmentation map. Finally, through the joint training of the edema network HydropicNet, the edema lesions in the pathological slices can be more accurately identified and segmented, improving the overall segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the method for segmenting hydatidiform mole edema lesions based on a large image model in the present invention;

[0039] Figure 2 It is a scanned image of a hydatidiform mole section in the present invention;

[0040] Figure 3 This is the actual slice edema label map in the present invention;

[0041] Figure 4 It is the network predicted slice edema segmentation map in the present invention;

[0042] Figure 5 The present invention is a flowchart for obtaining a training data set for annotating a hydatidiform mole slice scan image disclosed in the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0044] like Figures 1 to 5 As shown, the method for segmenting hydatidiform mole edema lesions based on a large image model of the present invention comprises the following steps:

[0045] A: Construct multiple edema image segmentation neural network models for segmenting pathological sections of different scales. Each edema image segmentation neural network model consists of a pre-trained pathological image encoding large model and a segmentation head MaskFormer connected thereto for distinguishing the background and edema lesions in the pathological sections. After training, the edema network HydropicNet containing multiple edema image segmentation neural network models is finally obtained. Multiple edema image segmentation neural network models in the edema network HydropicNet are used to segment edema lesions for pathological sections of different scales. The edema network HydropicNet finally averages the segmentation results of multiple different scales by scaling them to a unified scale to obtain the final edema segmentation map.

[0046] In the present invention, each edema image segmentation neural network model is established based on the pathology image coding model in cooperation with the segmentation head MaskFormer; the pathology image coding model can adopt the DinoV2 image coding model, and use the preset pathology slice data set to perform deep training on the DinoV2 image coding model through the segmentation task to obtain the trained pathology image coding model; each edema image segmentation neural network model is independent in a separate training process, that is, the parameters of each edema image segmentation neural network model and the segmentation head are different in the end,

[0047] In the present invention, a segmentation head MaskFormer is connected after the output layer of each pre-trained pathological image encoding model, which can perform segmentation prediction on the pathological section image to distinguish the background and edema lesions (Hydropic) in the pathological section; since the segmentation task is to distinguish the background and edema lesions, the output category of MaskFormer is set to 2.

[0048] In this embodiment, the pathological slice data set contains multiple pathological types, including cancer, inflammation and benign lesions; among them, cancer includes breast cancer, lung cancer and gastric cancer, etc., and the segmentation content is mainly the cancerous area and its boundaries; inflammation includes hepatitis and gastritis, etc., and the segmentation content is mainly the inflammatory reaction area and its boundaries; benign lesions include benign tumors and cysts, etc., and the segmentation content is mainly the lesion area and its boundaries.

[0049] In the present invention, a total of three edema image segmentation neural network models for segmenting pathological slices of different scales are constructed. The input of the first edema image segmentation neural network model is to scale the original pathological picture of 3000×3000 to 512×512 as input, the input of the second edema image segmentation neural network model is to scale the original pathological picture of 6000×6000 to 512×512 as input, and the input of the third edema image segmentation neural network model is to scale the original pathological picture of 9000×9000 to 512×512 as input. The above three edema image segmentation neural network models are respectively suitable for the segmentation of edema lesions of small, medium and large sizes and can achieve the optimal effect. The pathological slice image size of the input pathological image encoding large model in the three edema image segmentation neural network models is 512×512 pixels; each pathological image encoding large model extracts features from the input pathological slice image, outputs feature maps with dimensions of 16×16×768, and then outputs the feature maps to the corresponding segmentation head MaskFormer;

[0050] In the present invention, the segmentation head MaskFormer contains several layers of Transformer encoders and decoders, which are used to further process the feature maps output by the DinoV2 image coding model. The Transformer encoder has 6 layers, each layer contains 8 attention heads, and the feature embedding dimension is 768 dimensions; each self-attention head can capture different feature dimensions of the image and enhance the segmentation ability of the model.

[0051] After processing the feature map, the segmentation head MaskFormer outputs two probability maps: background and edema lesions (Hydropic). The dimension of the probability map is 512×512×2, and each pixel corresponds to two probability values, namely the probability of belonging to the background and the probability of belonging to the edema lesion.

[0052] In the present invention, in the probability map output by the segmentation head MaskFormer, the probability that a pixel is background is recorded as P background , the probability that a pixel is an edema lesion is recorded as P hydropic , these two probabilities satisfy P background +P hydropic =1.

[0053] In the present invention, for the probability results of different scales output by the three edema lesion image segmentation neural networks, namely, edema segmentation maps, a bilinear interpolation method is first used to scale the probability results of different scales to the same scale to ensure that each pixel point of the output probability map represents the same position, and then the three edema segmentation maps output by the three edema lesion image segmentation neural networks are averaged to obtain the final edema segmentation map.

[0054] In order to determine whether there is a lesion, the present invention also defines a probability threshold θ. When the output edema lesion probability P hydropic When it is greater than or equal to the probability threshold θ, the area is considered to be an edema lesion area, that is:

[0055]

[0056] B: The hydatidiform mole section was stained with he stain and a scan of the hydatidiform mole section was obtained;

[0057] In step B, the hydatidiform mole slice is firstly stained with HE, and then the hydatidiform mole slice stained with HE is placed on the stage of a digital microscope, and a digital microscope is selected; then the digital microscope is automatically focused using an autofocus module so that a clear hydatidiform mole slice can be seen in the field of view of the microscope; finally, a slice scanning module is used to obtain a scanned image of the hydatidiform mole slice under a microscope.

[0058] In step B, an existing fully automatic digital slice scanner may also be directly used to finally obtain a hydatidiform mole slice scan image.

[0059] C: Slice the hydatidiform mole slice scan image to obtain a number of scan image slices;

[0060] Since the scanned image of the hydatidiform mole obtained in step B has a high resolution and a pixel size of 65536×65536, and the pixel size of the input image of the edema network HydropicNet is required to be 512×512, the slice needs to be cut into blocks. In step C, the high-resolution scanned image of the hydatidiform mole is cut into blocks to obtain multiple scanned image blocks with a pixel size of 512×512. For the scanned image of the hydatidiform mole with a pixel size of 65536×65536, 128×128 scanned image blocks with a pixel size of 512×512 are finally obtained;

[0061] D: Input the blocks of several scanned images in step C into the edema network HydropicNet to obtain the corresponding edema segmentation map, and then fuse the block edema segmentation maps corresponding to all the scanned image blocks to obtain a preliminary complete slice edema segmentation map;

[0062] E: Optimize the initial complete slice edema segmentation map image, and finally obtain the optimized slice edema segmentation map reflecting the actual lesion morphology;

[0063] In the present invention, the noise in the edema lesion segmentation map can be firstly removed by morphological operation or filtering technology to ensure the clarity and accuracy of the segmentation result; then, the boundary of the lesion segmentation area is optimized by edge detection and smoothing algorithm to make the segmentation result more accurate and coherent; then, the small area in the segmentation map whose area is less than the set threshold due to the error in the segmentation process is removed; finally, the adjacent segmented areas are connected to form a continuous edema lesion area to ensure the integrity of the edema lesion; finally, the optimized slice edema segmentation map reflecting the actual lesion morphology is obtained, which can effectively improve the accuracy and reliability of the segmentation result.

[0064] In this embodiment, morphological operations can be erosion and dilation, or opening and closing operations, etc.; morphological operations can effectively remove isolated small noise points and fill small holes; filtering technology can use Gaussian filtering or median filtering, etc., which can smooth the noise in the image and retain the main structural features.

[0065] Edge detection can use the Canny edge detection algorithm to determine the edge position by detecting the gradient change in the image. The smoothing algorithm can use bilateral filtering or other edge-preserving smoothing algorithms to smooth the boundaries while retaining the edge clarity.

[0066] When removing areas with too small an area in the segmentation map, by calculating the area of ​​each segmented area and removing areas below a set threshold (such as 10 pixels), the above-mentioned small noise areas that have no clinical significance can be removed, which can effectively reduce errors and improve the accuracy of the segmentation results.

[0067] When connecting adjacent segmented regions, the connected component labeling algorithm can be used to connect adjacent pixels belonging to the same category to form a complete region. By merging these regions, it is ensured that the segmentation results can accurately reflect the morphology of the actual lesion.

[0068] Through the above optimization processing steps, not only the accuracy and reliability of the segmentation results are effectively improved, but also the clinical usability of the segmentation results is ensured, which can provide doctors with clearer and more accurate diagnostic basis.

[0069] In the present invention, the edema network HydropicNet needs to be trained on a hydatidiform mole slice data set. This embodiment also provides a method for training the edema network HydropicNet.

[0070] To train the network, you need training images to feed into the network and train it.

[0071] In this example, the morphological characteristics of hydatidiform mole required for the actual diagnosis of hydatidiform mole are combined, and the annotation results of 1000 typical hydatidiform mole slice scan images are obtained through hydatidiform mole slice annotation and annotation review. Each slice needs to be annotated with detailed edema lesions. Specific annotation examples can be found in Figure 1 .

[0072] At present, the bottleneck of medical imaging is the extreme lack of high-quality annotated data sets, and there is no good annotated data set for hydatidiform mole worldwide. The recognition rate of deep networks is based on a good data set, so first of all, standardized, reasonable, and strict annotation is required to obtain a good data set. In the present invention, the training data set is obtained by the following method:

[0073] Step 101: Develop a reliable annotation scheme

[0074] After several proposed annotation schemes were reviewed and confirmed by multiple professional clinicians, a single annotation scheme was finally determined, namely edema annotation. Specific annotation samples can be found in Figure 1 Edema annotation is to circle the trophoblast area with diffuse edema at the villous margin.

[0075] Step 102: Training of labelers

[0076] Several annotators with relevant medical knowledge were selected for training.

[0077] Step 103: Initial labeling by labelers

[0078] According to the labeling regulations, each annotator is responsible for about 80 scan slices and annotates the edema for network training.

[0079] Step 104: Pathologist Review

[0080] The preliminary annotations obtained in step 103 are strictly reviewed by clinicians with long-term clinical experience and fed back to the annotators.

[0081] Step 105: Detailed annotation by annotators

[0082] The annotation review results are analyzed and modified, and the annotators unify the annotation standards to obtain the final annotation data set.

[0083] Step 106: Automatic network annotation

[0084] Based on the annotated data set obtained in step 105, the edema network HydropicNet is trained; and the trained edema network HydropicNet is used to perform image semantic segmentation on the newly added slice scan images to obtain a new annotated data set, that is, the network automatically annotates the new slice scan images to expand the data set, and then train a network with better robustness.

[0085] In the present invention, the training method of the edema network HydropicNet is:

[0086] The first step is to use the preset pathological slice dataset to perform deep training on the DinoV2 image coding model through segmentation tasks to obtain the trained pathological slice image coding model;

[0087] The second step is to cut the images in the labeled dataset into three sizes: 3000×3000, 6000×6000, and 9000×9000, and then scale each block to a standardized image of 512×512 to construct the training datasets required for the training of the HydropicNet of three sizes.

[0088] The third step is to freeze the parameters of the pre-trained pathological slice image encoding model so that it remains unchanged in the initial training. The segmentation head MaskFormer is trained only through the training data set to optimize and update the parameters of the segmentation head MaskFormer.

[0089] The above steps can avoid the destruction of the feature extraction capability of the pathology image coding large model by the initial training, and can also make full use of the excellent performance of the pre-trained pathology image coding large model in pathology slice feature extraction, so that the segmentation head can effectively learn the segmentation task.

[0090] When training the segmentation head MaskFormer, we selected the same or similar loss function and optimization algorithm as the pathological image encoding model, optimized the parameters of the task head through the back propagation algorithm, and made the segmentation head MaskFormer adapt to the data distribution of hydatidiform mole slice images. We also set the learning rate, training cycle, batch size and other training parameters to ensure the stable convergence of the segmentation head MaskFormer during the training process. We monitored the training loss and validation set performance of the model in real time to ensure the stability and effectiveness of the training process.

[0091] The fourth step is to unfreeze all layers of the pre-trained pathological image encoding model after the segmentation head MaskFormer has been initially trained and verified to be effective, so that the parameters of the entire network can be updated together, thereby integrating the learning results of the pre-trained layers and the newly trained task head segmentation head MaskFormer, and optimizing the parameters of the entire edema network HydropicNet through global fine-tuning.

[0092] By jointly training the HydropicNet network, the feature extraction and segmentation capabilities of the model can be further optimized, and the accuracy and reliability of the segmentation results can be improved.

[0093] The initial learning rate of the HydropicNet is set to a low value (usually 0.0001) to fine-tune the model weights without losing or destroying important features learned in the pre-training phase. This phase uses the cross entropy loss function and the Adam optimizer, whose adaptive learning rate feature is particularly suitable for such scenarios that require fine-tuning.

[0094] During the fine-tuning process of the HydropicNet, the HydropicNet is continuously optimized and adjusted to ensure the efficiency of the training process and the performance of the model. The performance indicators of the model, including loss and accuracy, are monitored in real time, and the learning rate and regularization parameters are adjusted according to these indicators. In addition, different optimization algorithms and data enhancement strategies are experimented to enhance the generalization ability of the model to unseen data. Continuous optimization aims to respond to challenges found during training, such as overfitting or inappropriate learning rates, and gradually find the optimal parameter configuration and network settings through fine-tuning.

[0095] The fifth step is to evaluate the performance of the HydropicNet and further optimize and adjust the HydropicNet based on the evaluation results.

[0096] Evaluate the performance of the trained model on the validation set to ensure that it performs as expected on the pathology image encoding. Choose an appropriate evaluation metric, such as the Intersection over Union (IoU) for the segmentation task. Then further optimize and tune the model based on the evaluation results. Adjust the parameters of the task head MaskFormer or the training parameters to optimize the model performance. If necessary, perform more fine-tuning training cycles to ensure that the model's performance on the validation set continues to improve.

[0097] The device for segmenting hydatidiform mole edema lesions based on a large image model of the present invention comprises:

[0098] A slice image extraction module is used to amplify the microstructure of the hydatidiform mole slice and obtain a scanned image of the hydatidiform mole slice containing the microstructure of the hydatidiform mole slice;

[0099] In this embodiment, the slice image extraction module can use an existing digital microscope in conjunction with a slice scanning module, or can directly use an existing fully automatic digital slice scanner; ultimately, a hydatidiform mole slice scan image containing the hydatidiform mole slice microstructure is obtained through the slice image extraction module.

[0100] The slice edema segmentation map generation module is used to slice the hydatidiform mole slice scan image according to the set requirements, and input the obtained scan image slices into the edema network HydropicNet to obtain the corresponding slice edema segmentation map, and then fuse all the slice edema segmentation maps to obtain a preliminary and complete slice edema segmentation map;

[0101] Among them, the edema network HydropicNet includes multiple edema image segmentation neural network models, each of which includes a pathology image encoding large model and a segmentation head MaskFormer; the multiple pathology image encoding large models are used to extract features from input pathology slices of different scales, and output feature maps to the corresponding segmentation head MaskFormer; the segmentation head MaskFormer is used to segment and predict the pathology slice image, distinguish the background and edema lesions in the pathology slice and output the segmentation results; the edema network HydropicNet finally scales the segmentation results of multiple different scales to a unified scale and then averages them to obtain the final edema segmentation map;

[0102] The slice edema segmentation map post-processing module is used to perform image optimization processing on the preliminary and complete slice edema segmentation map, and finally obtain an optimized slice edema segmentation map reflecting the actual lesion morphology.

[0103] The post-processing module of the slice edema segmentation map first removes the noise in the segmentation map through morphological operations or filtering technology to ensure the clarity and accuracy of the segmentation results; then, the boundary of the lesion segmentation area is optimized through edge detection and smoothing algorithms to make the segmentation results more accurate and coherent; then, the areas in the segmentation map that are too small due to errors in the segmentation process are removed; finally, the adjacent segmented areas are connected to form a continuous edema lesion area to ensure the integrity of the edema lesion; finally, the optimized slice edema segmentation map reflecting the actual lesion morphology is obtained.

[0104] The specific processing process and processing steps of the above-mentioned hydatidiform mole edema lesion segmentation device based on the large image model have been described in detail in the above-mentioned hydatidiform mole edema lesion segmentation method based on the large image model, and will not be repeated here.

Claims

1. A method for segmenting hydatidiform mole edema lesions based on a large image model, characterized in that: The following steps are involved: A: Construct multiple edema image segmentation neural network models for segmenting pathological slices of different scales. Each edema image segmentation neural network model consists of a pre-trained pathological image encoding large model and a segmentation head MaskFormer connected thereto for distinguishing the background and edema lesions in the pathological slices. After training, an edema network HydropicNet containing multiple edema image segmentation neural network models is obtained. Multiple edema image segmentation neural network models are used to segment edema lesions for pathological slices of different scales. The edema network HydropicNet finally averages the segmentation results of multiple different scales by scaling them to a unified scale to obtain the final edema segmentation map. B: The hydatidiform mole section was stained with he stain and a scan of the hydatidiform mole section was obtained; C: Slice the hydatidiform mole slice scan image to obtain a number of scan image slices; D: Input the blocks of several scanned images in step C into the edema network HydropicNet to obtain the corresponding edema segmentation map, and then fuse the block edema segmentation maps corresponding to all the scanned image blocks to obtain a preliminary complete slice edema segmentation map; E: The initial complete slice edema segmentation map image is optimized, and finally an optimized slice edema segmentation map reflecting the actual lesion morphology is obtained.

2. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 1, characterized in that: In the step A, the pathology image coding large model can adopt the DinoV2 image coding large model, and use the preset pathology slice data set to deeply train the DinoV2 image coding large model through the segmentation task to obtain the trained pathology image coding large model.

3. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 1, characterized in that: In the step A, the segmentation head MaskFormer includes several layers of Transformer encoders and decoders, which are used to further process the feature map output by the edema image segmentation neural network model. The multiple attention heads of each layer in the Transformer encoder are used to capture different feature dimensions of the image.

4. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 1, characterized in that: In the step A, for edema segmentation maps of different scales output by multiple edema image segmentation neural network models, the edema network HydropicNet first scales the probability results of different scales to the same scale to ensure that each pixel of the output probability map represents the same position, and then averages the multiple edema segmentation maps output by the multiple edema image segmentation neural network models to obtain the final edema segmentation map.

5. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 3, characterized in that: In the step A, a probability threshold θ is also set. When the output edema lesion probability is greater than or equal to the probability threshold θ, the area is considered to be an edema lesion area.

6. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 1, characterized in that: The pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the corresponding segmentation head MaskFormer; the segmentation head MaskFormer is used to perform segmentation prediction on the pathological slice image, distinguish the background and edema lesions in the pathological slice, and finally output two types of probability maps of background and edema lesions. Each pixel point in the probability map corresponds to two probability values, namely the probability of belonging to the background and the probability of belonging to the edema lesion.

7. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 1, characterized in that: In the step E, the noise in the edema lesion segmentation map is first removed, and then the boundary of the lesion segmentation area is optimized. Then, the small area in the edema lesion segmentation map whose area is less than the set threshold due to errors in the segmentation process is removed, and finally, the adjacent segmented areas are connected to form a continuous edema lesion area; finally, an optimized slice edema segmentation map reflecting the actual lesion morphology is obtained.

8. The method for segmenting hydatidiform mole edema lesions based on a large image model according to claim 1, characterized in that: The training method of the edema network HydropicNet is: The first step is to use the preset pathology slice data set to perform deep training on the pathology slice image coding model through segmentation tasks to obtain the trained pathology slice image coding model; The second step is to cut the images in the labeled dataset into blocks according to the set size, and then scale each block into a set standardized image to construct the training dataset required for the HydropicNet training of various sizes. The third step is to freeze the parameters of the pre-trained pathological slice image encoding model so that it remains unchanged in the initial training. The segmentation head MaskFormer is trained only through the training data set to optimize and update the parameters of the segmentation head MaskFormer. The fourth step is to unfreeze all layers of the pre-trained pathological image encoding model after the segmentation head MaskFormer has been initially trained and verified to be effective, so that the parameters of the entire edema network HydropicNet can be updated together, thereby integrating the learning results of the pre-trained pathological image encoding model and the newly trained task head segmentation head MaskFormer, and optimizing the parameters of the entire edema network HydropicNet through global fine-tuning; The fifth step is to evaluate the performance of the HydropicNet and further optimize and adjust the HydropicNet based on the evaluation results.

9. A device for segmenting hydatidiform mole edema lesions based on a large image model using any one of the methods of claims 1 to 8, characterized in that: include: A slice image extraction module is used to amplify the structure of the hydatidiform mole slice and obtain a hydatidiform mole slice scan image containing the hydatidiform mole slice structure; The slice edema segmentation map generation module is used to slice the hydatidiform mole slice scan image according to the set requirements, and input the obtained scan image slices into the edema network HydropicNet to obtain the corresponding slice edema segmentation map, and then fuse all the slice edema segmentation maps to obtain a preliminary and complete slice edema segmentation map; The slice edema segmentation map post-processing module is used to perform image optimization processing on the preliminary and complete slice edema segmentation map, and finally obtain an optimized slice edema segmentation map reflecting the actual lesion morphology.

10. The device for segmenting hydatidiform mole edema lesions based on a large image model according to claim 9, characterized in that: The edema network HydropicNet includes multiple edema image segmentation neural network models, each of which includes a pathology image encoding large model and a segmentation head MaskFormer; the multiple pathology image encoding large models are respectively used to extract features from input pathology slices of different scales, and output feature maps to the corresponding segmentation head MaskFormer; the segmentation head MaskFormer is used to distinguish the background and edema lesions in the pathology slices through segmentation prediction, and output the segmentation results; the edema network HydropicNet finally scales the segmentation results of multiple different scales to a unified scale and then averages them to obtain the final edema segmentation map.

Images