Grape tire hyperplasia focus segmentation method and device based on image large model

Abstract

The invention discloses a grape tire hyperplasia focus segmentation method and device based on an image large model. The method comprises the following steps: A, constructing a villus segmentation neural network model; b, constructing a hyperplasia segmentation neural network model; c, obtaining a grape tire slice scanning graph; d, slicing the grape tire slice scan image to obtain scan image slices; e, inputting the scan image blocks into a villus network to obtain a villus region segmentation image, inputting the villus region segmentation image and the grape tire slice scan image blocks into a proliferation network at the same time to obtain a corresponding proliferation segmentation image, and fusing the block proliferation segmentation images corresponding to all the grape tire slice scan image blocks to obtain a fusion image; obtaining a preliminary complete slice hyperplasia segmentation map; and F, performing image optimization on the preliminary complete slice hyperplasia segmentation image, and finally obtaining an optimized slice hyperplasia segmentation image reflecting the actual focus form. According to the method, accurate hyperplasia focus area identification can be carried out on the grape embryo slices, so that the hyperplasia distribution condition of the grape embryos is obtained, and clinical doctors are assisted to carry out case screening more efficiently.

Description

Technical Field

[0001] The invention relates to the field of medical image processing, and in particular to a method and device for segmenting hydatidiform mole hyperplasia lesions based on a large image model. Background Art

[0002] Hydatidiform mole (HM) refers to a grape-shaped blister-like fetal mass formed by the placenta after pregnancy. Babies with hydatidiform mole often die or develop malformations, and very few full-term babies are born. Under normal circumstances, 10% to 20% of hydatidiform moles will develop into malignant hydatidiform mole and choriocarcinoma. These cancers can metastasize through blood groups and pose a life-threatening threat to the patient if not treated in time. Therefore, early pathological diagnosis of hydatidiform mole is of great significance to every pregnant woman with the disease.

[0003] In the prior art, there are two main ways to detect and screen hydatidiform mole. The first is to manually observe the sections under a microscope, and the second is to detect genes related to hydatidiform mole.

[0004] In the first method, pathologists generally use 5*10x and 10*10x microscopes to observe multiple sections of the patient, and then make a comprehensive diagnosis based on experience and the morphology of the tissue cells in the sections. The diagnosis of hydatidiform mole is mainly made by observing the characteristics of the villi in the sections. The pathological characteristics of the sections are mainly villous trophoblastic hyperplasia and interstitial edema inside the villi.

[0005] Pathologists in gynecological hospitals need to spend a lot of time every day diagnosing diseases such as hydatidiform mole, which have a lower risk factor than tumors. Most of these patients are not sick, but this takes up a lot of the pathologists' working time.

[0006] However, there are currently about 15,000 pathologists in China, resulting in a large talent gap and low detection efficiency. In addition, since the diagnosis of clinical hydatidiform mole is mainly performed by doctors through manual screening of sections, it is difficult to guarantee accuracy, especially for hydatidiform mole before 12 weeks. Since the hydatidiform mole has not reached maturity, the lesion is not fully developed, and the tissue morphology is similar to that of normal hydatidiform mole sections, making it difficult to distinguish. This results in an extremely low clinical diagnostic accuracy of less than 50%.

[0007] In the second method, the invention patent with application number 201310027715.1 and titled "Gene chip, detection reagent and kit for detecting NLRP7 gene" discloses that by detecting NLRP7 gene SNPs related to hydatidiform mole, it is of great significance to achieve clinical diagnosis of hydatidiform mole and early screening of high-risk populations and early preventive intervention, and can be widely used in clinical high-risk screening of hydatidiform mole. This invention patent constructs a gene chip detection system for screening high-risk populations of NLRP7 gene polymorphisms related to hydatidiform mole. The gene chip includes a solid phase carrier and an oligonucleotide probe synthesized on the carrier. The detection reagent includes a gene chip and 18 pairs of PCR primers for amplifying each SNPs in the sample. The kit includes a detection reagent, a negative control sample and a positive control sample. This invention patent can quickly and accurately detect each related SNPs site of the NLRP7 gene in clinical samples, which is of great significance for the clinical diagnosis of hydatidiform mole and early screening of high-risk populations and early preventive intervention.

[0008] Although screening for hydatidiform mole by testing genes is necessary, testing the NLRP7 gene for hydatidiform mole increases the number of test kit testing steps, which makes the entire testing cycle longer. It also involves the production of chips, reagents, and test kits, which significantly increases the screening cost. Its application scope in the clinical diagnosis of hydatidiform mole is very limited, and it is not easy to promote and apply.

[0009] Based on the above-mentioned current situation of small number of pathologists, low efficiency and low accuracy of manual screening of slices by physicians, high cost and long cycle of genetic testing screening, it is necessary to develop a complete set of methods and devices from automatically acquiring images with a microscope to generating distribution maps of pathological characteristics such as edema and hyperplasia, so as to assist clinicians in screening cases more efficiently. Summary of the invention

[0010] The purpose of the present invention is to provide a method and device for segmenting hydatidiform mole hyperplasia lesions based on a large image model, which can accurately identify the hyperplastic lesion area of ​​hydatidiform mole slices, thereby obtaining the hyperplasia distribution of hydatidiform mole, thereby assisting clinicians to perform case screening more efficiently.

[0011] The present invention adopts the following technical solutions:

[0012] A method for segmenting hydatidiform mole lesions based on a large image model comprises the following steps:

[0013] A: Construct a neural network model for villus segmentation; by connecting the first segmentation head MaskFormer used to distinguish the background and villus area in the pathological section after the pre-trained pathological image encoding model, the villus network VilliNet for villus segmentation is finally formed after training;

[0014] B: Constructing a hyperplasia segmentation neural network model; first, a convolutional neural network for realizing the preliminary feature fusion of pathological sections and villus area segmentation maps is set in front of the pre-trained pathological image encoding large model, and the second segmentation head MaskFormer for distinguishing the background and hyperplasia lesions in pathological sections is connected after the pathological image encoding large model. After training, a hyperplasia network HyperplasiaNet for segmenting hyperplasia lesions is finally formed;

[0015] C: The hydatidiform mole sections were stained with he stain and scanned images of the hydatidiform mole sections were obtained;

[0016] D: Slice the hydatidiform mole slice scan image to obtain a number of hydatidiform mole slice scan image slices;

[0017] E: Cut the several hydatidiform mole slice scan images in step D into the villi network VilliNet to obtain the villus region segmentation map, then simultaneously input the villus region segmentation map and the hydatidiform mole slice scan image cut into the hyperplasia network HyperplasiaNet to obtain the corresponding hyperplasia segmentation map, then fuse the cut hyperplasia segmentation maps corresponding to all hydatidiform mole slice scan image cuts to obtain a preliminary complete slice hyperplasia segmentation map;

[0018] F: Perform image optimization on the preliminary complete slice hyperplasia segmentation map, and finally obtain an optimized slice hyperplasia segmentation map reflecting the actual lesion morphology.

[0019] In the step A, the pre-trained pathological image coding model is based on the DinoV2 image coding model. The DinoV2 image coding model is deeply trained through segmentation tasks using a preset pathological slice data set to obtain the pre-trained pathological image coding model.

[0020] In the step A, the pathological image encoding large model is used to extract features from the input pathological slice image, and output a first feature map to the first segmentation head MaskFormer; after the first segmentation head MaskFormer processes the first feature map, it outputs two types of probability maps, background and villi, in which each pixel point in the probability map corresponds to two probability values, respectively representing the probability of belonging to the background and the probability of belonging to the villi area; finally, the villi area segmentation map is obtained by extracting the probability of each pixel point belonging to the villi area.

[0021] In the step B, the villous area segmentation map is first post-processed for normalization, and then the villous area segmentation map is merged with the normalized pathological section to obtain an input matrix; the convolutional neural network fuses the preliminary features of the pathological section and the villous area segmentation map in the input matrix and outputs a second feature map; the pathological image coding large model further extracts features from the input second feature map and outputs a third feature map; the second segmentation head MaskFormer processes the third feature map and outputs two types of probability maps of background and hyperplastic lesions, in which each pixel point in the probability map corresponds to two probability values, respectively representing the probability of belonging to the background and the probability of belonging to the hyperplastic lesion; finally, the probability of each pixel point belonging to the villous area is extracted, and combined with the set probability threshold θ, a hyperplasia segmentation map containing a hyperplastic lesion area is finally obtained.

[0022] The convolutional neural network has three layers. The first convolution layer uses 32 3×3 convolution kernels to extract features from four input channels, and applies the ReLU activation function to introduce nonlinearity. The second convolution layer uses 64 3×3 convolution kernels to further extract features, and also uses the ReLU activation function. The third convolution layer uses 3 1×1 convolution kernels to reduce the number of channels from 64 to 3 to match the output channel requirements, and uses the Sigmoid activation function to ensure that the output value is between 0 and 1.

[0023] The first and second segmentation heads MaskFormer both include several layers of Transformer encoders and decoders. The first segmentation head MaskFormer is used to further process the feature map output by the pathological image encoding large model in the villi network VilliNet; the second segmentation head MaskFormer is used to further process the feature map output by the pathological image encoding large model in the hyperplasia network HyperplasiaNet; the multiple attention heads in each layer of the Transformer encoder are used to capture different feature dimensions of the image.

[0024] In the step F, the noise in the slice hyperplasia segmentation map is first removed, and then the boundary of the lesion segmentation area is optimized, and then the small area in the segmentation map whose area is less than the set threshold due to errors in the segmentation process is removed, and finally the adjacent segmentation areas are connected to form a continuous hyperplasia lesion area; finally, the optimized slice hyperplasia segmentation map reflecting the actual lesion morphology is obtained.

[0025] The training method of the hyperplasia network HyperplasiaNet is:

[0026] The first step is to use the preset pathological slice dataset to perform deep training on the DinoV2 image coding model through segmentation tasks to obtain the trained pathological image coding model.

[0027] The second step is to cut the images in the labeled dataset into blocks according to the set size to construct the training dataset required for the HyperplasiaNet training;

[0028] The third step is to freeze the parameters of the pre-trained pathological image encoding model so that the parameters remain unchanged in the initial training, and only train the convolutional neural network and the second segmentation head MaskFormer through the training data set to optimize and update the parameters of the convolutional neural network and the second segmentation head MaskFormer;

[0029] The fourth step is to unfreeze all layers of the pre-trained DinoV2 image coding model after the convolutional neural network and the second segmentation head MaskFormer have been initially trained and verified to be effective, so that the parameters of the entire hyperplasia network HyperplasiaNet are updated together, thereby integrating the learning results of the pre-trained pathological image coding model with the newly trained convolutional neural network and the second segmentation head MaskFormer, and optimizing the parameters of the entire hyperplasia network HyperplasiaNet through global fine-tuning;

[0030] The fifth step is to evaluate the performance of the hyperplasia network HyperplasiaNet, and further optimize and adjust the hyperplasia network HyperplasiaNet based on the evaluation results.

[0031] The device for segmenting lesions of hydatidiform mole based on a large image model includes:

[0032] A slice image extraction module is used to amplify the structure of the hydatidiform mole slice and obtain a hydatidiform mole slice scan image containing the hydatidiform mole slice structure;

[0033] The slice hyperplasia segmentation map generation module is used to slice the hydatidiform mole slice scan image according to the set requirements, and input the obtained several hydatidiform mole slice scan image blocks into the villi network VilliNet to obtain the villus area segmentation map; then the villus area segmentation map and the hydatidiform mole slice scan image blocks are simultaneously input into the hyperplasia network HyperplasiaNet to obtain the corresponding sliced ​​hyperplasia segmentation map; then all the sliced ​​hyperplasia segmentation maps are merged to obtain a preliminary and complete sliced ​​hyperplasia segmentation map;

[0034] The slice hyperplasia segmentation map post-processing module is used to perform image optimization processing on the preliminary complete slice hyperplasia segmentation map, and finally obtain an optimized slice hyperplasia segmentation map reflecting the actual lesion morphology.

[0035] The villi network VilliNet includes a pre-trained pathological image encoding large model and a first segmentation head MaskFormer; the pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the first segmentation head MaskFormer; the first segmentation head MaskFormer is used to perform segmentation prediction on the pathological slice image and distinguish the background and villi area in the pathological slice;

[0036] The hyperplasia network HyperplasiaNet includes a convolutional neural network, a pathological image encoding large model and a second segmentation head MaskFormer; the convolutional neural network is used to realize the preliminary feature fusion of pathological sections and villus area segmentation maps, the pathological image encoding large model is used to extract features from the input pathological section images and output feature maps to the second segmentation head MaskFormer; the second segmentation head MaskFormer is used to perform segmentation prediction on the pathological section images and distinguish the background and hyperplastic lesions in the pathological sections.

[0037] In the prior art, clinical pathologists mainly make a comprehensive judgment on whether it is a hydatidiform mole based on pathological features such as villous stromal edema, diffuse hyperplasia of trophoblastic cells at the edge of villous edema, and information such as the patient's menopausal duration and pregnancy history. In the present invention, based on the pathological feature of villous stromal edema as the key diagnostic basis, through the combination of image coding large model technology and hydatidiform mole slice pathological recognition technology, the hydatidiform mole slice scan image is cut into pieces and first sent to the villus network VilliNet, and then the hydatidiform mole slice scan image and the post-processed villus segmentation result are simultaneously sent to the hyperplasia network HyperplasiaNet, and after integration and optimization, the slice hyperplasia segmentation map of the hydatidiform mole slice scan image is obtained. The slice hyperplasia segmentation map can directly provide data support for clinicians, so that doctors can intuitively obtain the distribution of hyperplasia, thereby assisting clinicians to screen cases more efficiently.

[0038] In addition, the pathological image encoding model used in the present invention has been deeply trained on the pathological slice data set, and has a strong pathological slice feature extraction capability; at the same time, the second segmentation head MaskFormer has been separately and deeply trained, and the network structure has been optimized through a specific training method, which effectively improves the accuracy of the slice hyperplasia segmentation map. Finally, through the joint training of the hyperplasia network HyperplasiaNet, it can more accurately identify and segment hyperplastic lesions in pathological slices, improving the overall segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic flow chart of the method for segmenting hydatidiform mole hyperplasia lesions based on a large image model in the present invention;

[0040] Figure 2It is a scanned image of a hydatidiform mole section in the present invention;

[0041] Figure 3 It is the actual labeled slice hyperplasia label map in the present invention;

[0042] Figure 4 It is the network prediction slice proliferation segmentation map in the present invention;

[0043] Figure 5 The present invention is a flowchart for obtaining a training data set for annotating a hydatidiform mole slice scan image disclosed in the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0045] like Figures 1 to 5 As shown, the method for segmenting hydatidiform mole lesions based on a large image model of the present invention comprises the following steps:

[0046] A: Construct a neural network model for villus segmentation. After the pre-trained pathological image encoding model, the first segmentation head MaskFormer is connected to distinguish the background and villus area in the pathological section. After training, the villus network VilliNet for villus segmentation is finally formed.

[0047] In the present invention, the pre-trained pathological image coding model is based on the DinoV2 image coding model, and the DinoV2 image coding model is deeply trained by segmentation tasks using a preset pathological slice data set to obtain a pre-trained pathological image coding model;

[0048] In this embodiment, the first segmentation head MaskFormer is connected after the pre-trained pathological image encoding large model to perform segmentation prediction on the pathological section image to distinguish the background and the villi area (Villi) in the pathological section; since the segmentation task is to distinguish the background and the villi area, the output category of the first segmentation head MaskFormer is set to 2.

[0049] In this embodiment, the pathological slice data set contains multiple pathological types, including cancer, inflammation and benign lesions; among them, cancer includes breast cancer, lung cancer and gastric cancer, etc., and the segmentation content is mainly the cancerous area and its boundaries; inflammation includes hepatitis and gastritis, etc., and the segmentation content is mainly the inflammatory reaction area and its boundaries; benign lesions include benign tumors and cysts, etc., and the segmentation content is mainly the lesion area and its boundaries.

[0050] In this embodiment, the size of the pathological slice image input to the pathological image coding large model is 512×512 pixels; the pathological image coding large model extracts features from the input pathological slice image, outputs a first feature map with a dimension of 16×16×768, and then outputs the first feature map to the first segmentation head MaskFormer;

[0051] In the present invention, the first segmentation head MaskFormer contains several layers of Transformer encoders and decoders, which are used to further process the first feature map output by the pathological image encoding model. The Transformer encoder has 6 layers, each layer contains 8 attention heads, and the feature embedding dimension is 768 dimensions; each self-attention head can capture different feature dimensions of the image and enhance the segmentation ability of the model.

[0052] After processing the first feature map, the first segmentation head MaskFormer outputs two probability maps: background and villi. The dimension of the probability map is 512×512×2, and each pixel corresponds to two probability values, namely the probability of belonging to the background and the probability of belonging to the villi area.

[0053] In the present invention, in the probability map output by the first segmentation head MaskFormer, the probability that a pixel is background is denoted as P background , the probability that the pixel is in the fluff area is recorded as P villi , these two probabilities satisfy P background +P villi =1, for the subsequent segmentation of hyperplastic lesions, the probability of each pixel belonging to the villous area is extracted to obtain a 512×512×1 villous area segmentation map;

[0054] B: Constructing a hyperplasia segmentation neural network model, firstly, a convolutional neural network for realizing the preliminary feature fusion of pathological sections and villus area segmentation maps is set in front of the pre-trained pathological image encoding large model, and the second segmentation head MaskFormer for distinguishing the background and hyperplasia lesions in pathological sections is connected after the pathological image encoding large model. After training, the hyperplasia network HyperplasiaNet for segmenting hyperplasia lesions is finally formed;

[0055] In the present invention, the output layer of the pre-trained pathological image encoding model is connected to the second segmentation head MaskFormer, which can perform segmentation prediction on the pathological section image to distinguish the background and hyperplasia lesions (hyperplasia) in the pathological section; since the segmentation task is to distinguish the background and hyperplasia lesions, the output category of the second segmentation head MaskFormer is set to 2.

[0056] In the present invention, the villus area segmentation map is first post-processed for normalization, and each value of the villus area segmentation map is first subtracted by 0.5 and then multiplied by 4 to ensure that the data has a zero mean and a unit standard deviation. The villus area segmentation map is then merged with the normalized pathological section to obtain an input matrix of 512×512×4; the input matrix is ​​first passed through the set convolutional neural network to output a second feature map of 512×512×3 to be sent to the pathological section encoding large model.

[0057] Among them, the number of convolutional neural network layers is 3. The first convolution layer uses 32 3×3 convolution kernels to extract features from 4 input channels, and applies the ReLU activation function to introduce nonlinearity. Then, the second convolution layer uses 64 3×3 convolution kernels to further extract more complex features, and also uses the ReLU activation function. Finally, the third convolution layer uses 3 1×1 convolution kernels to reduce the number of channels of the second feature map from 64 to 3 to match the output channel requirements, and uses the Sigmoid activation function to ensure that the output value is between 0 and 1.

[0058] After that, the second feature map enters the pathological image coding large model to further extract the input second feature map, output a third feature map with a dimension of 16×16×768, and then output the third feature map to the second segmentation head MaskFormer;

[0059] In the present invention, the second segmentation head MaskFormer contains several layers of Transformer encoders and decoders, which are used to further process the third feature map output by the pathological image encoding model. The Transformer encoder has 6 layers, each layer contains 8 attention heads, and the feature embedding dimension is 768 dimensions; each self-attention head can capture different feature dimensions of the image and enhance the segmentation ability of the model.

[0060] After processing the third feature map, the second segmentation head MaskFormer outputs two probability maps of background and hyperplasia. The dimension of the probability map is 512×512×2, and each pixel corresponds to two probability values, namely the probability of belonging to the background and the probability of belonging to the hyperplasia.

[0061] In the present invention, in the probability map output by the second segmentation head MaskFormer, the probability that a pixel is background is denoted as P background , the probability that a pixel is a hyperplastic lesion is recorded as P hyperplasi , these two probabilities satisfy P background +P hyperplasia =1.

[0062] In order to determine whether there is a lesion, the present invention also defines a probability threshold θ. When the output hyperplasia lesion probability Phyperplasia When it is greater than or equal to the probability threshold θ, the area is considered to be a hyperplastic lesion area, that is:

[0063]

[0064] C: The hydatidiform mole sections were stained with he stain and scanned images of the hydatidiform mole sections were obtained;

[0065] In step C, the hydatidiform mole slice is firstly stained with HE, and then the hydatidiform mole slice stained with HE is placed on the stage of a digital microscope, and a digital microscope is selected; then the digital microscope is automatically focused using an autofocus module so that a clear hydatidiform mole slice can be seen in the field of view of the microscope; finally, a slice scanning module is used to obtain a scanned image of the hydatidiform mole slice under a microscope.

[0066] In step C, an existing fully automatic digital slice scanner may also be directly used to finally obtain a hydatidiform mole slice scan image.

[0067] D: Slice the hydatidiform mole slice scan image to obtain a number of hydatidiform mole slice scan image slices;

[0068] Since the scanned image of the hydatidiform mole obtained in step C has a high resolution and a pixel size of 65536×65536, and the pixel size of the input image of the hyperplasia network HyperplasiaNet is required to be 512×512, the slice needs to be cut into blocks. In step D, the high-resolution scanned image of the hydatidiform mole is cut into blocks to obtain multiple scanned image blocks with a pixel size of 512×512. For the scanned image of the hydatidiform mole with a pixel size of 65536×65536, 128×128 scanned image blocks with a pixel size of 512×512 are finally obtained;

[0069] E: Cut the several hydatidiform mole slice scan images in step D into the villi network VilliNet to obtain the villus region segmentation map, then simultaneously input the villus region segmentation map and the hydatidiform mole slice scan image cut into the hyperplasia network HyperplasiaNet to obtain the corresponding hyperplasia segmentation map, then fuse the cut hyperplasia segmentation maps corresponding to all hydatidiform mole slice scan image cuts to obtain a preliminary complete slice hyperplasia segmentation map;

[0070] F: Perform image optimization on the preliminary complete slice hyperplasia segmentation map, and finally obtain an optimized slice hyperplasia segmentation map reflecting the actual lesion morphology;

[0071] In the present invention, the noise in the segmentation map can be firstly removed by morphological operation or filtering technology to ensure the clarity and accuracy of the segmentation result; then, the boundary of the lesion segmentation area is optimized by edge detection and smoothing algorithm to make the segmentation result more accurate and coherent; then, the too small area in the segmentation map whose area is less than the set threshold due to the error in the segmentation process is removed; finally, the adjacent segmentation areas are connected to form a continuous hyperplastic lesion area to ensure the integrity of the hyperplastic lesion; finally, the optimized slice hyperplasia segmentation map reflecting the actual lesion morphology is obtained, which can effectively improve the accuracy and reliability of the segmentation result.

[0072] In this embodiment, morphological operations can be erosion and dilation, or opening and closing operations, etc.; morphological operations can effectively remove isolated small noise points and fill small holes; filtering technology can use Gaussian filtering or median filtering, etc., which can smooth the noise in the image and retain the main structural features.

[0073] Edge detection can use the Canny edge detection algorithm to determine the edge position by detecting the gradient change in the image. The smoothing algorithm can use bilateral filtering or other edge-preserving smoothing algorithms to smooth the boundaries while retaining the edge clarity.

[0074] When removing areas with too small an area in the segmentation map, by calculating the area of ​​each segmented area and removing areas below a set threshold (such as 10 pixels), the above-mentioned small noise areas that have no clinical significance can be removed, which can effectively reduce errors and improve the accuracy of the segmentation results.

[0075] When connecting adjacent segmented regions, the connected component labeling algorithm can be used to connect adjacent pixels belonging to the same category to form a complete region. By merging these regions, it is ensured that the segmentation results can accurately reflect the morphology of the actual lesion.

[0076] Through the above optimization processing steps, not only the accuracy and reliability of the segmentation results are effectively improved, but also the clinical usability of the segmentation results is ensured, which can provide doctors with clearer and more accurate diagnostic basis.

[0077] In the present invention, the hyperplasia network HyperplasiaNet needs to be trained on a hydatidiform mole slice data set. This embodiment also provides a method for training the hyperplasia network HyperplasiaNet.

[0078] To train the network, you need training images to feed into the network and train it.

[0079] In this example, the morphological characteristics of hydatidiform mole required for the actual diagnosis of hydatidiform mole are combined, and the annotation results of 1000 typical hydatidiform mole slice scan images are obtained through hydatidiform mole slice annotation and annotation review. Each slice needs to be annotated with detailed hyperplastic lesions. Specific annotation samples can be found in Figure 1 .

[0080] At present, the bottleneck of medical imaging is the extreme lack of high-quality annotated data sets, and there is no good annotated data set for hydatidiform mole worldwide. The recognition rate of deep networks is based on a good data set, so first of all, standardized, reasonable, and strict annotation is required to obtain a good data set. In the present invention, the training data set is obtained by the following method:

[0081] Step 101: Develop a reliable annotation scheme

[0082] After several proposed annotation schemes were reviewed and confirmed by multiple professional clinicians, a single annotation scheme was finally determined, namely, hyperplasia annotation. Specific annotation samples can be found in Figure 1 The hyperplasia annotation is to circle the trophoblastic cell area with diffuse hyperplasia of the villus margin.

[0083] Step 102: Training of labelers

[0084] Several annotators with relevant medical knowledge were selected for training.

[0085] Step 103: Initial labeling by labelers

[0086] According to the labeling regulations, each annotator is responsible for about 80 scan slices and annotates the proliferation annotations used for network training.

[0087] Step 104: Pathologist Review

[0088] The preliminary annotations obtained in step 103 are strictly reviewed by clinicians with long-term clinical experience and fed back to the annotators.

[0089] Step 105: Detailed annotation by annotators

[0090] The annotation review results are analyzed and modified, and the annotators unify the annotation standards to obtain the final annotation data set.

[0091] Step 106: Automatic network annotation

[0092] Based on the annotated data set obtained in step 105, the hyperplasia network HyperplasiaNet is trained; and the trained hyperplasia network HyperplasiaNet is used to perform image semantic segmentation on the newly added slice scan images to obtain a new annotated data set, that is, the network automatically annotates the new slice scan images to expand the data set, and then train a network with better robustness.

[0093] In the present invention, the training method of the hyperplasia network HyperplasiaNet is:

[0094] The first step is to use the preset pathological slice dataset to perform deep training on the DinoV2 image coding model through segmentation tasks to obtain the trained pathological image coding model.

[0095] The second step is to cut the images in the labeled dataset into blocks of size 512×512 to construct the training dataset required for training the hyperplasia network HyperplasiaNet.

[0096] The third step is to freeze the parameters of the pre-trained pathological image coding model for the hyperplasia segmentation neural network model composed of the convolutional neural network, the pathological image coding model and the second segmentation head MaskFormer so that it remains unchanged in the initial training, and only train the convolutional neural network and the second segmentation head MaskFormer through the training data set, and optimize and update the parameters of the convolutional neural network and the second segmentation head MaskFormer;

[0097] The above steps can avoid the destruction of the feature extraction capability of the pathology image coding large model by the initial training, and can also make full use of the excellent performance of the pre-trained pathology image coding large model in pathology slice feature extraction, so that the segmentation head can effectively learn the segmentation task.

[0098] When training the convolutional neural network and the second segmentation head MaskFormer, the same or similar loss function and optimization algorithm as the pathological image encoding model are selected, and the parameters of the task head are optimized through the back propagation algorithm, so that the convolutional neural network and the second segmentation head MaskFormer can adapt to the data distribution of hydatidiform mole pathological slice images, and the convolutional neural network and the second segmentation head MaskFormer are ensured to converge stably during the training process by setting training parameters such as learning rate, training cycle, batch size, etc. The training loss and validation set performance of the model are monitored in real time to ensure the stability and effectiveness of the training process.

[0099] The fourth step is to unfreeze all layers of the pre-trained DinoV2 image coding model after the convolutional neural network and the second segmentation head MaskFormer have been initially trained and verified to be effective, so that the parameters of the entire network can be updated together, thereby integrating the learning results of the pre-trained layers with the newly trained convolutional neural network and the second segmentation head MaskFormer, and optimizing the parameters of the entire hyperplasia network HyperplasiaNet through global fine-tuning.

[0100] By jointly training the hyperplasia network HyperplasiaNet, the feature extraction and segmentation capabilities of the model can be further optimized, and the accuracy and reliability of the segmentation results can be improved.

[0101] The initial learning rate of the HyperplasiaNet is set to a low level (usually 0.0001) to fine-tune the model weights without losing or destroying important features learned in the pre-training phase. This phase uses the cross entropy loss function and the Adam optimizer, whose adaptive learning rate feature is particularly suitable for such scenarios that require fine-tuning.

[0102] During the fine-tuning process of the accretive network HyperplasiaNet, the accretive network HyperplasiaNet is continuously optimized and adjusted to ensure the efficiency of the training process and the performance of the model. The performance indicators of the model, including loss and accuracy, are monitored in real time, and the learning rate and regularization parameters are adjusted according to these indicators. In addition, different optimization algorithms and data augmentation strategies are experimented to enhance the generalization ability of the model to unseen data. Continuous optimization aims to respond to challenges found during training, such as overfitting or inappropriate learning rates, and gradually find the optimal parameter configuration and network settings through fine-tuning.

[0103] The fifth step is to evaluate the performance of the hyperplasia network HyperplasiaNet, and further optimize and adjust the hyperplasia network HyperplasiaNet based on the evaluation results.

[0104] Evaluate the performance of the trained model on the validation set to ensure that it performs as expected on the encoding of hydatidiform mole pathology slice images. Choose appropriate evaluation metrics, such as the intersection over union (IoU) for the segmentation task. Then further optimize and adjust the model based on the evaluation results. Adjust the parameters or training parameters of the convolutional neural network and the second segmentation head MaskFormer to optimize the model performance. If necessary, perform more fine-tuning training cycles to ensure that the model's performance on the validation set continues to improve.

[0105] The device for segmenting lesions of hydatidiform mole based on a large image model of the present invention comprises:

[0106] A slice image extraction module is used to amplify the microstructure of the hydatidiform mole slice and obtain a scanned image of the hydatidiform mole slice containing the microstructure of the hydatidiform mole slice;

[0107] In this embodiment, the slice image extraction module can use an existing digital microscope in conjunction with a slice scanning module, or can directly use an existing fully automatic digital slice scanner; ultimately, a hydatidiform mole slice scan image containing the hydatidiform mole slice microstructure is obtained through the slice image extraction module.

[0108] The slice hyperplasia segmentation map generation module is used to slice the hydatidiform mole slice scan image according to the set requirements, and input the obtained several hydatidiform mole slice scan image blocks into the villi network VilliNet to obtain the villus area segmentation map; then the villus area segmentation map and the hydatidiform mole slice scan image blocks are simultaneously input into the hyperplasia network HyperplasiaNet to obtain the corresponding sliced ​​hyperplasia segmentation map; then all the sliced ​​hyperplasia segmentation maps are merged to obtain a preliminary and complete sliced ​​hyperplasia segmentation map;

[0109] The villi network VilliNet includes a pre-trained pathological image encoding model and a first segmentation head MaskFormer; the pathological image encoding model is used to extract features from the input pathological slice image and output feature maps to the first segmentation head MaskFormer; the first segmentation head MaskFormer is used to segment and predict the pathological slice image and distinguish the background and villi area in the pathological slice (Villi);

[0110] The hyperplasia network HyperplasiaNet includes a convolutional neural network, a pre-trained pathological image encoding model and a second segmentation head MaskFormer; the convolutional neural network is used to realize the preliminary feature fusion of the pathological section and the villus area segmentation map, the pathological image encoding model is used to extract the features of the input pathological section image and output the feature map to the second segmentation head MaskFormer; the second segmentation head MaskFormer is used to perform segmentation prediction on the pathological section image and distinguish the background and hyperplastic lesions in the pathological section;

[0111] The slice hyperplasia segmentation map post-processing module is used to perform image optimization processing on the preliminary complete slice hyperplasia segmentation map, and finally obtain an optimized slice hyperplasia segmentation map reflecting the actual lesion morphology.

[0112] The post-processing module of the slice hyperplasia segmentation map first removes the noise in the segmentation map through morphological operations or filtering technology to ensure the clarity and accuracy of the segmentation result; then, the boundary of the lesion segmentation area is optimized through edge detection and smoothing algorithms to make the segmentation result more accurate and coherent; then, the area in the segmentation map that is too small due to errors in the segmentation process is removed; finally, the adjacent segmented areas are connected to form a continuous hyperplasia lesion area to ensure the integrity of the hyperplasia lesion; finally, the optimized slice hyperplasia segmentation map reflecting the actual lesion morphology is obtained.

[0113] The specific processing process and processing steps of the above-mentioned hydatidiform mole hyperplasia lesion segmentation device based on the large image model have been described in detail in the above-mentioned hydatidiform mole hyperplasia lesion segmentation method based on the large image model, and will not be repeated here.

Claims

1. A method for segmenting hydatidiform mole lesions based on a large image model, characterized in that: The following steps are involved: A: Construct a neural network model for villus segmentation; by connecting the first segmentation head MaskFormer used to distinguish the background and villus area in the pathological section after the pre-trained pathological image encoding model, the villus network VilliNet for villus segmentation is finally formed after training; B: Constructing a hyperplasia segmentation neural network model; first, a convolutional neural network for realizing the preliminary feature fusion of pathological sections and villus area segmentation maps is set in front of the pre-trained pathological image encoding large model, and the second segmentation head MaskFormer for distinguishing the background and hyperplasia lesions in pathological sections is connected after the pathological image encoding large model. After training, a hyperplasia network HyperplasiaNet for segmenting hyperplasia lesions is finally formed; C: The hydatidiform mole sections were stained with he stain and scanned images of the hydatidiform mole sections were obtained; D: Slice the hydatidiform mole slice scan image to obtain a number of hydatidiform mole slice scan image slices; E: Cut the several hydatidiform mole slice scan images in step D into the villi network VilliNet to obtain the villus region segmentation map, then simultaneously input the villus region segmentation map and the hydatidiform mole slice scan image cut into the hyperplasia network HyperplasiaNet to obtain the corresponding hyperplasia segmentation map, then fuse the cut hyperplasia segmentation maps corresponding to all hydatidiform mole slice scan image cuts to obtain a preliminary complete slice hyperplasia segmentation map; F: Perform image optimization on the preliminary complete slice hyperplasia segmentation map, and finally obtain an optimized slice hyperplasia segmentation map reflecting the actual lesion morphology.

2. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 1, characterized in that: In the step A, the pre-trained pathological image coding model is based on the DinoV2 image coding model. The DinoV2 image coding model is deeply trained through segmentation tasks using a preset pathological slice data set to obtain the pre-trained pathological image coding model.

3. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 1, characterized in that: In the step A, the pathological image encoding large model is used to extract features from the input pathological slice image, and output a first feature map to the first segmentation head MaskFormer; after the first segmentation head MaskFormer processes the first feature map, it outputs two types of probability maps, background and villi, in which each pixel point in the probability map corresponds to two probability values, respectively representing the probability of belonging to the background and the probability of belonging to the villi area; finally, the villi area segmentation map is obtained by extracting the probability of each pixel point belonging to the villi area.

4. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 1, characterized in that: In the step B, the villous area segmentation map is first post-processed for normalization, and then the villous area segmentation map is merged with the normalized pathological section to obtain an input matrix; the convolutional neural network fuses the preliminary features of the pathological section and the villous area segmentation map in the input matrix and outputs a second feature map; the pathological image coding large model further extracts features from the input second feature map and outputs a third feature map; the second segmentation head MaskFormer processes the third feature map and outputs two types of probability maps of background and hyperplastic lesions, in which each pixel point in the probability map corresponds to two probability values, respectively representing the probability of belonging to the background and the probability of belonging to the hyperplastic lesion; finally, the probability of each pixel point belonging to the villous area is extracted, and combined with the set probability threshold θ, a hyperplasia segmentation map containing a hyperplastic lesion area is finally obtained.

5. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 4, characterized in that: The convolutional neural network has three layers. The first convolutional layer uses 32 3×3 convolution kernels to extract features from four input channels and applies a ReLU activation function to introduce nonlinearity. The second convolution layer uses 64 3×3 convolution kernels to further extract features, and also uses the ReLU activation function; the third convolution layer uses 3 1×1 convolution kernels to reduce the number of channels from 64 to 3 to match the output channel requirements, and uses the Sigmoid activation function to ensure that the output value is between 0 and 1.

6. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 1, characterized in that: The first and second segmentation heads MaskFormer both include several layers of Transformer encoders and decoders. The first segmentation head MaskFormer is used to further process the feature map output by the pathological image encoding large model in the villi network VilliNet; the second segmentation head MaskFormer is used to further process the feature map output by the pathological image encoding large model in the hyperplasia network HyperplasiaNet; the multiple attention heads in each layer of the Transformer encoder are used to capture different feature dimensions of the image.

7. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 1, characterized in that: In the step F, the noise in the slice hyperplasia segmentation map is first removed, and then the boundary of the lesion segmentation area is optimized, and then the small area in the slice hyperplasia segmentation map whose area is less than the set threshold due to errors in the segmentation process is removed, and finally the adjacent segmentation areas are connected to form a continuous hyperplasia lesion area; finally, the optimized slice hyperplasia segmentation map reflecting the actual lesion morphology is obtained.

8. The method for segmenting hydatidiform mole lesions based on a large image model according to claim 1, characterized in that: The training method of the hyperplasia network HyperplasiaNet is: The first step is to use the preset pathological slice dataset to perform deep training on the DinoV2 image coding model through segmentation tasks to obtain the trained pathological image coding model. The second step is to cut the images in the labeled dataset into blocks according to the set size to construct the training dataset required for the HyperplasiaNet training; The third step is to freeze the parameters of the pre-trained pathological image encoding model so that the parameters remain unchanged in the initial training, and only train the convolutional neural network and the second segmentation head MaskFormer through the training data set to optimize and update the parameters of the convolutional neural network and the second segmentation head MaskFormer; The fourth step is to unfreeze all layers of the pre-trained DinoV2 image coding model after the convolutional neural network and the second segmentation head MaskFormer have been initially trained and verified to be effective, so that the parameters of the entire hyperplasia network HyperplasiaNet are updated together, thereby integrating the learning results of the pre-trained pathological image coding model with the newly trained convolutional neural network and the second segmentation head MaskFormer, and optimizing the parameters of the entire hyperplasia network HyperplasiaNet through global fine-tuning; The fifth step is to evaluate the performance of the hyperplasia network HyperplasiaNet, and further optimize and adjust the hyperplasia network HyperplasiaNet based on the evaluation results.

9. A device for segmenting lesions of hydatidiform mole based on a large image model using the method of any one of claims 1 to 8, characterized in that: include: A slice image extraction module is used to amplify the structure of the hydatidiform mole slice and obtain a hydatidiform mole slice scan image containing the hydatidiform mole slice structure; The slice hyperplasia segmentation map generation module is used to slice the hydatidiform mole slice scan image according to the set requirements, and input the obtained several hydatidiform mole slice scan image blocks into the villi network VilliNet to obtain the villus area segmentation map; then the villus area segmentation map and the hydatidiform mole slice scan image blocks are simultaneously input into the hyperplasia network HyperplasiaNet to obtain the corresponding sliced ​​hyperplasia segmentation map; then all the sliced ​​hyperplasia segmentation maps are merged to obtain a preliminary and complete sliced ​​hyperplasia segmentation map; The slice hyperplasia segmentation map post-processing module is used to perform image optimization processing on the preliminary complete slice hyperplasia segmentation map, and finally obtain an optimized slice hyperplasia segmentation map reflecting the actual lesion morphology.

10. The device for segmenting hydatidiform mole lesions based on a large image model according to claim 9, characterized in that: The villi network VilliNet includes a pre-trained pathological image encoding large model and a first segmentation head MaskFormer; the pathological image encoding large model is used to extract features from the input pathological slice image and output the feature map to the first segmentation head MaskFormer; the first segmentation head MaskFormer is used to perform segmentation prediction on the pathological slice image and distinguish the background and villi area in the pathological slice; The hyperplasia network HyperplasiaNet includes a convolutional neural network, a pathological image encoding large model and a second segmentation head MaskFormer; the convolutional neural network is used to realize the preliminary feature fusion of pathological sections and villus area segmentation maps, the pathological image encoding large model is used to extract features from the input pathological section images and output feature maps to the second segmentation head MaskFormer; the second segmentation head MaskFormer is used to distinguish the background and hyperplasia lesions in the pathological sections through segmentation prediction.

Images