Grape tire slice image processing method and device based on image large model
Through the hydatidiform sectioning method based on the image large model, the distribution map of villus, edema and hyperplasia is automatically extracted, which solves the problem of traditional low diagnostic efficiency, and realizes efficient and accurate identification of hydatidiform lesions and calculation of abnormal villus proportion, supporting clinical auxiliary diagnosis.
Patent Information
- Application Number
- CN202510552463.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-07-29
- Filing Date
- 2025-04-29
- Publication Date
- 2025-08-01
AI Technical Summary
Traditional hydatidiform diagnostic methods rely on manual observation and empirical estimation, resulting in low diagnostic efficiency and susceptible to subjective factors. The promotion of digital pathological sections has also failed to effectively solve the problem of data islands, making it difficult to realize automated lesion feature recognition and analysis.
The distribution map extraction module is constructed by using the image-based big model, and the distribution maps of villus, edema and hyperplasia in the hydatidi molar section are automatically extracted, and the proportion of abnormal villus is calculated through the distribution map fusion and analysis module to assist in clinical diagnosis.
Automatic lesion feature recognition of hydatidiform sections is realized, reducing labeling costs, improving diagnostic efficiency and accuracy, enhancing the robustness of microscopic imaging differences, and supporting clinicians to efficient case screening.
Smart Images
Figure CN120411053A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image processing, and particularly to a method and device for processing images of hydatidiform mole sections based on an image large model. Background Art
[0002] Hydatidiform mole refers to a vesicular mole in the shape of a grape cluster formed by the edema of placental villous stroma and the hyperplasia of trophoblasts after pregnancy. If not treated in time, it may develop into life-threatening diseases such as choriocarcinoma. Traditional diagnostic methods mainly rely on pathologists to manually observe sections to identify and classify lesions. Especially for key indicators such as the proportion of abnormal (lesioned) villi in all villi, only the doctor's experience can be relied on for estimation. This method is not only time-consuming and laborious, but also easily affected by subjective factors, resulting in low diagnostic efficiency and poor real-time performance. In addition, the popularization of digital pathological sections is still in its initial stage, and it is easy to form data islands for hospital inspections and diagnostic information, which is not conducive to the further improvement of digital medical level and patient health management. Therefore, there is an urgent need for a system that can automatically process medical images and provide accurate auxiliary diagnoses.
[0003] The lesion characteristics observed in the clinical diagnosis of hydatidiform mole are mainly divided into two types: the edema of villous stroma in the section tissue and the diffuse hyperplasia of trophoblasts at the edge of the villi. As a lesion characteristic for the diagnosis of hydatidiform mole, the edema of villous stroma (hereinafter referred to as edema) mainly has a tissue morphology of pool-like and coastline-like shapes in the villous stroma, forming a morphology with sparse intermediate stromal cells and denser surrounding areas. The diffuse hyperplasia of trophoblasts at the edge of the villi (hereinafter referred to as hyperplasia) is another lesion characteristic for the diagnosis of hydatidiform mole. The main tissue characteristics of hyperplasia are the multifocal and non-polar hyperplasia of trophoblasts around the villi, and the trophoblasts show morphological characteristics such as flower-like and serrated shapes. In addition, the villous tissue where the lesion occurs is called abnormal villi, and abnormal villi are also an important target area for doctors to analyze the pathological sections of hydatidiform mole. Therefore, the automatic recognition and analysis of edema, hyperplasia, and abnormal villi can greatly reduce the workload of professional doctors and improve the work efficiency of doctors. Summary of the Invention
[0004] The purpose of the present invention is to provide a method and device for processing images of hydatidiform mole sections based on an image large model, which can obtain the distribution of normal and abnormal villi of hydatidiform mole based on the constructed distribution map extraction module, and calculate the proportion of abnormal villi in hydatidiform mole, so as to assist clinicians in more efficiently screening cases and solving the problem of low efficiency in the clinical diagnosis and detection of hydatidiform mole caused by manual observation and manual estimation in the prior art.
[0005] The present invention adopts the following technical solutions:
[0006] A method for processing images of hydatidiform mole sections based on an image large model, comprising the following steps:
[0007] A: Construct a distribution map extraction module including a villus distribution extraction sub-module, an edema distribution extraction sub-module, and a hyperplasia distribution extraction sub-module; each sub-module includes an image encoding large model and a corresponding task head segmentation network;
[0008] B: Use the distribution map extraction module to obtain the villus distribution map, edema distribution map, and hyperplasia distribution map of the hydatidiform mole section image;
[0009] C: Input the villus distribution map, edema distribution map, and hyperplasia distribution map into the distribution map fusion module to obtain the edema villus distribution map, hyperplasia villus distribution map, normal villus distribution map, and abnormal villus distribution map of the hydatidiform mole section;
[0010] D: Input the villus distribution map, edema villus distribution map, hyperplasia villus distribution map, abnormal villus distribution map, and normal villus distribution map into the distribution map analysis module to output the proportion of abnormal villi in the hydatidiform mole section.
[0011] Step A includes:
[0012] A1: Collect and organize a set of hydatidiform mole section images with villus distribution annotation, edema distribution annotation, and hyperplasia distribution annotation to construct an image training set and a test set;
[0013] A2: Construct an image encoding large model and perform pre-training;
[0014] A3: Corresponding add a task head segmentation network at the output end of the image encoding large model to construct a distribution map extraction module; the three task head segmentation networks are respectively used to distinguish the background and villus area, background and edema lesions, and background and hyperplasia lesions in the pathological section, so as to obtain the villus distribution map, edema distribution map, and hyperplasia distribution map;
[0015] A4: Input the hydatidiform mole section training set data into the image encoding large model, extract the feature vector of the hydatidiform mole section image, and input the feature vector into the task head segmentation network to generate a predicted distribution map through the task head segmentation network; use the distribution map extraction module to calculate the loss between the predicted distribution map and the true annotation distribution map, and update the parameters of the distribution map extraction module through the backpropagation algorithm;
[0016] A5: For the trained distribution map extraction module, use the hydatidiform mole section image test set for testing and calculate the segmentation accuracy of the distribution map extraction module.
[0017] Step C includes:
[0018] C1: Overlay the villus distribution map, edema distribution map, and hyperplasia distribution map of the same hydatidiform mole section image obtained in step B to form a comprehensive distribution map;
[0019] C2: Traverse each villus area, obtain the pixel area of each villus through pixel counting, and count the number of pixels with edema distribution and the number of pixels with hyperplasia distribution within the villus area;
[0020] C3: Calculate the proportion of edema pixels and the proportion of hyperplasia pixels within each villus area. If the proportion of edema pixels within a certain villus area exceeds the critical threshold of villus edema proportion, it is marked as an edematous villus; if the proportion of hyperplasia pixels exceeds the critical threshold of villus hyperplasia proportion, it is marked as a hyperplastic villus; if a villus is marked as an edematous villus or a hyperplastic villus, then this villus is marked as an abnormal villus, otherwise it is marked as a normal villus;
[0021] C4: Generate an edematous villus distribution map using the comprehensive distribution map and the edema mark, generate a hyperplastic villus distribution map using the comprehensive distribution map and the hyperplasia mark, generate an abnormal villus distribution map using the comprehensive distribution map and the abnormal mark, and generate a normal villus distribution map using the comprehensive distribution map and the normal mark.
[0022] Step D includes:
[0023] D1: Count the number of edematous villi with an area exceeding the villus area threshold in the edematous villus distribution map; count the number of villi with an area exceeding the villus area threshold in the villus distribution map; then calculate the proportion of edematous villi, that is, the number of edematous villi divided by the total number of villi;
[0024] D2: Count the number of hyperplastic villi with an area exceeding the villus area threshold in the hyperplastic villus distribution map; count the number of villi with an area exceeding the villus area threshold in the villus distribution map; then calculate the proportion of hyperplastic villi, that is, the number of hyperplastic villi divided by the total number of villi;
[0025] D3: Count the number of abnormal villi with an area exceeding the villus area threshold in the abnormal villus distribution map; count the number of villi with an area exceeding the villus area threshold in the villus distribution map; then calculate the proportion of abnormal villi, that is, the number of abnormal villi divided by the total number of villi;
[0026] D4: Output the edematous villus distribution map, the hyperplastic villus distribution map, the proportion of edematous villi, the proportion of hyperplastic villi, and the proportion of abnormal villi.
[0027] In step B, the hydatidiform mole section image is cut into image blocks to obtain the corresponding block label map; in step C, all the block label maps are stitched according to their corresponding positions in the original section, and the overlapping part of adjacent blocks is fitted according to the pixel average value of the overlapping parts of multiple block images.
[0028] The task head segmentation network includes: an input layer for receiving the feature map output by the image encoding large model; a convolutional layer for extracting local features to enhance the network's non-linear expression ability, with batch normalization and ReLU activation function connected after each convolutional layer; an upsampling layer for upsampling to restore the feature map to the original image size, with batch normalization and ReLU activation function connected after each upsampling layer; and an output layer for mapping the feature map to the required number of segmentation channels, with the final output using the Softmax activation function for multi-class segmentation or the Sigmoid activation function for binary classification segmentation.
[0029] When training the villus distribution map extraction module, after cutting the corresponding training pictures into several training picture chunks of a set size, the training picture chunks are input into the corresponding image encoding large model, and the output of the image encoding large model is used as the input of the corresponding task head segmentation network. The actual segmentation output corresponding to each training picture chunk is obtained by processing the corresponding villus annotation file, edema annotation file, and hyperplasia annotation file. The parameters of the image encoding large model are frozen, and only the parameters of the task head segmentation network are updated during training. Finally, the corresponding villus distribution map extraction sub-module, edema distribution map extraction sub-module, and hyperplasia distribution map extraction sub-module are trained.
[0030] In step A5, the segmentation accuracy is represented by the intersection over union (IoU), Dice coefficient, or distance between distribution centers.
[0031] The image encoding large model is the DINOv2 model based on the Vision Transformers architecture; Vision Transformers divides the image into fixed-size patches and treats each patch as an embedding vector for processing, modeling the relationships between these patches through the self-attention mechanism; DINOv2 uses the Vision Transformers architecture as the backbone network and adopts a self-supervised learning method to train Vision Transformers.
[0032] A hydatidiform mole section image processing device based on an image large model includes:
[0033] A section image extraction module for obtaining the hydatidiform mole pathological section image of the hydatidiform mole section sample under the microscope;
[0034] A distribution map extraction module for extracting the villus distribution map, edema distribution map, and hyperplasia distribution map of the hydatidiform mole section image;
[0035] The distribution map extraction module is constructed based on the image encoding large model and the task head segmentation network, including a villus distribution extraction sub-module, an edema distribution extraction sub-module, a hyperplasia distribution extraction sub-module, and the corresponding task head segmentation network at the back; the three task head segmentation networks are respectively used to distinguish the background and villus regions, the background and edema lesions, and the background and hyperplasia lesions in the pathological section, so as to obtain the villus distribution map, the edema distribution map, and the hyperplasia distribution map;
[0036] The distribution map fusion module is used to overlay the villus distribution map, the edema distribution map, and the hyperplasia distribution map to generate a comprehensive distribution map, and after analysis, obtain the edema-villus distribution map, the hyperplasia-villus distribution map, the abnormal villus distribution map, and the normal villus distribution map;
[0037] The distribution map analysis module is used to analyze the edema-villus distribution map, the hyperplasia-villus distribution map, the abnormal villus distribution map, and the normal villus distribution map, and finally obtain the proportion of abnormal villi in the hydatidiform mole section.
[0038] The present invention uses an image encoding large model after self-supervised pre-training for encoding. Through the general visual representation ability obtained by pre-training with a large amount of image data, the dependence on fine segmentation annotation is significantly reduced. When migrating to downstream tasks, by freezing the pre-trained model parameters and only fine-tuning the lightweight task head segmentation network, the rapid adaptation from general features to the specific pathological features of hydatidiform mole is realized, enabling each lesion sub-module (villus / edema / hyperplasia) to achieve high-precision segmentation with only a small number of labeled samples. At the same time, it can also be fine-tuned with a small amount of fine-tuning after unfreezing the pre-trained model parameters to achieve better adaptation. Compared with the traditional fully supervised segmentation model, this method greatly reduces the annotation cost, and at the same time improves the robustness to microscopic imaging differences through self-supervised pre-training, significantly enhancing the generalization performance. This technical route of "self-supervised pre-training + small sample fine-tuning" effectively solves the core contradiction between the scarcity of labeled data and the model generalization demand in medical image analysis.
[0039] The present invention decouples the lesion segmentation and statistical decision-making processes in stages, and innovatively encodes the diagnostic experience of pathologists (such as the threshold of the pixel ratio of edema / hyperplasia and the screening conditions of villus area) into the algorithm process explicitly, rather than relying on the neural network to implicitly learn the pathological rules. Compared with the traditional scheme that directly uses an end-to-end regression model to predict the proportion of villus types, this method effectively avoids the misjudgment risk caused by the training data deviation of the neural network model through the segmentation-labeling-statistics mechanism guided by prior knowledge, and shows stronger robustness in the test.
[0040] The present invention can utilize an image coding large model to obtain the distribution of villi, edema, and hyperplasia in hydatidiform mole, and further obtain the distribution of normal and abnormal villi in hydatidiform mole, and count the proportion of abnormal villi in hydatidiform mole, so as to assist clinicians in more efficiently screening cases and solving the problem of low efficiency in the clinical diagnosis and detection of hydatidiform mole caused by manually observing the pathological characteristics of sliced tissues and manually estimating the proportion of abnormal villi in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic flow chart of the method for hydatidiform mole lesion segmentation and integration in the present invention;
[0042] Figure 2 It is a schematic logical diagram of the hydatidiform mole lesion segmentation and integration in the present invention;
[0043] Figure 3 It is a schematic structural diagram of the hydatidiform mole lesion segmentation and integration device in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0044] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0045] As Figure 1 and 2 shown, the method for processing hydatidiform mole sliced images based on an image large model according to the present invention includes the following steps:
[0046] A: Construct a distribution map extraction module including a villus distribution extraction sub-module, an edema distribution extraction sub-module, and a hyperplasia distribution extraction sub-module; each sub-module includes an image coding large model and a corresponding task head segmentation network;
[0047] In the present invention, the task head segmentation network can be trained by using a training set of hydatidiform mole sliced images with distribution annotations.
[0048] In the present invention, step A includes the following specific steps:
[0049] A1: Collect and organize a set of hydatidiform mole sliced images with villus distribution annotations, edema distribution annotations, and hyperplasia distribution annotations, and construct an image training set and a test set; the image training set and the test set include a villus training set and a test set, an edema training set and a test set, and a hyperplasia training set and a test set.
[0050] The hydatidiform mole slice image set can be obtained through a digital microscope. That is, the stained hydatidiform mole slice is placed on the microscope stage. After the microscope is focused, a camera is used to obtain the scanned image of the hydatidiform mole slice under the microscope field of view. By moving the hydatidiform mole slice, all the scanned images of the hydatidiform mole slice are finally stitched together to form a complete hydatidiform mole slice image. The hydatidiform mole slice image set can also be obtained through a pathological slice scanner. After calibrating and cleaning the scanner, the stained hydatidiform mole slice is fixed on the slide holder. The scanner is started and the control software is opened. The high-resolution scanning mode is selected and the image is previewed. After adjusting the scanning area, the scanning starts. After the scanning is completed, the hydatidiform mole slice image is saved;
[0051] The hydatidiform mole slice image set is obtained through manual annotation. Professional pathologists respectively outline the contours of all villi, edema, and hyperplasia on the hydatidiform mole slice images to generate villus distribution annotation, edema distribution annotation, and hyperplasia distribution annotation; the splitting ratio of the hydatidiform mole slice image training set to the test set is 4:1, that is, the images are randomly assigned to the training set and test set folders according to the ratio of 4:1.
[0052] In the hydatidiform mole slice image set with distribution annotation, there are villus distribution annotation, edema distribution annotation, and / or hyperplasia distribution annotation on any hydatidiform mole slice image, serving as the corresponding villus, edema, and / or hyperplasia training set or test set;
[0053] In step A1, preprocessing is performed on the hydatidiform mole slice image training set, including but not limited to image preprocessing operations such as image normalization, size adjustment, image cutting, and noise removal, to ensure that the image is suitable for input into the image encoding large model in the subsequent steps. Since the size of the obtained scanned image of the hydatidiform mole slice is large, and the existing image encoding large model has requirements for the input image size, it is necessary to perform preprocessing on the image input into the image encoding large model by cutting. For the training of different distribution extraction sub-modules in the subsequent steps, the preprocessing methods of the image training set can be different. The preprocessing operation of the villus set is denoted as a-p, the preprocessing operation of the edema set is denoted as b-p, and the preprocessing operation of the hyperplasia set is denoted as c-p.
[0054] The cutting of the hydatidiform mole slice image is to cut the slice into several pieces with a size of size at an interval of s. There may be partial area overlap between two adjacent pieces in the up, down, left, and right directions. Generally, after the preprocessing cutting operation a-p of the villus set, the image size size1 is a pixel size of 3000×3000 to 18000×18000. After the preprocessing cutting operation b-p of the edema set, the image size size2 is a pixel size of 1500×1500 to 9000×9000. After the preprocessing cutting operation c-p of the hyperplasia set, the image size size3 is a pixel size of 1500×1500 to 9000×9000.
[0055] A2: Build an image encoding large model and perform pre-training.
[0056] In the present invention, the image encoding large model can adopt the Vision Transformer image encoding large model; the villus distribution extraction sub-module uses the image encoding large model a-m, the edema distribution extraction sub-module uses the image encoding large model b-m, and the hyperplasia distribution extraction sub-module uses the image encoding large model c-m.
[0057] The villus distribution extraction sub-module uses the image encoding large model a-m, and the image encoding large model a-m is the DINOv2 model based on the Vision Transformers architecture; Vision Transformers divides the image into fixed-size patches and treats each patch as an embedding vector for processing, and models the relationships between these patches through the self-attention mechanism; DINOv2 adopts the Vision Transformers architecture as its backbone network and uses self-supervised learning methods to train Vision Transformers. Since DINOv2 is trained on a large-scale and diverse dataset, it can generalize well to various visual tasks.
[0058] The edema distribution extraction sub-module uses the image encoding large model b-m, and the image encoding large model b-m is also the DINOv2 model; the hyperplasia distribution extraction sub-module uses the image encoding large model c-m, where the image encoding large model c-m is also the DINOv2 model.
[0059] A3: Add a task head segmentation network at the output end of the image encoding large model to build a distribution map extraction module. The task head segmentation network includes multiple convolutional layers, upsampling layers, and activation functions for fine-segmenting the image; and design the loss function of the task head segmentation network. Commonly used loss functions include cross-entropy loss, Dice loss, IoU loss, etc. to measure the accuracy of network segmentation.
[0060] In the present invention, the task head segmentation network a-net is used to distinguish the background and villus regions in the pathological section to obtain the villus distribution map; the task head segmentation network b-net is used to distinguish the background and edema lesions in the pathological section to obtain the edema distribution map, and the task head segmentation network c-net is used to distinguish the background and hyperplasia lesions in the pathological section to obtain the hyperplasia distribution map.
[0061] In the present invention, the network architecture of the task head segmentation network a-net is as follows: an input layer that receives the feature map from the large image coding model; a convolutional layer that uses three 3x3 convolutional layers to extract local features and increase the non-linear expression ability of the network, and each convolutional layer is followed by batch normalization and a ReLU activation function; an upsampling layer that uses two transposed convolutional layers or bilinear interpolation for upsampling to restore the feature map to the original image size, and each upsampling layer is followed by batch normalization and a ReLU activation function; an output layer that maps the feature map to the required number of segmentation channels using a 1x1 convolution in the last convolutional layer, and the final output uses a Softmax activation function for multi-class segmentation or a Sigmoid activation function for binary classification segmentation.
[0062] The network architectures of the task head segmentation networks b-net and c-net are the same as that of the task head segmentation network a-net.
[0063] In this embodiment, the loss function of the task head segmentation networks a-net, b-net, and c-net can adopt one of the following two loss functions:
[0064]
[0065] where N is the number of samples, and y i and are the true label and the predicted value respectively.
[0066] In this embodiment, the task head segmentation networks (a-net / b-net / c-net) adopt a unified architecture design, and the multi-dimensional feature decoupling of medical images is realized through a hierarchical structure of three convolutional layers + two transposed convolutional layers: the convolutional layer adopts a combination of batch normalization and ReLU activation to effectively suppress the abnormal gradient fluctuations in small-sample training, enabling the model to converge stably; the transposed convolutional upsampling accurately reconstructs the villous serrated edge, overcoming the problem of detail blurring in the traditional interpolation method; the Softmax / Sigmoid output layer provides a threshold-adjustable segmentation probability map, which supports the dynamic control of the sensitivity of edema / hyperplasia determination. This design realizes the sharing of computing resources while ensuring the independence of the three tasks, taking into account the requirements of high-precision segmentation and clinical real-time performance.
[0067] A4: Input the training set data of hydatidiform mole slices into the corresponding large image encoding model to extract the corresponding feature vectors of the hydatidiform mole slice images, including villus feature vectors, edema feature vectors, and hyperplasia feature vectors. Then, input the corresponding feature vectors into the corresponding task head segmentation network to generate a predicted distribution map through the corresponding task head segmentation network. The distribution map extraction module calculates the loss between the predicted distribution map and the true annotation distribution map, and updates the parameters of the distribution map extraction module, including the large image encoding model parameters and the task head segmentation network parameters, through the backpropagation algorithm.
[0068] In the present invention, the large image encoding model a-m and the task head segmentation network a-net for obtaining the villus distribution map are trained using the training set of hydatidiform mole slice images with villus distribution annotations, the large image encoding model b-m and the task head segmentation network b-net for obtaining the edema distribution map are trained using the training set of hydatidiform mole slice images with edema distribution annotations, and the large image encoding model c-m and the task head segmentation network c-net for obtaining the hyperplasia distribution map are trained using the training set of hydatidiform mole slice images with hyperplasia distribution annotations.
[0069] In this embodiment, the parameters of the pre-trained large image encoding model can be set to the frozen mode, that is, its parameters are not updated when training the task head segmentation network; or it can be set to the thawed mode, that is, its parameters are updated when training the task head segmentation network to make the large image encoding model more adaptable to the hydatidiform mole slice image processing task.
[0070] This embodiment also provides a method for network training:
[0071] Before training the network, it is necessary to obtain the training pictures for inputting into the network and training.
[0072] Combined with the morphological characteristics of hydatidiform moles required for actual diagnosis, through nearly one year of hydatidiform mole slice annotation training and annotation review from hydatidiform mole slice annotation, the annotation results of 157 typical hydatidiform mole slice scanned images are obtained. Each slice needs to be annotated in three ways. The villus annotation is to circle the villus area of the slice, the edema annotation is to circle the edematous part in the villus area of the slice, and the hyperplasia annotation is to circle the diffuse hyperplasia area of trophoblast cells in the villus area.
[0073] The training method of the villus distribution map extraction module is:
[0074] Cut the training images into several training image chunks with size size1, input the training image chunks into the image encoding large model a-m, and use the output of the image encoding large model a-m as the input of the task head segmentation network a-net. The actual segmentation output corresponding to each training image chunk is obtained by processing the villus annotation file. Freeze the parameters of the image encoding large model a-m, and only update the parameters of the task head segmentation network a-net during training. Finally, train to obtain the villus distribution map extraction sub-module;
[0075] The training method of the edema distribution map extraction module is as follows:
[0076] Cut the training images into several training image chunks with size size2, input the training image chunks into the image encoding large model b-m, and use the output of the image encoding large model b-m as the input of the task head segmentation network b-net. The actual segmentation output corresponding to each training image chunk is obtained by processing the edema annotation file. Freeze the parameters of the image encoding large model b-m, and only update the parameters of the task head segmentation network b-net during training. Finally, train to obtain the edema distribution map extraction sub-module;
[0077] The training method of the hyperplasia distribution map extraction module is as follows:
[0078] Cut the training images into several training image chunks with size size3, input the training image chunks into the image encoding large model c-m, and use the output of the image encoding large model c-m as the input of the task head segmentation network c-net. The actual segmentation output corresponding to each training image chunk is obtained by processing the hyperplasia annotation file. Freeze the parameters of the image encoding large model c-m, and only update the parameters of the task head segmentation network c-net during training. Finally, train to obtain the hyperplasia distribution map extraction sub-module;
[0079] A5: For the trained distribution map extraction module, use the hydatidiform mole slice image test set for detailed evaluation, calculate the segmentation accuracy of the complete distribution map extraction module, and save the trained distribution map extraction module as a deployable model file. The segmentation accuracy is represented by the intersection over union (IoU), Dice coefficient, or distribution center distance. The distribution map extraction module includes the villus distribution map extraction sub-module, the edema distribution map extraction sub-module, and the hyperplasia distribution map extraction sub-module.
[0080] In the present invention, the IoU is used to measure the overlap degree between the predicted distribution area and the true distribution area, and the calculation formula is as follows:
[0081]
[0082] Among them, |A∩B| is the intersection area of the predicted distribution area A and the true distribution area B, and |A∪B| is the union area of the predicted distribution area A and the true distribution area B;
[0083] The Dice coefficient is used to measure the similarity between two sets and is particularly suitable for scenarios such as medical image segmentation. The calculation formula is as follows:
[0084]
[0085] Among them, |A∩B| is the intersection area of the predicted distribution region A and the true distribution region B, |A| is the area of the predicted distribution region A, and |B| is the union area of the true distribution region B;
[0086] The distribution center distance is used to measure the Euclidean distance between the centroid of the predicted segmentation region and the centroid of the true segmentation region. The calculation formula is as follows:
[0087] Let the centroid coordinates of the predicted distribution region be (x p , y p ), and the centroid coordinates of the true segmentation region be (x t , y t ). Then the calculation formula for the distribution center distance D is:
[0088]
[0089] Among them, (x p , y p ) are the centroid coordinates of the predicted segmentation region, and the calculation method is where P i represents the value of pixel i in the predicted distribution region A; (x t , y t ) are the centroid coordinates of the predicted segmentation region, and the calculation method is where T j represents the value of pixel j in the predicted distribution region B.
[0090] B: Use the distribution map extraction module constructed in step A to obtain the villus distribution map, edema distribution map, and hyperplasia distribution map of the hydatidiform mole section image;
[0091] In this embodiment, the hydatidiform mole section image can be obtained by a pathological section scanner or by stitching microscope fields of view.
[0092] In step B, the hydatidiform mole section image can be preprocessed according to the preprocessing method in step A, including but not limited to image normalization, size adjustment, image chunking, noise removal, and other image preprocessing operations to ensure that the image is suitable for input into the distribution map extraction module in the subsequent steps.
[0093] The preprocessing operation should be consistent with the preprocessing operation in step A, that is, perform preprocessing operations a-p on the hydatidiform mole slice image, input it into the villus distribution extraction sub-module to obtain the villus distribution map, perform preprocessing operations b-p on the hydatidiform mole slice image, input it into the edema distribution extraction sub-module to obtain the edema distribution map, perform preprocessing operations c-p on the hydatidiform mole slice image, and input it into the hyperplasia distribution extraction sub-module to obtain the hyperplasia distribution map;
[0094] Among them, the villus distribution extraction sub-module includes the image encoding large model a-m and the task head segmentation network a-net, the edema distribution extraction sub-module includes the image encoding large model b-m and the task head segmentation network b-net, and the hyperplasia distribution extraction sub-module includes the image encoding large model c-m and the task head segmentation network c-net.
[0095] C: Input the villus distribution map, edema distribution map, and hyperplasia distribution map obtained in step B into the distribution map fusion module, and output the edema villus distribution map, hyperplasia villus distribution map, normal villus distribution map, and abnormal villus distribution map of the hydatidiform mole slice;
[0096] In the present invention, step C includes the following specific steps:
[0097] C1: Superimpose the villus distribution map, edema distribution map, and hyperplasia distribution map of the same hydatidiform mole slice image obtained in step B to form a comprehensive distribution map.
[0098] Preferably, when superimposing the villus distribution map, edema distribution map, and hyperplasia distribution map, if preprocessing operations, especially size adjustment operations, are performed in step B, a reduction transformation is required to ensure that the distribution maps during superimposition are aligned with the original hydatidiform mole slice image.
[0099] Since the preprocessing operation in step B includes block processing, the preprocessing operation in step C1 is block fusion. Specifically, splice all the block label maps according to their corresponding positions in the slice during the original block cutting, and fit the overlapping parts of adjacent blocks up, down, left, and right according to the pixel mean value of the overlapping parts of multiple block images.
[0100] C2: The villi appear as an independent island-like area on the comprehensive distribution map. Traverse and analyze each villus on the comprehensive distribution map. That is, use the connected domain analysis algorithm in the image processing algorithm to traverse each villus area, count all the pixels in each villus area, calculate the pixel area of each villus, and simultaneously count the number of edema distribution pixels and hyperplasia distribution pixels in the villus area; for the villus area R i (i = 1,..., N), its villus pixel area is Sa i , the number of edema distribution pixels is Sb i , the number of hyperplasia distribution pixels is Sc i ;
[0101] C3: Calculate the proportion β of edematous pixels in each villous area i = Sb i / Sa i and the proportion γ of hyperplastic pixels i = Sc i / Sa i ;
[0102] Set the critical threshold β for villous edema proportion d , the critical threshold γ for villous hyperplasia proportion d . If the proportion β of edematous pixels in a certain villous area i exceeds the set threshold β d , it is marked as an edematous villus; if the proportion γ of hyperplastic pixels i exceeds the set threshold γ d , it is marked as a hyperplastic villus; if the villus is marked as either an edematous villus or a hyperplastic villus, the villus is marked as an abnormal villus, otherwise it is marked as a normal villus;
[0103] C4: Generate a distribution map of edematous villi using the comprehensive distribution map and the edema marker, generate a distribution map of hyperplastic villi using the comprehensive distribution map and the hyperplasia marker, generate a distribution map of abnormal villi using the comprehensive distribution map and the abnormal marker, and generate a distribution map of normal villi using the comprehensive distribution map and the normal marker;
[0104] In this embodiment, through pixel - level fusion and object - level statistical analysis of the multi - modal distribution map, the lesion localization accuracy and clinical interpretability are significantly improved: the mean fusion splicing technology of the cut - block label map is adopted to eliminate the image deformation error caused by pre - processing, ensuring the strict alignment of the villus, edema, and hyperplasia distribution maps in the original coordinate system; independent traversal and pixel statistics are performed on each villous area through connected - component analysis, accurately simulating the diagnostic mode of a pathologist observing villi one by one under the microscope; the finally generated abnormal / normal villus distribution map supports spatial superposition visualization with the original pathological section, and doctors can quickly locate the targeted area marked by the algorithm, realizing full - transparency verification of the algorithm decision - making process.
[0105] D: Input the villus distribution map in step B and the distribution maps of edematous villi, hyperplastic villi, abnormal villi, and normal villi obtained in step C into the distribution map analysis module, and output the proportion of abnormal villi in the hydatidiform mole section.
[0106] The step D includes the following specific steps:
[0107] D1: Set the villus area threshold S d1 , in the distribution map of edematous villi, use the connected - component analysis algorithm to count all areas that exceed the villus area threshold S d1The number l1 of edematous villi; in the villus distribution map, use the connected component analysis algorithm to count all villi with an area exceeding the villus area threshold S d1 The number l2 of villi; calculate the proportion r1 of edematous villi, that is, the number of edematous villi divided by the total number of villi r1 = l1 / l2.
[0108] D2: Set the villus area threshold S d2 , in the hyperplastic villus distribution map, use the connected component analysis algorithm to count all hyperplastic villi with an area exceeding the villus area threshold S d2 The number m1 of hyperplastic villi; in the villus distribution map, use the connected component analysis algorithm to count all villi with an area exceeding the villus area threshold S d2 The number m2 of villi; calculate the proportion r2 of hyperplastic villi, that is, the number of hyperplastic villi divided by the total number of villi r2 = m1 / m2.
[0109] D3: Set the villus area threshold S d3 , in the abnormal villus distribution map, use the connected component analysis algorithm to count all abnormal villi with an area exceeding the villus area threshold S d3 The number n1 of abnormal villi; in the villus distribution map, use the connected component analysis algorithm to count all villi with an area exceeding the villus area threshold S d3 The number n2 of villi; calculate the proportion r of abnormal villi, that is, the number of abnormal villi divided by the total number of villi r = n1 / n2.
[0110] D4: Output the edematous villus distribution map and the hyperplastic villus distribution map, and output the proportion r1 of edematous villi, the proportion r2 of hyperplastic villi, and the proportion r of abnormal villi. The edematous villus distribution map and hyperplastic villus distribution map of the hydatidiform mole, as well as the proportion r1 of edematous villi, the proportion r2 of hyperplastic villi, and the proportion r of abnormal villi, can provide a quantitative basis for pathologists and assist doctors in diagnosis.
[0111] This embodiment provides precise quantitative indicators for the pathological diagnosis of hydatidiform mole through a hierarchical area screening and multi-dimensional lesion statistical mechanism: set the villus area threshold to filter out small interference regions, use connected component analysis to count the number of independent regions that meet the area criteria in edematous, hyperplastic, and abnormal villi, calculate the proportion of edematous / hyperplastic / abnormal villi, and finally output a quantitative report accompanied by a distribution map. This design improves the statistical signal-to-noise ratio through area filtering and combines multi-lesion dimension data for synchronous output, enabling pathologists to quickly cross-verify abnormal distribution trends.
[0112] As Figure 3 shown, the hydatidiform mole section image processing device based on an image large model according to the present invention includes:
[0113] A section image extraction module for obtaining the pathological section image of the hydatidiform mole under a microscope for a hydatidiform mole section sample;
[0114] In this embodiment, the sliced image extraction module can adopt an existing digital microscope or a pathological slice scanner.
[0115] The distribution map extraction module is used to extract the villus distribution map, the edema distribution map, and the hyperplasia distribution map of the hydatidiform mole sliced image;
[0116] The distribution map extraction module can be constructed based on an image coding large model and a task head segmentation network. The distribution map extraction module includes a villus distribution extraction sub-module, an edema distribution extraction sub-module, and a hyperplasia distribution extraction sub-module; each sub-module includes an image coding large model and a corresponding task head segmentation network; the three task head segmentation networks are respectively used to distinguish the background and the villus area, the background and the edema lesions, and the background and the hyperplasia lesions in the pathological slice, so as to obtain the villus distribution map, the edema distribution map, and the hyperplasia distribution map;
[0117] The specific structure of the distribution map extraction module has been described in detail above and will not be elaborated here;
[0118] The distribution map fusion module is used to superimpose the villus distribution map, the edema distribution map, and the hyperplasia distribution map to generate a comprehensive distribution map, and after analysis, obtain an edematous villus distribution map, a hyperplastic villus distribution map, an abnormal villus distribution map, and a normal villus distribution map;
[0119] The distribution map analysis module is used to analyze the edematous villus distribution map, the hyperplastic villus distribution map, the abnormal villus distribution map, and the normal villus distribution map, and finally obtain the proportion of abnormal villi in the hydatidiform mole slice.
[0120] The above-mentioned each module has been described in detail above and will not be elaborated here.
[0121] Those skilled in the art can also understand that the various illustrative logical blocks and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of the two. Whether such a function is implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art can use various methods to implement the described function for each specific application, but such implementation should not be construed as exceeding the scope protected by the embodiments of the present application.
[0122] In addition, as used herein, the word "exemplary" is used to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the word exemplary is intended to present concepts in a concrete fashion. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified, or clear from the context, "X applies A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then "X applies A or B" is satisfied under any of the foregoing instances. Additionally, unless otherwise specified or clear from the context that it is referring to the singular form, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more".
[0123] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. Specifically with respect to the various functions performed by the components (e.g., elements, resources, etc.) described above, unless otherwise indicated, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if not structurally equivalent to the disclosed structure. Additionally, although a particular feature of the present disclosure may have been disclosed with respect to only one of several implementations, such a feature may, as may be desired and advantageous for any given or particular application, be combined with one or more other features of other implementations. Further, with respect to the use of "comprises", "comprising", "has", "having", "includes", or variants thereof in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "including".
[0124] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known in the art or conventional techniques not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the following claims.
[0125] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method for hydatidiform mole slices based on an image large model, characterized in that: It includes the following steps: A: Construct a distribution map extraction module including a villus distribution extraction sub-module, an edema distribution extraction sub-module, and a hyperplasia distribution extraction sub-module; each sub-module includes an image encoding large model and a corresponding task head segmentation network; B: Use the distribution map extraction module to obtain the villus distribution map, edema distribution map, and hyperplasia distribution map of the hydatidiform mole section image; C: Input the villus distribution map, edema distribution map, and hyperplasia distribution map into the distribution map fusion module to obtain the edema villus distribution map, hyperplasia villus distribution map, normal villus distribution map, and abnormal villus distribution map of the hydatidiform mole section; D: Input the villus distribution map, as well as the edema villus distribution map, hyperplasia villus distribution map, abnormal villus distribution map, and normal villus distribution map into the distribution map analysis module, and output the proportion of abnormal villi in the hydatidiform mole section.
2. The method for processing images of hydatidiform mole slices according to claim 1, wherein Step A includes: A1: Collect and organize a set of hydatidiform mole section images with villus distribution annotation, edema distribution annotation, and hyperplasia distribution annotation, and construct an image training set and a test set; A2: Construct an image encoding large model and perform pre-training; A3: Corresponding add a task head segmentation network at the output end of each image encoding large model to construct a distribution map extraction module; the three task head segmentation networks are respectively used to distinguish the background and villus areas, background and edema lesions, and background and hyperplastic lesions in the pathological section, so as to obtain the villus distribution map, edema distribution map, and hyperplasia distribution map; A4: Input the hydatidiform mole section training set data into the image encoding large model, extract the feature vector of the hydatidiform mole section image, and input the feature vector into the task head segmentation network to generate a predicted distribution map through the task head segmentation network; use the distribution map extraction module to calculate the loss between the predicted distribution map and the true annotation distribution map, and update the parameters of the distribution map extraction module through the backpropagation algorithm; A5: For the trained distribution map extraction module, use the hydatidiform mole section image test set for testing, and calculate the segmentation accuracy of the distribution map extraction module.
3. The method for processing the hydatidiform mole slice image according to claim 1, wherein Step C includes: C1: Overlay the villus distribution map, edema distribution map, and hyperplasia distribution map of the same hydatidiform mole section image obtained in step B to form a comprehensive distribution map; C2: Traverse each villus area and obtain the pixel area of each villus through pixel counting, and count the number of edema distribution pixels and hyperplasia distribution pixels in the villus area; C3: Calculate the proportion of edema pixels and hyperplasia pixels in each villus area. If the proportion of edema pixels in a certain villus area exceeds the critical threshold of villus edema proportion, it is marked as an edema villus; if the proportion of hyperplasia pixels exceeds the critical threshold of villus hyperplasia proportion, it is marked as a hyperplasia villus; if a villus is marked as an edema villus or a hyperplasia villus, the villus is marked as an abnormal villus, otherwise it is marked as a normal villus; C4: Use the comprehensive distribution map and edema marks to generate an edema villus distribution map, use the comprehensive distribution map and hyperplasia marks to generate a hyperplasia villus distribution map, use the comprehensive distribution map and abnormal marks to generate an abnormal villus distribution map, and use the comprehensive distribution map and normal marks to generate a normal villus distribution map.
4. The method for processing hydatidiform mole slice images according to claim 1, wherein Step D includes: D1: Count the number of edematous villi with an area exceeding the villus area threshold in the edematous villi distribution map; count the number of villi with an area exceeding the villus area threshold in the villi distribution map; then calculate the proportion of edematous villi, that is, the number of edematous villi divided by the total number of villi. D2: Count the number of hyperplastic villi with an area exceeding the villus area threshold in the hyperplastic villi distribution map; count the number of villi with an area exceeding the villus area threshold in the villi distribution map; then calculate the proportion of hyperplastic villi, that is, the number of hyperplastic villi divided by the total number of villi. D3: Count the number of abnormal villi with an area exceeding the villus area threshold in the abnormal villi distribution map; count the number of villi with an area exceeding the villus area threshold in the villi distribution map; then calculate the proportion of abnormal villi, that is, the number of abnormal villi divided by the total number of villi. D4: Output the edematous villi distribution map, hyperplastic villi distribution map, proportion of edematous villi, proportion of hyperplastic villi, and proportion of abnormal villi.
5. The method for processing images of hydatidiform mole slices according to claim 1, wherein: In step B, the hydatidiform mole section image is cut into image blocks to obtain corresponding block label maps; in step C, all the block label maps are stitched together at the corresponding positions in the section according to the original cutting, and the overlapping parts of adjacent blocks are fitted according to the pixel mean of the overlapping parts of multiple block images.
6. The method for processing the image of hydatidiform mole section according to claim 1, wherein: The task head segmentation network includes: an input layer for receiving the feature map output by the image encoding large model; a convolutional layer for extracting local features to increase the non-linear expression ability of the network, and each convolutional layer is followed by batch normalization and a ReLU activation function; an upsampling layer for upsampling to restore the feature map to the original image size, and each upsampling layer is followed by batch normalization and a ReLU activation function; an output layer for mapping the feature map to the required number of segmentation channels, and the final output uses a Softmax activation function for multi-class segmentation or a Sigmoid activation function for binary classification segmentation.
7. The method for processing images of hydatidiform mole slices according to claim 1, characterized in that: When training the villi distribution map extraction module, after cutting the corresponding training pictures into several training picture blocks with a set size, the training picture blocks are input into the corresponding image encoding large model, and the output of the image encoding large model is used as the input of the corresponding task head segmentation network. The actual segmentation output corresponding to each training picture block is obtained by processing the corresponding villi annotation file, edema annotation file, and hyperplasia annotation file. The parameters of the image encoding large model are frozen, and only the parameters of the task head segmentation network are updated during training. Finally, the corresponding villi distribution map extraction sub-module, edema distribution map extraction sub-module, and hyperplasia distribution map extraction sub-module are trained.
8. The method for processing images of hydatidiform mole slices according to claim 2, characterized in that: In step A5, the segmentation accuracy is represented by the intersection over union (IoU), Dice coefficient, or distance to the distribution center.
9. The method for processing hydatidiform mole slice images according to claim 1, wherein: The image encoding large model is based on the DINOv2 model on top of the Vision Transformers architecture; Vision Transformers divides an image into tiles of a fixed size, treats each tile as an embedding vector for processing, and models the relationships between these tiles through the self-attention mechanism; DINOv2 uses the Vision Transformers architecture as the backbone network and adopts a self-supervised learning method to train Vision Transformers.
10. A hydatidiform mole slice image processing apparatus applying the method according to any one of claims 1 to 9, characterized in that, It includes: A sliced image extraction module for obtaining a pathological section image of a hydatidiform mole. A distribution map extraction module for extracting the villus distribution map, edema distribution map, and hyperplasia distribution map of the hydatidiform mole sliced image. The distribution map extraction module includes a villus distribution extraction sub-module, an edema distribution extraction sub-module, and a hyperplasia distribution extraction sub-module; each sub-module includes an image encoding large model and a corresponding task head segmentation network; the three task head segmentation networks are respectively used to distinguish the background and villus regions, background and edema lesions, and background and hyperplastic lesions in the pathological section to obtain the villus distribution map, edema distribution map, and hyperplasia distribution map. A distribution map fusion module for superimposing the villus distribution map, edema distribution map, and hyperplasia distribution map to generate a comprehensive distribution map, and obtaining an edematous villus distribution map, hyperplastic villus distribution map, abnormal villus distribution map, and normal villus distribution map after analysis. A distribution map analysis module for analyzing the edematous villus distribution map, hyperplastic villus distribution map, abnormal villus distribution map, and normal villus distribution map, and finally obtaining the proportion of abnormal villi in the hydatidiform mole sliced section.