A training method and device based on an image labeling and segmentation model

Abstract

The application provides a training method and device based on an image labeling and segmentation model, comprising the following steps: obtaining a sample image, and labeling a corresponding mask image by comparing an immunohistochemical staining registration image of a continuous section; taking the sample image and the mask image as a group of training samples, adaptively performing color transformation on the sample image according to a template of different staining spaces, and training a segmentation model based on the group of training samples. The technical scheme provided by the application improves the problem that a target of interest of a pathological image is difficult to accurately label, improves labeling efficiency and effectiveness, improves the generalization of an algorithm to images generated by different staining conditions and scanning instruments, uses an active learning mode and a transfer learning technology, and makes the model achieve good segmentation performance on small data set training.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a training method and apparatus based on image annotation and segmentation models. Background Technology

[0002] Malignant tumors severely restrict human health. Pathological diagnosis is the gold standard for determining tumor type, prognostic analysis, and guiding tumor treatment. Quantitative pathology is an important method for tumor diagnosis, and accurate segmentation of lesions from pathological images is a crucial research area and data foundation in quantitative pathology. For example, tumor microvessels are an important basis for determining the biological characteristics of malignant tumors. Interpreting the density, morphology, structure, and composition of tumor microvessels can reflect the permeability and sensitivity of chemotherapy drugs in tumor tissue. Quantitative tumor microvessel classification characteristics can assist in accurate tumor diagnosis, prognosis assessment, and chemotherapy regimen selection. However, the traditional "microscope + human eye recognition" pathological diagnosis model has the following problems in quantitative pathological diagnosis: 1. Due to the limited field of view of the microscope, it is difficult to accurately determine the overall characteristics of tumor lesions in a local field of view, which may lead to selective bias in quantitative calculations; 2. Pathologists are influenced by diagnostic experience, which may result in individual differences and human interpretation bias in quantitative calculations; 3. Pathologists have heavy clinical workloads, and accurate diagnosis of tumor microvessels is time-consuming and laborious, making it difficult to widely implement. Therefore, there is an urgent need to develop standardized and regulated computer-aided microvessel segmentation strategies based on the whole-slice field of view.

[0003] Fully Convolutional Networks (FCNs) in deep learning, employing encoder-decoder structures and skip connections, have significantly outperformed traditional image segmentation methods in semantic segmentation and are increasingly being applied in pathological image analysis. However, pathological images suffer from low standardization during staining processes and are greatly influenced by digital scanners, resulting in significant color space variations. This leads to low generalization ability in trained neural network models, hindering their widespread application. Compared to segmentation targets such as glandular and lymph node metastases, which are currently the most studied, there are no publicly available datasets related to microvessels. Furthermore, accurate data annotation for these small targets requires specialized pathologists, which is time-consuming and labor-intensive, making it difficult to obtain large datasets and posing a significant challenge to training neural network models. Moreover, the scale of microvessels and other small targets is highly variable. Compared to common segmentation targets like cell nuclei, microvessels exhibit high morphological diversity, are tightly integrated with other tissues, and have indistinct boundaries, making accurate identification very difficult. Therefore, it is necessary to construct accurate, fast, and highly generalizable network models and training methods to improve the accuracy and feasibility of tumor microvascular diagnosis. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a training method and apparatus based on image annotation and segmentation models, which can improve the accuracy of tumor microvascular image annotation and segmentation efficiency.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A training method based on an image annotation and segmentation model includes the following steps:

[0007] Step 1: Acquire panoramic images of continuous sections of the sample pathological tissue, including H&E staining images and CD34 and α-SMA immunohistochemical staining images; select typical images from the sample dataset, and reselect the remaining data after model training feedback; classify microvessels based on prior medical knowledge; obtain accurate reference standards for vessel presence and classification results by registering the corresponding CD34 and α-SMA immunohistochemical staining images with the H&E images to be labeled; annotate the H&E images against the registered immunohistochemical images to obtain a multi-classification vessel annotation mask for the sample H&E images;

[0008] Step 2: Preprocess the training samples obtained from the labeled images in Step 1; train the image segmentation convolutional neural network model to complete the microvascular classification and segmentation task; evaluate the performance of the currently trained model; select images with poor model performance based on the performance evaluation results, and reselect sample data that need to be labeled; repeat the training based on the new round of training samples until the predetermined performance index is reached.

[0009] Furthermore, the registration algorithm in step 1 includes manual registration, automatic registration based on image intensity, or semi-automatic registration based on landmarks, applying rigid or non-rigid deformation.

[0010] Furthermore, training the image segmentation convolutional neural network model specifically includes:

[0011] Step (1): Collect sample images and corresponding vascular mask images as a set of samples;

[0012] Step (2): Extract the tissue block image from the pyramid structure of the panoramic image at a set magnification factor, or scale the tissue block image at a set magnification factor; randomly allocate multiple sets of samples to the training set or validation set according to a certain ratio, and train the original segmentation model based on multiple sets of training samples.

[0013] Step (3): Construct the network model;

[0014] Step (4) Perform random sampling, that is, the network model performs random and consistent cropping of the sample image and the corresponding labeled mask image according to a fixed input size. Images with edges that are not large enough to the input size are filled with reflections to increase the diversity of the dataset.

[0015] Step (5): Perform data augmentation;

[0016] Step (6): Perform hard sample mining to improve class imbalance in multi-class image segmentation.

[0017] Furthermore, the performance of the currently trained model is evaluated; based on the performance evaluation results, images with poor model performance are selected, and new sample data requiring labeling are chosen; training is repeated using a new round of training samples until the predetermined performance indicators are achieved, specifically including the following steps:

[0018] Step (1): Divide the validation set samples into blocks according to the sliding window algorithm;

[0019] Step (2): Input the slice image into the trained network model to obtain the blood vessel and background segmentation probability matrices for the five subtypes of the corresponding slice;

[0020] Step (3) Post-processing of segmentation results includes: performing image reconstruction and weighted averaging of overlapping sub-segmentation probability matrices using a sliding window image fusion algorithm; combining multiple sub-segmentation matrices and determining the segmentation result of each pixel in the image to be segmented; and post-processing the combined segmentation results using connected component algorithm, majority voting algorithm, segmentation region morphology processing algorithm, and noise processing algorithm to obtain the final segmentation result.

[0021] Step (4): Evaluate the performance of the trained model;

[0022] Step (5): Apply an active learning strategy to iterate the training model until the performance parameters reach the predetermined target.

[0023] Further, step (1) includes: using a global thresholding algorithm to extract the region where the tissue block is located, removing blank areas on the image with an area greater than a set minimum area value, extracting continuous tissue blocks according to the connected component algorithm, and if there are multiple connected components, segmenting the region according to connectivity to obtain independent tissue block images; the extracted tissue block images have irregular shapes, calculating the minimum bounding box of each tissue block shape, and using each bounding box as the range of the tissue block to be segmented; recording the upper left corner (x, y) coordinates of each tissue block image to be segmented so that the tissue block image and the blank area can be reconstructed into the size of the region to be segmented.

[0024] Furthermore, in step (3), the network model is a convolutional neural network, a deep neural network, or a recurrent neural network.

[0025] Further, step (5) includes: using sample images collected from different hospitals in the public dataset as multiple color templates, decomposing each template into three channels: hematoxylin, eosin, and residual in the RGB space, calculating the weight values ​​of the hematoxylin and eosin channels, setting them as template values, and creating a template value set; decomposing the channels of the sample image into three channels: hematoxylin, eosin, and residual, and randomly adjusting the weights of the hematoxylin and eosin channels of the sample image to a value in the template value set; recombining the adjusted hematoxylin and eosin channels and the unadjusted residual channel of the image to synthesize a new sample image, and using the vascular mask image as a set of training samples.

[0026] Furthermore, step (6) includes: combining a weighted focus loss function and a weighted Dice loss function to increase the proportion of hard-to-distinguish samples in the loss function.

[0027] This invention also provides a training device for implementing a training method based on image annotation and segmentation models, comprising: a preprocessing module for extracting and cropping tissue from multiple pathological panoramic images and annotation masks respectively, obtaining multiple small images of the same size and corresponding mask images, constituting a sample set; a training module for inputting the sample set into an initial neural network model, and calculating the output probability vector of each pixel on each small image in the sample set by using the Softmax function; a loss function calculation module for calculating the gradient magnitude and gradient density based on the output probability matrix of each small image in the preset sample set, and calculating the loss function based on the gradient magnitude and gradient density; a postprocessing module for fusing and reconstructing the probability map predicted by the model corresponding to the sample images cropped by the preprocessing module into a panoramic image, and performing operations such as classification voting, contour smoothing, hole filling, and noise reduction; and a performance evaluation module for evaluating the performance of the trained neural network model, extracting samples with poor model prediction performance, and performing an iterative process of data annotation and algorithm training.

[0028] Furthermore, the initial neural network model is trained through transfer learning and serves as a neural network model for segmenting tissues of interest in a pathological panoramic image.

[0029] Beneficial effects:

[0030] The annotation method based on a clear gold standard provided by this invention can save resources on case studies and annotating physicians, effectively improving model performance. The iterative image segmentation model training method and apparatus provided by this invention can achieve good segmentation results even with relatively small datasets. This model can quickly acquire precise location information of tissues of interest (such as microvessels) across the entire field of view, avoiding interpretation differences and low repeatability issues that may arise from manual field selection. This provides objective and highly interpretable data for further quantitative analysis, improving the speed of quantitative analysis. This method can be applied to various tumor and tissues of interest segmentation tasks, providing a data foundation for fields such as quantitative pathology and biomarker development. Attached Figure Description

[0031] Figure 1 A schematic diagram illustrating a training method based on an image annotation and segmentation model provided in an embodiment of the present invention;

[0032] Figure 2a , Figure 2b , Figure 2c , Figure 2d This is a schematic diagram illustrating a method for annotation based on continuous slice image registration provided in an embodiment of the present invention; wherein, Figure 2a This is a schematic diagram of a sample image. Figure 2b To and Figure 2a A schematic diagram of the corresponding vascular mask image. Figure 2c To and Figure 2a A schematic diagram of the corresponding registered CD34 immunohistochemical staining images. Figure 2d To and Figure 2a A schematic diagram of the corresponding registered α-SMA immunohistochemical staining image;

[0033] Figure 3 This is a schematic diagram of the structure of a training device based on an image annotation and segmentation model provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0035] This invention provides a training method and apparatus based on image annotation and segmentation models. The following explanation uses pathological image annotation as an example to illustrate related technologies. Pathology is the microscopic study of tissue and cell morphology. Many tasks require the detection and segmentation of tissues and cells. Currently, the annotation of tissues and cells in pathological sections is mainly achieved manually, outlining the contours of the target tissue. This specifically includes: acquiring pathological images; selecting typical tissue regions; setting a gold standard for annotation, i.e., this invention proposes using continuous sections with immunohistochemical staining and H&E images for local registration as the gold standard; and performing fine annotation, typically using drawing tools or annotation tools to manually delineate the boundaries of the target tissue in the pathological image and set classifications to obtain pixel-level annotation results corresponding to the pathological image.

[0036] like Figure 1 As shown, this invention proposes an iterative image annotation method with a reference standard. This image annotation method can be applied to the field of medical image analysis, as well as other fields that require annotation of images with complex details. The following uses microvascular annotation in the field of medical image analysis as an example to elaborate on the implementation details of the technical solution of this invention:

[0037] Step 1: Obtain a whole-slide image sequence of continuous pathological tissue sections of the sample glioma, including H&E staining images and CD34 and α-SMA immunohistochemical staining images.

[0038] The panoramic image sequence of the sample was obtained by scanning stained tissue sections with a pathological digital scanner. The continuous tissue sections can provide a reference standard for H&E image annotation. CD34 immunohistochemical staining can show vascular endothelial cells in the tissue, and α-SMA immunohistochemical staining can show perivascular cells and smooth muscle cells in the tissue. The two markers fully demonstrate the main components, tissue extent and morphology of blood vessels.

[0039] Step 2: Select typical images from the sample dataset. The remaining data will be reselected based on the feedback from model training. Based on prior medical knowledge, such as the pathological diagnosis and classification of gliomas, collect and label typical pathological tissues diagnosed as grade IV glioblastoma, grade II and III astrocytic glioma, and grade II and III oligodendroglioma.

[0040] Step 3: Classify blood vessels using prior medical knowledge. This prior knowledge includes the morphology, size, and number of vascular endothelial cells and pericytes, the area and shape of blood vessels, and the degree of basement membrane thickening. Mature arteries and veins differ greatly in scale from microvessels, with area ratios exceeding 1000 times, posing a significant challenge to image segmentation models. Tumor microvessels exhibit extremely diverse morphologies, with multiple signaling pathways involved in regulating their growth. Current medical research shows that different morphologies and densities of microvessels in a patient's tumor tissue are correlated with prognosis and treatment responsiveness. Therefore, from an image perspective, separating microvessels from mature vessels, distinguishing different microvessel morphologies, reducing intra-class variations, and increasing inter-class variations can significantly improve the accuracy of image segmentation models.

[0041] Step 4: By registering the corresponding CD34 and α-SMA immunohistochemical staining images with the H&E images to be labeled, a reference standard for accurate vascular presence and classification results can be obtained. Microvessels are tightly integrated with tumor tissue and have indistinct boundaries. The registered immunohistochemical images can accurately display the endothelial and pericyte components and morphology of microvessels, thereby guiding precise vascular labeling.

[0042] The registration algorithm can be manual registration, automatic registration based on image intensity, or semi-automatic registration based on annotated landmarks, applying rigid or non-rigid deformation. This invention does not specifically limit the algorithm. For example, this invention uses a landmark-based rigid registration method to improve performance. A low-magnification (e.g., 5x magnification) H&E image and the corresponding immunohistochemical image are extracted. Using the H&E image as a template, the centers of microvessels within corresponding regions of interest (ROIs) are selected in both images as anchor landmarks. Each landmark is uniquely numbered. The registration algorithm is then applied to obtain the deformed immunohistochemical image.

[0043] Step 5: Annotate the immunohistochemical images after H&E image registration. The multi-class vascular mask images are manually annotated color images, with each color representing a category. These can be used as the expected output during model training. For example, Figure 2a , Figure 2b , Figure 2c , Figure 2d As shown, Figure 2a This is a schematic diagram of a sample image. Figure 2b Is with Figure 2a A schematic diagram of the corresponding vascular mask image. Figure 2c Is with Figure 2aA schematic diagram of the corresponding registered CD34 immunohistochemical staining images. Figure 2d Is with Figure 2a A schematic diagram of the corresponding registered α-SMA immunohistochemical staining image. Compared with annotations without reference standards, this method significantly improves annotation accuracy and reduces disputes regarding segmentation and typing. This method can be widely applied to segmentation tasks of various pathological tissues, enhancing the reliability of the gold standard for annotation.

[0044] After image annotation, the image segmentation model is trained. For example... Figure 1 As shown, this invention provides an iterative image segmentation model training method, comprising: preprocessing training samples obtained from the labeled images; training an image segmentation convolutional neural network model to complete the task of microvascular classification and segmentation; evaluating the performance of the currently trained model; selecting images with poor model performance based on the performance evaluation results, and re-selecting sample data to be labeled; repeating the training process based on a new round of training samples until a predetermined performance index is reached. Specifically, it includes the following steps:

[0045] Step 1: Train an image segmentation model based on the labeled dataset:

[0046] (1) Collect sample images and corresponding vascular mask images as a set of samples; use the global thresholding algorithm to extract the region where the tissue block is located, and remove the blank areas on the image with an area larger than the set minimum area value. Since the proportion of pathological tissue slices in the entire slide scan image is generally less than 50%, this operation can greatly reduce the amount of computation required for the panoramic image to be segmented. Extract continuous tissue blocks according to the connected component algorithm. If there are multiple connected components, the region is segmented according to connectivity to obtain independent tissue block images. The extracted tissue block images are irregular in shape. Calculate the minimum bounding box of each tissue block shape, and use each bounding box as the range of the tissue block to be segmented. Record the upper left corner (x, y) coordinates of each tissue block image to be segmented so that the tissue block image and the blank area can be reconstructed into the size of the region to be segmented.

[0047] (2) Since the pathological images to be segmented are usually 10,000 pixels * 10,000 pixels in size, which is far beyond the image size range that neural networks can process, the tissue block images are extracted from the pyramid structure of the panoramic image by a set magnification factor, or the tissue block images are scaled by a set magnification factor. Multiple sets of samples are randomly assigned to the training set or validation set according to a certain ratio, the common ratio being 7:3, and the original segmentation model is trained based on multiple sets of training samples.

[0048] (3) Network Model Construction: This invention adopts a fully convolutional neural network encoder-decoder structure to perform image segmentation tasks. The encoder can be transformed into various classic network architectures in the field of computer vision, such as ResNet, EfficientNet, and MobileNet series networks. Weights trained on large image datasets such as ImageNet and Coco are loaded for transfer learning, which can effectively improve the performance of training on small pathological datasets. The decoder structure can be transformed into classic fully convolutional neural networks such as UNet, PSPNet, and LinkNet. This invention does not limit the specific type of network model; the network model can be composed of any type of neural network. Optionally, the network model can be a convolutional neural network (CNN), a deep neural network (DNN), or a recurrent neural network (RNN), etc. The network model can include neural network layers such as input layers, convolutional layers, pooling layers, and connection layers, which this invention does not limit. Furthermore, this invention does not limit the number of each type of neural network layer.

[0049] (4) Random sampling: The sample image and the corresponding blood vessel mask image are not limited in length and width. During training, the network model randomly crops the image according to a fixed input size. Images with edges smaller than the input size are filled with reflections to increase the diversity of the dataset.

[0050] (5) Data Augmentation: This includes adaptive color transformation, precise object stitching enhancement, and common deformation and blurring enhancement methods. This invention does not limit the color transformation method. An example method is as follows: Using sample images collected from different hospitals in a public dataset as multiple color templates, each template is decomposed into three channels: hematoxylin, eosin, and residual in the RGB space. The weight values ​​of the hematoxylin and eosin channels are calculated and set as template values ​​to create a template value set. The channels of the sample image are decomposed into three channels: hematoxylin, eosin, and residual. The weights of the hematoxylin and eosin channels of the sample image are randomly adjusted to a value in the template value set. The adjusted hematoxylin and eosin channels and the unadjusted residual channel of the image are recombined to synthesize a new sample image, which, along with the blood vessel mask image, serves as a set of training samples.

[0051] The advantage of this invention is that images obtained from a single center can be projected onto the color spaces of images obtained from multiple hospitals under different staining conditions and scanning instruments, artificially increasing the diversity of training data; adjusting to real-world template values ​​avoids image artifacts and the problem of some tissues being unrecognizable due to arbitrary adjustments to color channel weights; Leo et al. found that color-related features have poor stability in deep learning models, and color standardization of training samples cannot solve the problem of inconsistent color spaces, and further aggravates the instability of these features; conversely, multiple studies by Bandi, Tellez et al. show that introducing color enhancement techniques during training can effectively improve model performance.

[0052] The following is an example method for precise object stitching enhancement: In the training set, multiple sub-images within a precise range of microvessels are randomly cropped from the sample images according to the microvessel mask image, and corresponding mask image sub-images are cropped. Starting from a random position in a randomly selected target image, the sample microvessel sub-image with randomly selected values ​​is pasted to that position, overwriting the original pixel values. The corresponding mask microvessel sub-image is then pasted to the mask image position corresponding to the target image, overwriting the original pixel values. Pixel-level majority voting is performed on the microvessel classification on the stitched mask image to update the classification of the covered microvessels. This method avoids image distortion caused by object cutting due to common stitching methods such as mosaic tiling and segmentation blending. Instead, it enriches the combination morphology of tissues and microvessels without changing the main background and maintaining the integrity of the object, effectively helping to improve the model training performance.

[0053] (6) Hard mining improves the class imbalance problem encountered in multi-class image segmentation. The distribution of different microvascular subtypes varies significantly under natural conditions and is related to the tumor grade and the nature of local tissues. In this invention, a combination of weighted focal loss and weighted dice loss is used to increase the proportion of hard samples in the loss function and improve the problem of insufficient sample size for some types of blood vessels in the training set.

[0054] Step 2: Evaluate the performance of the trained model and apply an active learning strategy to select samples. This includes the following steps:

[0055] (1) The validation set samples are sliced ​​into blocks using a sliding window tiling algorithm to reduce the impact of slicing on objects near the image edges and improve the model's performance in recognizing the edges of the sliced ​​images. This invention does not limit the size or magnification of the slices, nor does it limit the overlapping pixel values ​​set by the sliding window algorithm.

[0056] (2) Input the slice image into the trained network model to obtain the blood vessel and background segmentation probability matrix of the five subtypes of the corresponding slice.

[0057] (3) Post-processing of segmentation results, including: image reconstruction and weighted averaging of overlapping sub-segmentation probability matrices by using a sliding window image fusion algorithm, combining multiple sub-segmentation matrices and determining the segmentation result of each pixel in the image to be segmented; post-processing of the combined segmentation results by using connected component algorithm, majority voting algorithm, segmentation region morphology processing algorithm and noise processing algorithm to obtain the final segmentation result.

[0058] The sliding window image fusion algorithm includes: extracting the classification probability of each pixel on each slice image; re-overlapping the slice image to the original image position according to the coordinates of the slice image; generating a pixel overlap counting matrix; performing a weighted average of the pixel probability values ​​according to the overlap counting matrix; and selecting the classification with the highest probability value for each pixel as the final classification of that pixel.

[0059] The segmentation region morphological processing algorithm includes: extracting all connected objects on the fused image, performing an opening operation (i.e., erosion followed by dilation) to remove spots formed by image noise; performing a closing operation (i.e., dilation followed by erosion) to connect non-connected objects that have been mistakenly divided into small blocks; and filling holes in all connected regions.

[0060] The noise processing includes: the noise may include false positives with excessively small connected region areas; based on prior medical knowledge, a minimum area for microvessels is set, and all connected regions smaller than the set minimum area are deleted. This embodiment of the invention does not specifically limit this; the noise can also be other points that do not meet the requirements.

[0061] (4) Evaluate the performance of the trained model. For the samples in the validation set, compare them with the labeled results. The parameters include the average F1 score, the average Jaccard score, and the Hausdorff distance.

[0062] For each pixel in each category, its classification accuracy can be calculated: TP represents a positive sample predicted as positive, TN represents a negative sample predicted as negative, FP represents a negative sample predicted as positive, and FN represents a positive sample predicted as negative. Based on the accuracy of each pixel's classification, an F1 score can be calculated. The average F1 score is obtained by averaging the F1 scores of all categories, and this average score serves as a parameter for comparing the performance of the algorithm model. For the correct answer of a pixel in each category, its IoU coefficient can be calculated. The average IoU coefficient can be obtained by averaging the IoU coefficients of all categories, and can be used as a comparison parameter for the performance of the algorithm model.

[0063] For each segmented object, the Hausdorff distance based on the object's contour point set can be calculated: h(A, B) and h(B, A) are the one-way Hausdorff distances from set A to set B and from set B to set A, respectively. In fact, firstly, for each point a in point set A... i to a distance from this point a i The nearest midpoint of set B, b j Distance between ||a i -b j Sort the points and then take the maximum value of the distance as the value of h(A,B); h(B,A) can be obtained in the same way; H(A,B)=max(h(A,B),h(B,A)) that is, the bidirectional Hausdorff distance H(A,B) is the larger of the one-way distances h(A,B) and h(B,A), which measures the maximum degree of mismatch between two point sets and is a measure of the similarity of the outer contour shape of objects.

[0064] (5) Apply an active learning strategy to iterate the training model until the performance parameters reach the predetermined target. Specific steps include: calculating the Brier score on the probability matrix obtained from the segmentation, and measuring the prediction uncertainty of the trained model at the pixel level.

[0065]

[0066] in, Given a sample image and its corresponding mask image, y is one and only one of k fractals (k∈{1,...,K}). This represents the classification prediction result of the trained model for each pixel of the input image. This represents the probability value of the classification prediction result. The Brier score is the mean squared error between the probability matrix predicted by the model for each sample and the labeled result. Therefore, the lower the Brier score of a set of predictions, the higher the certainty of the prediction.

[0067] Based on the Brier score, samples with poor model prediction performance are selected, and new samples with high similarity to these samples are selected for labeling. The classifier model is updated iteratively.

[0068] The annotation of pathological images requires the expertise of physicians in the field. Due to the difficulties in image segmentation annotation, the number of available training samples is relatively limited. However, with a small number of training samples, the performance of the classifier can be significantly affected. How to achieve good classification performance with minimal manual annotation of samples has become a key issue in image segmentation tasks. Active learning strategies can efficiently utilize case and physician annotation resources, improving the efficiency of training set construction.

[0069] like Figure 3 As shown, the present invention also provides a training device based on image annotation and segmentation models, which is a deep neural network training device for pathological panoramic images. The device includes: a preprocessing module 100, used to extract and crop tissues from multiple pathological panoramic images and annotation masks respectively, to obtain multiple small images of the same size and corresponding mask images, forming a sample set; a training module 200, used to input the sample set into an initial neural network model, and calculate the output of the initial neural network model through a Softmax function to obtain the output probability vector of each pixel on each small image in the sample set; wherein, the initial neural network model is obtained through transfer learning training to obtain a neural network for segmenting tissues of interest in pathological panoramic images. The system includes a network model; a loss function calculation module 300, used to calculate the gradient magnitude and gradient density based on the output probability matrix of each small image in a preset sample set, and to calculate the loss function based on the gradient magnitude and gradient density; a post-processing module 400, used to fuse and reconstruct a panoramic image from the probability map predicted by the model corresponding to the sample image cropped by the preprocessing module, and to perform fractal voting, contour smoothing, hole filling, and noise reduction operations to improve the problem of difficult identification of edge objects in the sample image caused by cropping, and to improve the algorithm performance; and a performance evaluation module 500, used to evaluate the performance of the trained neural network model, extract samples with poor model prediction performance, and perform iterative processes of data labeling and algorithm training to further improve the model performance.

[0070] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method based on image annotation and segmentation models, characterized in that, Includes the following steps: Step 1: Obtain a panoramic image sequence of continuous sections of the sample pathological tissue, including H&E staining images and CD34 and α-SMA immunohistochemical staining images; select typical images from the sample dataset, and reselect the remaining data after model training feedback; classify microvessels based on prior medical knowledge; obtain an accurate reference standard for vascular presence and classification results by registering the corresponding CD34 and α-SMA immunohistochemical staining images with the H&E images to be labeled; annotate the H&E images against the registered immunohistochemical images to obtain a multi-class vascular annotation mask for the sample H&E images; Step 2: Preprocess the training samples obtained from the labeled images in Step 1; train the image segmentation convolutional neural network model to complete the microvascular classification and segmentation task; evaluate the performance of the currently trained model; select images with poor model performance based on the performance evaluation results, and reselect sample data that need to be labeled; repeat the training based on the new round of training samples until the predetermined performance index is reached.

2. The training method based on image annotation and segmentation model according to claim 1, characterized in that, The registration algorithm in step 1 includes manual registration, automatic registration based on image intensity, or semi-automatic registration based on landmarks, applying rigid or non-rigid deformation.

3. The training method based on image annotation and segmentation model according to claim 1, characterized in that, Training an image segmentation convolutional neural network model specifically includes: Step (1): Collect sample images and corresponding microvascular mask images as a set of samples; Step (2): Extract the tissue block image from the pyramid structure of the panoramic image at a set magnification factor, or scale the tissue block image at a set magnification factor; randomly allocate multiple sets of samples to the training set or validation set according to a certain ratio, and train the original segmentation model based on multiple sets of training samples; Step (3): Construct the network model; Step (4) Perform random sampling, that is, the network model performs random and consistent cropping of the sample image and the corresponding microvascular mask image according to a fixed input size, and fills the image with reflections if the edge is insufficient for the input size, so as to increase the diversity of the dataset; Step (5): Perform data augmentation; Step (6): Perform hard sample mining to improve class imbalance in multi-class image segmentation.

4. The training method based on image annotation and segmentation model according to claim 1, characterized in that, The training process involves: evaluating the performance of the currently trained model; selecting images where the model performs poorly based on the evaluation results, and then selecting new sample data for labeling; repeating the training process with a new set of training samples until the predetermined performance metrics are achieved. This includes the following steps: Step (1): Divide the validation set samples into blocks according to the sliding window algorithm; Step (2): Input the slice image into the trained network model to obtain the blood vessel and background segmentation probability matrices for the five subtypes of the corresponding slice; Step (3) Post-processing of segmentation results includes: performing image reconstruction and weighted averaging of overlapping sub-segmentation probability matrices using a sliding window image fusion algorithm; combining multiple sub-segmentation matrices and determining the segmentation result of each pixel in the image to be segmented; and post-processing the combined segmentation results using connected component algorithm, majority voting algorithm, segmentation region morphology processing algorithm, and noise processing algorithm to obtain the final segmentation result. Step (4): Evaluate the performance of the trained model; Step (5): Apply an active learning strategy to iterate the training model until the performance parameters reach the predetermined target.

5. The training method based on image annotation and segmentation model according to claim 3, characterized in that, Step (1) includes: using a global thresholding algorithm to extract the region where the tissue block is located, removing blank areas on the image with an area greater than a set minimum area value, extracting continuous tissue blocks according to the connected component algorithm, and if there are multiple connected components, segmenting the region according to connectivity to obtain independent tissue block images; the extracted tissue block images have irregular shapes, calculating the minimum bounding box of each tissue block shape, and using each bounding box as the range of the tissue block to be segmented; recording the upper left corner (x, y) coordinates of each tissue block image to be segmented so that the tissue block image and the blank area can be reconstructed into the size of the region to be segmented.

6. The training method based on image annotation and segmentation model according to claim 3, characterized in that, In step (3), the network model is a convolutional neural network, a deep neural network, or a recurrent neural network.

7. The training method based on image annotation and segmentation model according to claim 3, characterized in that, Step (5) includes: using sample images collected from different hospitals in the public dataset as multiple color templates, decomposing each template into three channels: hematoxylin, eosin, and residual in the RGB space, calculating the weight values ​​of the hematoxylin and eosin channels, setting them as template values, and creating a template value set; decomposing the channels of the sample image into three channels: hematoxylin, eosin, and residual, and randomly adjusting the weights of the hematoxylin and eosin channels of the sample image to a value in the template value set; recombining the adjusted hematoxylin and eosin channels and the unadjusted residual channel of the image to synthesize a new sample image, and using the vascular mask image as a set of training samples.

8. The training method based on image annotation and segmentation model according to claim 3, characterized in that, Step (6) includes: combining a weighted focus loss function and a weighted Dice loss function to increase the proportion of hard-to-distinguish samples in the loss function.

9. A training apparatus for implementing the training method based on image annotation and segmentation model as described in any one of claims 1-8, characterized in that, include: The preprocessing module is used to extract and crop tissue from multiple pathological panoramic images and labeled masks to obtain multiple small images of the same size and corresponding mask images, which constitute a sample set. The training module is used to input the sample set into an initial neural network model and calculate the output probability vector of each pixel in each small image in the sample set by using the output of the initial neural network model. The loss function calculation module is used to calculate the gradient magnitude and gradient density based on the output probability matrix of each small image in the preset sample set, and calculate the loss function based on the gradient magnitude and gradient density; the post-processing module is used to fuse and reconstruct the probability map predicted by the model corresponding to the sample image cropped by the preprocessing module into a panoramic image, and perform operations such as fractal voting, contour smoothing, hole filling, and noise reduction; the performance evaluation module is used to evaluate the performance of the trained neural network model, extract samples with poor model prediction performance, and perform an iterative process of data labeling and algorithm training.

10. The training device according to claim 9, characterized in that, The initial neural network model was trained through transfer learning and serves as a neural network model for segmenting tissues of interest in pathological panoramic images.

Images