Resolution adaptive seamless semantic segmentation method for digital pathological panoramic slice

Abstract

The invention discloses a resolution adaptive seamless semantic segmentation method for a digital pathological panoramic slice, and belongs to the technical field of image segmentation. The method comprises the following steps: acquiring a to-be-processed pathological panoramic slice image; performing adaptive indexing on the pathological panoramic slice image based on the physical resolution information to obtain a target image block; constructing a semantic segmentation model, and training the semantic segmentation model by using a self-supervised learning method based on a mask based on a data set with a specified resolution; and performing semantic segmentation on the pathological panoramic slice with any physical resolution by using the target semantic segmentation model. According to the method, a local visual field can be generated at any resolution level, and a high-quality full-slice panoramic segmentation result can be correctly generated through splicing.

Description

Technical Field

[0001] The invention relates to a digital pathology panoramic slice segmentation method, in particular to a resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices, belonging to the technical field of pathology image segmentation. Background Art

[0002] Digital pathology slides refer to images formed by digitizing slides using a dedicated digital slide scanner. Data can be read using a database or after format conversion. Nevertheless, due to differences in scanner manufacturers, optional components such as eyepieces, objectives, optical paths, etc., and scanning configurations, the image magnification may vary. This phenomenon is particularly evident in multi-center, large-scale cohorts. In digital pathology images, the image magnification information is recorded in resolution, often in units of microns per pixel (μm / px), and 0.227μm / px ~0.552μm / px are common resolution ranges.

[0003] Computational pathology uses techniques such as image analysis and deep learning to identify regions of interest or objects of interest in pathological images. Computational pathology models are usually developed in a single queue, extracting images of a fixed level, such as level 0, and combining them with physician annotations to generate image patches for training image classification or semantic segmentation tasks.

[0004] However, when applying the trained model to subsequent test slices in the hope of obtaining a complete segmentation map of the entire slice, two severe challenges will be encountered: First, the actual physical resolution of the slices is inconsistent with the resolution of the images used for model training, and it is impossible to find a similar resolution level. Directly applying the model will lead to a serious decrease in accuracy. When the resolutions of different slices are inconsistent, a more complicated manual selection process will be faced. Second, when processing pathological panoramic slices, non-overlapping sliding window sampling is used, and the local images are processed by the model separately, and then the results are stitched together. This solution will introduce noise at the edges due to zero padding in the convolution operation of the local image, resulting in obvious stitching seams composed of noise in the stitching results, which reduces the quality of the results. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices.

[0006] Technical solution: The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices of the present invention comprises the following steps: Step 1, obtaining a pathological panoramic slice image to be processed; Step 2, adaptively indexing the pathological panoramic slice image based on the physical resolution information to obtain the target image block; Step 3: construct a semantic segmentation model. Based on the dataset of the specified resolution, the semantic segmentation model is trained using a mask-based self-supervised learning method to obtain a target semantic segmentation model. Step 4: Use the target semantic segmentation model to perform semantic segmentation on pathological panoramic slices of arbitrary physical resolution.

[0007] Furthermore, step 2 includes: According to the original physical resolution of the panoramic pathology slice and the downsampling ratio of each layer, the physical resolution corresponding to each layer is calculated; Traverse all levels of physical resolution and find the most suitable level as the target level; Calculate the deviation proportionality coefficient according to the target resolution and the physical resolution of the target layer, adjust the sampling size according to the deviation proportionality coefficient, and obtain a new sampling size; The corresponding local image block is read from the panoramic pathological slice according to the target level, position and new sampling size, and the local image block is adjusted to the specified target size to obtain the target image block.

[0008] Furthermore, the semantic segmentation model is trained using a mask-based self-supervised learning method including: For each slice in the dataset, randomly select the slice The image area of ​​the proportion is masked, and the remaining image area is regarded as the unmasked area. The value range is from 0 to 1; The feature extraction submodule of the semantic segmentation model is used to extract features of the unmasked area; The reconstruction submodule of the semantic segmentation model is used to predict the pixel values ​​of the masked area.

[0009] Furthermore, the semantic segmentation model adopts a weighted mixed loss function, the formula is: , In the formula, is the loss for the self-supervised task, To monitor the loss of tasks, is the loss weight of the self-supervised task, is the loss weight of the supervision task, and .

[0010] Further, step 4 includes: The difference coefficient r between the resolution Q specified by the semantic segmentation model and the actual resolution s of the pathological panoramic slice is calculated as follows: ; According to the size of the original pathological panoramic slice The difference coefficient r of the resolution is used to calculate the size of the segmentation result image at the equivalent resolution Q. The formulas are: , , Where H represents the height of the original pathological panoramic slice at the highest resolution, h represents the height of the segmentation result image at a resolution equivalent to Q, W represents the width of the original pathological panoramic slice at the highest resolution, and w represents the width of the segmentation result image at a resolution equivalent to Q; Initialize the semantic segmentation label map M according to the calculated segmentation result map size; Perform a window sampling cycle with a step length of s on the semantic segmentation label map M, and fill the local image block obtained in each cycle into the corresponding area of ​​the semantic segmentation label map M; After completing the cycle, the complete semantic segmentation label map M is obtained.

[0011] Furthermore, a window sampling cycle with a step size of s is performed on the semantic segmentation label map M, including: Determine the position of the sampling starting point in the original pathological panoramic slice, the formula is: , , In the formula, , Respectively represent the horizontal and vertical coordinates of the sampling starting point on the original pathological panoramic slice; , Respectively represent the horizontal and vertical coordinates of the current sliding window position on the segmentation result graph, which are loop variables used to traverse the entire segmentation result graph; According to the position of the sampling starting point, the target image block is read from the original pathological panoramic slice, and the size is Semantic segmentation map of Crop at the center of the semantic segmentation map to get a size of The effective part R; Fill the valid part R into the semantic segmentation label map M area.

[0012] Further, finding the most appropriate level as the target level includes: When the target resolution of a certain level is greater than or equal to the physical resolution of the current level multiplied by the resolution tolerance ratio, the level is taken as the most appropriate level for sampling.

[0013] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. The present invention calculates the physical resolution of each level of the slice to find the most suitable sampling level, ensuring that any semantic segmentation model can run at the correct image resolution level, solving the problem in the prior art that the accuracy of the semantic segmentation model decreases due to inconsistent slice resolution; 2. The present invention proposes a generally applicable edge cutting and stitching scheme. In the sliding window sampling process, only the central effective area of ​​the segmentation result is used for filling, which avoids the influence of edge noise, forms a high-quality seamless panoramic segmentation result, improves the stitching accuracy, and avoids the problems of stitching seams and inaccurate segmentation results caused by edge noise in the prior art; 3. The present invention is applicable to any digital slice format with physical resolution information and data reading interface, as well as digital pathological slice analysis of different organs and different staining types; 4. The present invention supports multiple model input and output sizes by adjusting the filling mode of the convolutional layer, which meets the applicable conditions of different models and improves the flexibility and applicability of the model; 5. The present invention has efficient data reading capability. Through the adaptive resolution image indexing algorithm, the required local image blocks can be quickly and accurately acquired, thus reducing the time for data reading and processing; 6. The present invention reduces the dependence on labeled data by introducing a self-supervised learning method, while improving the generalization ability of the semantic segmentation model and its ability to understand complex tissue structures; through mask processing and reconstruction modules, the model can learn the global structure and contextual information of the image without labeled data, thereby showing higher accuracy and robustness when processing complex tissue structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of a resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices; Figure 2 It is a structural diagram of digital pathology images; Figure 3 A flowchart for adaptive indexing of full-slice images based on physical resolution; Figure 4 This is the workflow diagram of the semantic segmentation model; Figure 5 Flowchart for constructing semantic segmentation label map M; Figure 6 This is the effect diagram of lung cancer area detection in the panoramic pathology section; Figure 7 This is a schematic diagram of multi-category tissue segmentation results; Figure 8 It is the overall training loss curve; Fig. 9is the overall dice coefficient graph. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0016] The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices described in this embodiment is as follows: Figure 1 As shown, the method comprises the following steps: Step 1: Obtain the pathological panoramic slice image to be processed.

[0017] To improve the response speed in user interaction, digital pathology images are usually stored in multi-resolution image formats, using different levels to represent images at different zoom levels. Level 0 stores images of original size and highest resolution. The higher the level, the higher the downsampling ratio and the smaller the image size. Figure 2 shown.

[0018] Step 2: adaptively index the pathological panoramic slice image based on the physical resolution information to obtain the target image block.

[0019] Furthermore, step 2 includes: Query location: Based on the original physical resolution of panoramic pathology slices and the downsampling ratios at each level , calculate the physical resolution corresponding to each level , the calculation formula is: , Traverse all levels of physical resolution and find the most suitable level as the target level; Calculate the sampling size: The deviation ratio coefficient is calculated according to the target resolution and the physical resolution of the target layer, and the sampling size is adjusted according to the deviation ratio coefficient to obtain a new sampling size; wherein, the calculation method of adjusting the sampling size is to multiply the original sampling size by the deviation ratio coefficient to obtain the new sampling size.

[0020] Sampling by level, scaling and post-processing: The corresponding local image block is read from the panoramic pathology slice according to the target level, position and new sampling size, and the local image block is adjusted to the specified target size to obtain the target image block.

[0021] The acquired local image block is scaled, and the scaling ratio is the inverse of the deviation ratio coefficient, and finally a target image block that meets the requirements is obtained.

[0022] Further, finding the most appropriate level as the target level includes: When the target resolution of a certain level is greater than or equal to the physical resolution of the current level Multiply by the resolution tolerance ratio , then this level is taken as the most appropriate level for sampling.

[0023] In one example, if The value can be 0.85.

[0024] The adaptive indexing described in this embodiment is applicable to any digital slice format with physical resolution information and a data reading interface, and can process any physical resolution required by the user. The flowchart of the full slice image adaptive indexing based on physical resolution is shown in FIG. Figure 3 The adaptive resolution image indexing algorithm described in this example can further establish a full-slice reasoning framework to achieve sub-region-by-sub-region semantic segmentation of the full slice and perform in-situ stitching.

[0025] Step 3: Build a semantic segmentation model. Based on the dataset of the specified resolution, use a mask-based self-supervised learning method to train the semantic segmentation model to obtain the target semantic segmentation model.

[0026] The specified resolution is the resolution adapted to the training phase and is the resolution specified by the user. The dataset can be any pathology image dataset with the same resolution, for example, a dataset of the cancer genome atlas can be selected.

[0027] Furthermore, the semantic segmentation model is trained using a mask-based self-supervised learning method including: For each slice in the dataset, randomly select the slice The image area of ​​the proportion is masked, and the remaining image area is regarded as the unmasked area. The value range is from 0 to 1; The feature extraction submodule of the semantic segmentation model is used to extract features of the unmasked area. The features of the unmasked area provide contextual information for the model to help it better predict the content of the masked area. The reconstruction submodule of the semantic segmentation model is used to predict the pixel values ​​of the mask area. The pixel values ​​of the mask area are the targets that the model needs to predict. By predicting these pixel values, the model can learn the global structure and contextual information of the image.

[0028] Combination Figure 4 As shown, further, step 4 includes: The difference coefficient r between the resolution Q specified by the semantic segmentation model and the actual resolution s of the pathological panoramic slice is calculated as follows: ; According to the size of the original pathological panoramic slice The difference coefficient r of the resolution is used to calculate the size of the segmentation result image at the equivalent resolution Q. The formulas are: , , Where H represents the height of the original pathological panoramic slice at the highest resolution, h represents the height of the segmentation result image at a resolution equivalent to Q, W represents the width of the original pathological panoramic slice at the highest resolution, and w represents the width of the segmentation result image at a resolution equivalent to Q; Initialize the semantic segmentation label map M according to the calculated segmentation result map size; Among them, the label map means that each pixel represents a category, and a The full zero matrix of is used to calculate the size of the segmentation result map to obtain h and w, and then an empty label map M is created based on h and w to store the final semantic segmentation result; Perform a window sampling cycle with a step length of s on the semantic segmentation label map M, and fill the local image block obtained in each cycle into the corresponding area of ​​the semantic segmentation label map M; After completing the cycle, the complete semantic segmentation label map M is obtained.

[0029] Combination Figure 5 As shown, further, a window sampling cycle with a step size of s is performed on the semantic segmentation label map M, including: Determine the position of the sampling starting point in the original pathological panoramic slice, the formula is: , , In the formula, , Respectively represent the horizontal and vertical coordinates of the sampling starting point on the original pathological panoramic slice; , They represent the horizontal and vertical coordinates of the current sliding window position on the segmentation result graph, which are loop variables used to traverse the entire segmentation result graph; S represents the input size of the semantic segmentation model, and s represents the effective output size of the semantic segmentation model; According to the position of the sampling starting point, the target image block is read from the original pathological panoramic slice, and the size is Semantic segmentation map of Crop at the center of the semantic segmentation map to get a size of The effective part R; Fill the valid part R into the semantic segmentation label map M area.

[0030] Step 3 improves the stitching accuracy of the segmentation results. The size result will be adopted in the end. However, due to the discard of the edge, this factor must be taken into account when splicing, and appropriate adjustments must be made to achieve a seamless splicing effect.

[0031] Step 4: Use the target semantic segmentation model to perform semantic segmentation on pathological panoramic slices of arbitrary physical resolution.

[0032] The target semantic segmentation model obtained after training can be used to perform semantic segmentation on pathological panoramic slices of any physical resolution and obtain segmentation results of specified resolution.

[0033] If the output size of the semantic segmentation model is a downsample of the input size, the result can be upsampled proportionally and the above process can still be used. For a fully convolutional network with no padding mode, its convolutional layer should be adjusted to pad to the same size mode to meet the algorithm's applicable conditions.

[0034] Figure 6 The figure shows the lung cancer region recognition result in the panoramic pathology slice using the method proposed in this embodiment. In the figure, image represents the original pathology image, cutoff represents the cropping width, and in this example, four different cropping widths of 0, 32, 64, and 128 are selected respectively. The part indicated by the black arrow in the figure is the segmentation gap. By configuring the appropriate cropping size and cooperating with the full slice segmentation algorithm, it can be clearly seen that the gap is eliminated and a high-quality segmentation result is obtained.

[0035] Figure 7 The multi-category tissue segmentation results obtained using the target semantic segmentation model described in this embodiment are as follows: epithelial tissue corresponds to cyan in the figure, stroma corresponds to orange, immune infiltration corresponds to blue, and microvessels correspond to yellow in the figure. In order to establish this multi-category tissue segmentation model, 669 regions of interest (ROIs) were extracted from 5 cancer genome atlas cohorts. When extracting the regions of interest, the most appropriate image level was first obtained. The average area of ​​these ROIs was 1.991 mm2, with a standard deviation of , roi is saved in TIF format with a fixed resolution of .

[0036] Annotation of these ROIs was done using the Automated Slide Analysis Platform (ASAP) software and saved in XML format. After the annotation stage, the XML files were converted to TIF masks using the Python interface of ASAP. The ROIs were then aligned to the masks and jointly cropped into patches for model training and validation. Specifically, a sliding window approach was used to extract the ROIs from the tissue images and masks. The pixel patches are 448 pixels in stride. Overlap sampling is used to avoid under-training of pixels at the edge. A total of 17,159 patches are generated by image cropping, which are then randomly divided into training and validation sets in a ratio of 7:3. Sampling is then performed to avoid edge effects, make full use of image information, and reduce errors in the stitching process.

[0037] In order to further improve the generalization ability of the model, a mask-based self-supervised learning method is introduced in the training process. In the data preparation stage, some image areas are randomly selected for masking, such as randomly setting 30% of the image area to zero, and then the model is asked to predict the pixel values ​​of these masked areas. In this way, the model can learn the global structure and contextual information of the image without labeled data, thereby enhancing its ability to understand complex tissue structures. For the architecture of the semantic segmentation model, it can be adjusted based on the existing U-Net model, where the encoder is used as a feature extraction submodule to extract features from unmasked areas, and the decoder is used as a reconstruction submodule to predict the pixel values ​​of the masked area. In order to balance the impact of the two tasks, self-supervision and supervision, a weighted hybrid loss function is used, where the loss weight of the self-supervision task can be set to 0.3 and the loss weight of the supervision task can be set to 0.7. In this way, the semantic segmentation model can simultaneously learn the supervision information in the labeled data and the contextual information in the unlabeled data.

[0038] For example, an existing semantic segmentation model can be used and appropriately adjusted and optimized during training. The generated slices are used to train a semantic segmentation model, such as using a U-Net model with a VGG-19 encoding branch as a semantic segmentation model for training, and adjusting the channels in the decoder path, while introducing a lightweight coordinate attention mechanism. During training, the input size of the model is set to Pixels, and introduce mild color enhancement to improve the generalization ability of the model. The resolution-adaptive seamless semantic segmentation method described in this embodiment has achieved remarkable results. Through fixed-resolution image input, optimization of model structure and application of sliding window method, the model performs well in the segmentation of various tissue types, such as Figures 8 to 9 As shown in Figure 2, the Dice coefficient for most categories exceeds 0.8, and even the challenging small blood vessel category can reach a score of more than 0.6 in the later stages of training. The loss function is a 1:1 mixture of cross entropy and Dice loss, using the Adam optimizer with an initial learning rate of , after 60 times, the learning rate is multiplied by 0.9 every 10 times, and the training stops after 160 times.

Claims

1. A resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices, characterized in that: The following steps are involved: Step 1, obtaining a pathological panoramic slice image to be processed; Step 2, adaptively indexing the pathological panoramic slice image based on the physical resolution information to obtain the target image block; Step 3: construct a semantic segmentation model. Based on the dataset of the specified resolution, the semantic segmentation model is trained using a mask-based self-supervised learning method to obtain a target semantic segmentation model. Step 4: Use the target semantic segmentation model to perform semantic segmentation on pathological panoramic slices of arbitrary physical resolution.

2. The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices according to claim 1, characterized in that: Step 2 includes: According to the original physical resolution of the panoramic pathology slice and the downsampling ratio of each layer, the physical resolution corresponding to each layer is calculated; Traverse all levels of physical resolution and find the most suitable level as the target level; Calculate the deviation proportionality coefficient according to the target resolution and the physical resolution of the target layer, adjust the sampling size according to the deviation proportionality coefficient, and obtain a new sampling size; The corresponding local image block is read from the panoramic pathological slice according to the target level, position and new sampling size, and the local image block is adjusted to the specified target size to obtain the target image block.

3. The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices according to claim 2, characterized in that: Training a semantic segmentation model using a mask-based self-supervised learning approach involves: For each slice in the data set, the image area with a γ ratio of the slice is randomly selected for masking, and the remaining image area is used as the unmasked area, where the value of γ ranges from 0 to 1; The feature extraction submodule of the semantic segmentation model is used to extract features of the unmasked area; The reconstruction submodule of the semantic segmentation model is used to predict the pixel values ​​of the masked area.

4. The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices according to claim 3, characterized in that: The semantic segmentation model uses a weighted mixed loss function, the formula is: ， In the formula, is the loss for the self-supervised task, To monitor the loss of tasks, is the loss weight of the self-supervised task, is the loss weight of the supervision task, and .

5. The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices according to claim 4, characterized in that: Step 4 includes: The difference coefficient r between the resolution Q specified by the semantic segmentation model and the actual resolution s of the pathological panoramic slice is calculated as follows: ; According to the size of the original pathological panoramic slice The difference coefficient r of the resolution is used to calculate the size of the segmentation result image at the equivalent resolution Q. The formulas are: ， ， Where H represents the height of the original pathological panoramic slice at the highest resolution, h represents the height of the segmentation result image at a resolution equivalent to Q, W represents the width of the original pathological panoramic slice at the highest resolution, and w represents the width of the segmentation result image at a resolution equivalent to Q; Initialize the semantic segmentation label map M according to the calculated segmentation result map size; Perform a window sampling cycle with a step length of s on the semantic segmentation label map M, and fill the local image block obtained in each cycle into the corresponding area of ​​the semantic segmentation label map M; After completing the cycle, the complete semantic segmentation label map M is obtained.

6. The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices according to claim 5, characterized in that: A window sampling cycle with a step size of s is performed on the semantic segmentation label map M, including: Determine the position of the sampling starting point in the original pathological panoramic slice, the formula is: ， ， In the formula, , Respectively represent the horizontal and vertical coordinates of the sampling starting point on the original pathological panoramic slice; , Respectively represent the horizontal and vertical coordinates of the current sliding window position on the segmentation result graph, which are loop variables used to traverse the entire segmentation result graph; According to the position of the sampling starting point, the target image block is read from the original pathological panoramic slice, and the size is Semantic segmentation map of Crop at the center of the semantic segmentation map to get a size of The effective part R; Fill the valid part R into the semantic segmentation label map M area.

7. The resolution-adaptive seamless semantic segmentation method for digital pathology panoramic slices according to any one of claims 2 to 6, characterized in that: Finding the most appropriate tier as the target tier involves: When the target resolution of a certain level is greater than or equal to the physical resolution of the current level multiplied by the resolution tolerance ratio, the level is taken as the most appropriate level for sampling.

Images