A pathological image segmentation method, system, storage medium, device and terminal

Abstract

The present application belongs to the technical field of image segmentation, and discloses a pathological image segmentation method, system, storage medium, equipment and terminal, acquires pathological image slice data with lesion annotation; the pathological image slice data with lesion annotation is sent into a model for training to obtain a pathological image segmentation model; a target pathological image slice needing lesion annotation is input into the pathological image segmentation model to obtain an image with completed lesion annotation. The present application proposes a pathological image segmentation method based on multi-view feature fusion, which can help doctors quickly locate the lesion tissue area and improve the diagnosis efficiency; the multi-view fusion module can obtain the range information of the multiple receptive fields of the low magnification image; the feature soft alignment module can obtain the surrounding information of the low magnification image and can smooth the edge information; the channel attention module can screen the features extracted by multiple convolutions to extract features more suitable for segmentation.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and particularly relates to a method, system, storage medium, device and terminal for segmenting pathological images. Background Technology

[0002] Currently, cancer is one of the leading diseases affecting people's health and well-being. Improving diagnostic efficiency and addressing the uneven distribution of medical resources are urgent issues that need to be resolved. Introducing artificial intelligence-based automated diagnostic methods into cancer treatment is key to solving these two problems. Pathological image analysis is a gold standard in cancer diagnosis. Pathological images are generated by scanning slides along the vertical z-axis at different focal planes and stacking the images to create composite multiplanar images. Therefore, multi-scale image information can be obtained at different magnifications.

[0003] Pathologists need to quantify the area of ​​lesions in pathological images, as this is crucial for subsequent assessment of cancer cell burden, treatment planning, and postoperative examinations. With the successful application of deep learning in the medical field, using deep learning technology for pathological image segmentation has become a key research direction for quantifying lesion area. For example, the Unet structure proposed by Ronneberger et al. is a classic model in medical image segmentation. However, in the field of pathological images, due to the large file size (often several gigabytes), it is necessary to first slice the entire image and then segment each slice. However, this method often results in the loss of contextual information in the image. Compared to segmentation at a single magnification, introducing a corresponding low-magnification image can supplement the contextual information. Furthermore, low-magnification images contain a wide field of view of tissue or structural features, offering more easily distinguishable features compared to high-magnification images. While high-magnification images have a narrower field of view, they provide cellular-level detail, similar to how pathologists observe and, when tissue structural features cannot be accurately identified, can be further confirmed at the cellular level by magnifying the image.

[0004] For the reasons mentioned above, compared to the traditional method of using only single-scale images, the multi-scale approach can achieve information complementarity by fusing high-magnification images containing detailed information with their corresponding low-magnification images containing structural information, thus improving the segmentation accuracy of the model. Currently, there are several main methods for pathological image segmentation based on the idea of ​​multi-scale feature fusion. Gu et al.'s scheme uses multi-level encoders for images at different scales, but only uses a single decoder for image segmentation in the image with the highest magnification. Next, the work of Tokunaga et al. led to the proposal of a multi-level encoder and corresponding multi-level decoder network, but feature fusion was only performed in the last layer of the model. Finally, the multi-level multi-encoder multi-decoder network proposed by Ho et al. achieves multi-level feature fusion, providing more diverse segmentation information for the model.

[0005] However, the existing technology also has its problems and shortcomings:

[0006] (1) In the multi-scale feature fusion segmentation network of pathological images, the previous models often used a relatively simple method to fuse multi-scale pathological image slices, but this fusion method cannot extract more information that is conducive to improving segmentation accuracy.

[0007] (2) In the existing multi-scale pathological image segmentation model, the low-magnification image features and local fusion features are aligned by direct cropping, which will cause the loss of information around the low-magnification image that is conducive to the segmentation of the image edge.

[0008] (3) In existing multi-scale pathological image segmentation models, feature extraction of low-magnification images is generally achieved by using a single receptive field convolution. However, since the morphological structures of lesions and normal tissues in pathological images are similar, the single receptive field is narrow and cannot provide information on adjacent tissues for discrimination, which will reduce the accuracy of the model. Summary of the Invention

[0009] To address the problems existing in the prior art, the present invention provides a pathological image segmentation method, system, storage medium, device, and terminal, and particularly relates to a pathological image segmentation method, system, storage medium, device, and terminal based on multi-view feature fusion.

[0010] Compared with traditional multi-scale pathological image segmentation schemes, (1) this invention designs a feature soft alignment method for feature extraction. By introducing multiple dilated convolutions to extract adjacent features of low-magnification image edges and then performing feature alignment, more reference information is provided for image edge pixels, thereby improving the accuracy of model segmentation and also smoothing edge noise points. (2) In view of the problem that the receptive field of low-magnification image feature extraction methods in multi-scale image fusion is narrow and cannot provide adjacent tissue information to help the model make judgments, resulting in low model accuracy, we use multiple dilated convolutions to obtain multiple receptive field features of low-magnification images. At the same time, we introduce a channel attention mechanism to filter multiple receptive field features, so that the model can combine appropriate adjacent features for discrimination, thereby enhancing the accuracy of the model.

[0011] This invention is implemented as follows: a pathological image segmentation method, the pathological image segmentation method comprising:

[0012] Obtain pathological image slice data with lesion annotation; feed the pathological image slice data with lesion annotation into the model for training to obtain the pathological image segmentation model; input the target pathological image slice that needs lesion annotation into the pathological image segmentation model to obtain the image with lesion annotation.

[0013] Furthermore, the pathological image segmentation model adopts a UNet structure. The input of the pathological image segmentation model is a high-magnification slice and a low-magnification slice of a pathological image with a size of 256*256*3. Each input image has features and receptive fields with different spatial resolutions.

[0014] Furthermore, the pathological image segmentation method includes the following steps:

[0015] Step 1: Use an encoder to input high-magnification and low-magnification slices of the pathological image into the pathological image segmentation model for encoding;

[0016] Step 2: The high-magnification slice features and low-magnification slice features of the coding layer, as well as the features of the fourth coding layer, are input into the multi-view feature fusion module using the decoder.

[0017] Step 3: After fusing the two input feature maps with different magnifications using the multi-view feature fusion module, the feature maps are concatenated with the upper-layer features and then the fused features are input into the subsequent decoding layer.

[0018] Furthermore, each of the decoding layers consists of an upsampling layer and two convolutional layers. The upsampling layer enlarges the original feature map by two times in width and height. Each of the two convolutional layers consists of a 3x3 convolutional layer, a BN layer, and a ReLU activation function. The feature map enters the next decoding layer, and each layer uses the same operation. Finally, the fifth layer is connected to a segmentation layer, which outputs the feature map as a 256*256 pixel segmentation map with predicted values ​​of 0 and 1. Finally, the segmentation map and the ground truth labels are used to calculate the DICE loss and train the model.

[0019] The formula for calculating the dice loss is as follows:

[0020]

[0021] In the formula, the Dice coefficient is a set similarity metric used to calculate the similarity between two samples, with a value range of [0, 1]; True Positive, is judged as a positive sample and is indeed a positive sample; True Negative, is judged as a negative sample and is indeed a negative sample; False Positive, is judged as a positive sample but is actually a negative sample; False Negative, is judged as a negative sample but is actually a positive sample.

[0022] Furthermore, the multi-view feature fusion module employs diverse dilated convolutions to extract diverse features from low-magnification images, which are then fused to achieve soft feature alignment and multi-view feature extraction.

[0023] The feature soft alignment includes: extracting features from low-magnification feature maps by introducing multiple dilated convolutions with different receptive fields, and then cropping the low-magnification image based on spatial relative positions. Low-magnification features are processed by convolutional layers to extract feature maps. Three dilated convolutions with different rate rates (1, 3, 6) are designed to obtain information from different receptive fields, enhancing the ability to extract edge features from subsequent feature maps. Each convolution is 3x3 in size, followed by ReLU and BN. It also includes an average global pooling layer, followed by ReLU activation and BN, used to compress the feature map channels to 1 to extract global information from the image. The extracted feature maps are then concatenated along the channels, aligned by relative positions, and then cropped.

[0024] Furthermore, the multi-view feature extraction includes:

[0025] Multiple convolutions are used to extract features from low-magnification images. In the multi-view feature extraction structure, L and H represent low-magnification and high-magnification feature maps, respectively. The spur represents the relationship between upper and lower layers in the UNet structure. Rate = 1, 3, 5 represent dilated convolutions with Rate values ​​of 1, 3, and 5, respectively. Crop means cropping the image at the top left corner of the original image at a ratio of 1 / 16 of the total size. The decoding layer contains a 2x upsampling layer, a feature concatenation layer, and two 3x3 convolutional layers. Each convolutional layer is followed by ReLU and BN after each convolution.

[0026] High-magnification images are processed through multiple dilated convolutional and average pooling layers to extract features from various receptive fields. These features are then concatenated across channels. A 1x1 convolutional layer is used to fuse the extracted features, which are then cropped according to their relative positions. The resulting data is then fed into a channel attention layer, where different weights are assigned to each channel. Features extracted from different convolutional layers are filtered and fed into the decoding layer. In the decoding layer, high- and low-magnification image features are concatenated and fused using two convolutional layers. The high- and low-magnification images are then upsampled and concatenated with features from the previous layer at high magnification. This upsampling is then fed into subsequent convolutional layers to extract features, and finally, the data is fed into the subsequent layers of the entire model.

[0027] Another object of the present invention is to provide a pathological image segmentation system for implementing the aforementioned pathological image segmentation method, the pathological image segmentation system comprising an encoder, a multi-view feature fusion module, and a decoder.

[0028] The encoder is used to input high-magnification and low-magnification slices of pathological images into the model for encoding. The encoder adopts the ResNet50 structure, which contains a total of 5 stages. Each stage contains various convolutional, BN, activation layers and residual structures. The feature maps of the five different stages of ResNet50 are obtained as features of the five encoding layers of UNet.

[0029] The decoder is used to input the high-magnification slice features and low-magnification slice features of the coding layer, as well as the features of the coding layer of the fourth layer, into the multi-view feature fusion module, starting from the fifth coding layer.

[0030] The multi-view feature fusion module includes a feature soft alignment module and a multi-view convolution module, which are used to fuse the two input feature maps with different magnifications and then concatenate them with the upper-layer features on the channel side, and then input the fused features into the subsequent decoding layer.

[0031] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the pathological image segmentation method.

[0032] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the pathological image segmentation method.

[0033] Another objective of the present invention is to provide an information data processing terminal for implementing the aforementioned pathological image segmentation system.

[0034] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0035] This invention discloses a pathological image segmentation method based on multi-view feature fusion. The segmentation network includes modules: an encoder, a multi-view fusion module, and a decoder. The multi-view fusion module is located between the encoder and the decoder and includes two functions: feature soft alignment and multi-view feature extraction. Pathological image segmentation can help doctors quickly locate lesion tissue regions, thereby improving their diagnostic efficiency. In multi-scale feature fusion segmentation networks for pathological images, previous models often used relatively simple methods to fuse multi-scale pathological image slices. However, such fusion methods often fail to extract more information that is beneficial to improving segmentation accuracy. Therefore, this invention proposes a multi-level, multi-view fusion method for multi-resolution slices of pathological images.

[0036] The pathological image segmentation method provided by this invention has the following advantages: 1. The multi-view fusion module can acquire the range information of multiple receptive fields in low-magnification images. 2. The feature soft alignment module, compared with traditional models, can acquire the surrounding information of low-magnification images and can smooth the edge information. 3. The channel attention module filters the features extracted from various convolutions to extract features that are more suitable for segmentation. 4. Feature fusion between multiple levels can extract information from different levels.

[0037] Does the technical solution of this invention solve a technical problem that people have long desired to solve but have never been able to successfully address?

[0038] (1) In the multi-scale feature fusion segmentation network of pathological images, the previous models often used a relatively simple method to fuse multi-scale pathological image slices, but this fusion method cannot extract more information that is conducive to improving segmentation accuracy.

[0039] (2) In the existing multi-scale pathological image segmentation model, the low-magnification image features and local fusion features are aligned by direct cropping, which will cause the loss of information around the low-magnification image that is conducive to the segmentation of the image edge.

[0040] (3) In existing multi-scale pathological image segmentation models, feature extraction of low-magnification images is generally achieved by using a single receptive field convolution. However, since the morphological structures of lesions and normal tissues in pathological images are similar, the single receptive field is narrow and cannot provide information on adjacent tissues for discrimination, which will reduce the accuracy of the model.

[0041] To overcome the shortcomings of the prior art, the present invention proposes the following solutions:

[0042] (1) This invention designs a feature soft alignment method for feature extraction. By introducing multiple dilated convolutions to extract adjacent features of low-magnification image edges and then aligning the features, more reference information is provided for image edge pixels, thereby improving the accuracy of model segmentation and smoothing edge noise points. (2) To address the problem that the receptive field of low-magnification image feature extraction methods in multi-scale image fusion is narrow and cannot provide adjacent tissue information to help the model make judgments, resulting in low model accuracy, we use multiple dilated convolutions to obtain multiple receptive field features of low-magnification images. At the same time, we introduce a channel attention mechanism to filter multiple receptive field features, so that the model can combine appropriate adjacent features for discrimination, thereby enhancing the accuracy of the model. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the pathological image segmentation method provided in the embodiments of the present invention;

[0045] Figure 2 This is a flowchart of the pathological image segmentation network provided in an embodiment of the present invention;

[0046] Figure 3 This is a diagram of the multi-scale pathological image segmentation network structure provided in the embodiments of the present invention;

[0047] Figure 4 This is a structural diagram of the feature soft alignment module provided in an embodiment of the present invention;

[0048] Figure 5 This is a structural diagram of the multi-view feature extraction module provided in an embodiment of the present invention;

[0049] Figure 6 This is a structural diagram of the decoding layer module provided in an embodiment of the present invention;

[0050] Figure 7 This is a comparison diagram of the segmentation effect provided in the embodiments of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] To address the problems existing in the prior art, the present invention provides a method, system, storage medium, device, and terminal for pathological image segmentation. The present invention will be described in detail below with reference to the accompanying drawings.

[0053] To enable those skilled in the art to fully understand how the present invention is specifically implemented, this section provides an explanatory description of the embodiments that expand upon the technical solutions of the claims.

[0054] like Figure 1 As shown, the pathological image segmentation method provided in this embodiment of the invention includes the following steps:

[0055] S101, Obtain pathological image slice data with lesion annotations;

[0056] S102, Input the pathological image slice data with lesion annotations into the model for training to obtain the pathological image segmentation model;

[0057] S103: Input the target pathological image slice that needs lesion annotation into the pathological image segmentation model to obtain the image with completed lesion annotation.

[0058] The flowchart of the pathological image segmentation network provided in this embodiment of the invention is as follows: Figure 2 As shown.

[0059] This invention proposes a pathological image segmentation network based on multi-view feature fusion, whose main modules include an encoder, a multi-view fusion module, and a decoder. The specific descriptions of each module are as follows:

[0060] I. Overall Network Structure

[0061] Because pathological images exceed the current model's acceptable size, the mainstream approach is to magnify the images to the highest possible magnification, slice them, and then segment the slices. Segmentation from a single magnification level presents two main challenges: firstly, the image provides a narrow receptive field; secondly, it doesn't utilize images at lower magnification levels to provide structural features.

[0062] Therefore, this invention inputs corresponding slices of pathological images at two different magnifications into its multi-scale network, thereby providing the model with both cellular-level detail and tissue-level structural information. The model of this invention includes multiple encoders and decoders, as well as multiple links between them, to obtain feature maps with richer information. Simultaneously, this invention also links images at different magnifications in each layer of the model, allowing the model to use images at different magnifications to achieve more accurate segmentation.

[0063] Specific model information such as Figure 3 As shown, the overall model architecture adopts the UNet structure. The model input consists of high-magnification and low-magnification slices of a 256*256*3 size pathological image, each with different spatial resolution features and receptive fields. First, the two slices are input into the model for encoding. The encoder of this invention uses a ResNet50 structure, containing a total of 5 stages. Each stage includes various convolutional layers, BN layers, activation layers, and residual structures. Feature maps from the five different ResNet50 stages are obtained as features for the five encoding layers of the UNet. Then, the model enters the decoding process, starting from the fifth encoding layer. The high-magnification and low-magnification slice features of this layer, along with the features from the fourth encoding layer, are input into a multi-view feature fusion module. This module contains two sub-modules: a feature soft alignment module and a multi-view convolution module. In this module, the two different magnification feature maps are fused and then concatenated with the upper-layer features along the channel. The fused features are then input into the subsequent decoding layers. Each decoding layer consists of an upsampling layer and two convolutional layers. The upsampling layer enlarges the original feature map by a factor of two. Each of the two convolutional layers consists of a 3x3 convolutional layer, a batch normalization (BN) layer, and a ReLU activation function. The feature map then enters the next decoding layer, and each layer performs the same operation. Finally, the fifth layer connects to a segmentation layer, which outputs a 256*256 pixel segmentation map with predicted values ​​of 0 and 1. The segmentation map is then compared with the ground truth labels to calculate the Dice loss for model training.

[0064] The dice loss is given by the following formula:

[0065]

[0066] The Dice coefficient is a set similarity metric, typically used to calculate the similarity between two samples. The value ranges from [0, 1]. True Positive means the sample is considered positive and is indeed positive; True Negative means the sample is considered negative and is indeed negative; False Positive means the sample is considered positive but is actually negative; False Negative means the sample is considered negative but is actually positive.

[0067] II. Multi-view feature fusion module

[0068] The multi-view feature fusion module of this invention mainly uses diverse dilated convolutions to extract diverse features from low-magnification images, and then fuses them. It has two main functions: (1) soft feature alignment; (2) multi-view feature extraction. These two functions are described below.

[0069] (1) Feature soft alignment

[0070] To fuse multi-scale features, existing methods employ two different approaches: one is to directly stitch feature maps of different scales together from the channels. This approach does not consider the relative positional relationships in feature integration (i.e., the spatial positions of the two different feature images are not aligned), and this method may introduce noise into the model; the second approach is to directly crop the low-magnification feature map according to its relative position (e.g., ...). Figure 3 The dashed lines in the image indicate the spatial relative positions of the two images. Using this method leads to the loss of information in the low-magnification feature map itself, such as structural features that could help the model identify pathological tissues. To address these issues, this invention proposes a soft-alignment structure. This involves introducing multiple dilated convolutions with different receptive fields to extract features from the low-magnification feature map, and then cropping the low-magnification image based on spatial relative positions. This allows for the introduction of a wider range of structural features into the model and supplements the edge structure information of the low-magnification image. Furthermore, because convolution calculates more edge structure features, it also smooths out edge noise points in the low-magnification feature map compared to direct cropping. The specific structure is as follows: Figure 4As shown, the low-magnification features are first extracted and mapped into feature maps through convolutional layers. Inspired by the analysis of LC Chen et al., this invention designs three dilated convolutions with different rate rates (1, 3, 6) to obtain information from different receptive fields, enhancing the ability to extract edge features from subsequent feature maps. Each convolution is 3x3 in size, followed by ReLU and BN. In addition, there is an average global pooling layer, followed by ReLU activation and BN. This layer compresses the channels of the feature map to 1 to extract global information of the image. The extracted feature maps are then concatenated along the channels, and then cropped after relative alignment (for example, in this experiment, the relative position relationship is such that, from the perspective of the entire image, the area in the upper left corner of the low-magnification image is the field of view of the high-magnification image in the entire image).

[0071] (2) Multi-view feature extraction

[0072] This invention employs multiple convolutions to extract features from low-magnification images. Compared to traditional single-convolution methods, this invention provides the model with diverse visual features and increases its receptive field. Furthermore, because the model of this invention uses a UNet structure, its feature maps at different levels provide different visual representations; for example, lower layers provide local detail information, while higher layers provide global information. This invention further enriches the model's receptive field information by performing multi-view feature extraction on each layer's feature map. The structure of multi-view feature extraction is as follows: Figure 5 As shown, L and H represent low-magnification and high-magnification feature maps, respectively. "Hawk" represents the relationship between upper and lower layers in the UNet structure. Rate = 1, 3, 5 represent dilated convolutions with Rate values ​​of 1, 3, and 5, respectively. "Crop" indicates cropping the image at the top left corner by 1 / 16 of its total size. The structure of the decoding layer is as follows... Figure 6 As shown, the model consists of a 2x upsampling layer, a feature concatenation layer, and two 3x3 convolutional layers. Each convolutional layer is followed by ReLU and BN. The overall process is as follows: the high-magnification image first passes through multiple dilated convolutional and average pooling layers to extract features from its various receptive fields. These features are then concatenated across channels, and a 1x1 convolutional layer is used to fuse the extracted features. The features are then cropped according to their relative positions and input into the channel attention layer, where different weights are assigned to each channel to filter the features extracted by different convolutions. The resulting features are then input into the decoding layer. In the decoding layer, the features of the high- and low-magnification images are first concatenated and then fused using two convolutional layers. After that, the high- and low-magnification images are upsampled and then concatenated with the features from the previous layer of the high-magnification image. These features are then input into subsequent convolutional layers to extract features, and finally input into the subsequent layers of the entire model.

[0073] To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides specific product or related technology application examples of the technical solution claimed.

[0074] To address the problems of time-consuming and labor-intensive manual examination of pathological images, which requires a high level of professional knowledge from relevant personnel, this invention provides an automatic pathological image segmentation method based on deep learning, thus solving the above problems.

[0075] To address the low accuracy of automatic pathological image segmentation methods, this invention provides a pathological image segmentation method based on multi-view feature fusion. This method optimizes the feature fusion scheme through soft feature alignment, incorporating edge-adjacent features from low-magnification images to provide more reference information for image segmentation. Furthermore, this invention employs various dilated convolutions to acquire multiple receptive field features from low-magnification images and introduces a channel attention mechanism to filter these features, enabling the model to combine appropriate receptive field features for discrimination, thereby enhancing the model's accuracy.

[0076] This invention has conducted extensive comparative and ablation experiments on large-scale pathological image datasets. The experiments demonstrate that the above scheme improves both Dice and mIoU values ​​compared to traditional models, and also has a certain degree of interpretability in the visualization of segmentation.

[0077] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes them in conjunction with the data, charts and other information of the experimental process.

[0078] 1. Data Introduction

[0079] The dataset used in this invention comprises 215 colorectal cancer pathological images stained with hematoxylin and eosin (H&E), all of which have been manually annotated by pathologists. Each pathological image in this dataset can be observed at different magnification ratios of 16x, 4x, and 0x. This invention uses the Otsu method to remove non-tissue regions from the pathological images, and then employs a sliding window approach to crop each processed WSI image into 256*256 pixel slices. This invention yielded a total of 151,228 slices at 16x magnification, along with their corresponding annotated images and images at 4x magnification.

[0080] 2. Evaluation Indicators

[0081] To quantitatively evaluate the performance of this invention, embodiments of this invention use two commonly used metrics in image segmentation: mIoU and Dice.

[0082]

[0083] In the formula, mIoU: the ratio of the intersection and union of the sets of true values ​​and predicted values, where M is the number of classes, TP c FP c and FN c These represent the number of true positives, false positives, and false negatives for category C.

[0084]

[0085] The Dice coefficient is a set similarity metric, typically used to calculate the similarity between two samples. The value ranges from [0, 1]. True Positive means the sample is considered positive and is indeed positive; True Negative means the sample is considered negative and is indeed negative; False Positive means the sample is considered positive but is actually negative; False Negative means the sample is considered negative but is actually positive.

[0086] 3. Experimental Comparison

[0087] In this embodiment of the invention, the original dataset is randomly shuffled, and then divided into training, validation, and test sets in a 7:2:1 ratio. The following experimental results are all based on data from the test set. The backbone is a UNet with ResNet. The experiments in this invention use the Adam optimizer with a learning rate of 0.0001, weight decay of 1e-6, a batch size of 16, an early stopping rate of 10, and a random seed of 7.

[0088] The UNet method uses only high-magnification images at a 16x magnification. Gu et al.'s method uses both 4x and 16x images as input, but it employs a multi-encoder, single-decoder structure. The 4x low-magnification image is concatenated directly with the 16x image without a decoding layer, and then used in conjunction with the 16x image's decoding layer. Tokunaga et al.'s method involves multiple encoding and decoding, but the feature maps are only concatenated in the last layer. Ho et al.'s method uses a multi-level, multi-encoder, multi-decoder structure. The 16x image in each layer first passes through the decoding layer and is then concatenated with the feature map of the 4x magnification model before entering the model's decoding layer. This invention builds upon Ho et al.'s method. Each layer employs multiple dilated convolutions to extract features from the multiple receptive fields of the low-magnification image, and uses a different feature soft alignment method than the methods mentioned above, thereby improving model performance. The results of the method provided in this embodiment are shown in Table 1. It can be seen that compared to traditional methods, the Dice value is improved by 0.419%, and mIoU is improved by 0.744%.

[0089] Table 1 Comparison with traditional methods

[0090]

[0091] In addition, this invention also conducted various ablation experiments to demonstrate the effectiveness of the added module. The results are shown in Table 2. First, this invention verified the effect of adding the module only in a single layer (when just entering the decoding layer) and adding it in multiple layers. It can be seen that when adding it in multiple layers, information from different levels can be extracted, thereby increasing the accuracy of the model. The Dice value increased by 0.642%, and the mIoU increased by 1.134%.

[0092] The embodiments of the present invention also compared the feature extraction with and without multi-level dilated convolutions. It can be seen that compared with the method without dilation, the method provided by the embodiments of the present invention improves the Dice value by 0.354% and the mIoU by 0.628%.

[0093] In addition, the embodiments of the present invention also verified the effect of the soft alignment module of the present invention. Compared with the previous method of directly cropping according to relative position, the method provided by the embodiments of the present invention improves Dice by 0.28% and mIoU by 0.488%.

[0094] Finally, the embodiments of the present invention verified the effect of the channel attention module. Compared with the method of performing multiple convolutions on low-magnification feature maps without adding the channel attention mechanism, the addition of the channel attention mechanism improved Dice by 0.065% and mIoU by 0.116%.

[0095] Table 2 Ablation Experiment

[0096]

[0097]

[0098] 4. Comparison of segmentation effect images

[0099] The embodiments of this invention also compare the segmentation results with those of other methods, and the results are as follows: Figure 7 As shown, traditional single-ratio networks (Unet single-ratio) only use high-ratio cell structures for segmentation, resulting in an overemphasis on detail. This is evident in their sensitivity to color in high-ratio images, leading to a tendency to segment based on color differences. In contrast, multi-ratio networks, by incorporating tissue-level information from low-ratio images, achieve better recognition. The network of this invention, by introducing multi-domain information and soft alignment from high-ratio images, allows the model to extract more surrounding information and achieve edge smoothing. This results in better discrimination at image edges, and smoothing of uneven segmentation edges.

[0100] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for segmenting pathological images, characterized in that, The pathological image segmentation method includes: Obtain pathological image slice data with lesion annotation; feed the pathological image slice data with lesion annotation into the model for training to obtain the pathological image segmentation model; input the target pathological image slice that needs lesion annotation into the pathological image segmentation model to obtain the image with lesion annotation. The method uses soft feature alignment for feature extraction. By introducing multiple dilated convolutions, it extracts adjacent features of low-magnification image edges and then performs feature alignment, providing more reference information for image edge pixels, thereby improving the accuracy of model segmentation and also smoothing edge noise points. To address the narrow receptive field of low-magnification image feature extraction methods in multi-scale image fusion, this paper proposes to acquire multiple receptive field features of low-magnification images by employing various dilated convolutions and introducing a channel attention mechanism to filter multiple receptive field features, enabling the model to combine appropriate adjacent features for discrimination. The feature soft alignment includes: extracting features from low-magnification feature maps by introducing multiple dilated convolutions with different receptive fields, and then cropping the low-magnification image based on spatial relative positions; the low-magnification features are extracted and mapped to feature maps through convolutional layers, and three dilated convolutions with different rate rates (1, 3, 5) are designed to obtain information from different receptive fields, enhancing the ability to extract edge features of subsequent feature maps, where each convolution is 3x3 in size, followed by ReLU and BN; it also includes an average global pooling layer, followed by ReLU activation and BN, used to compress the channels of the feature map to 1 to extract global information of the image; the extracted feature maps are then stitched together on the channels, and then cropped after relative position alignment.

2. The pathological image segmentation method as described in claim 1, characterized in that, The pathological image segmentation model adopts the UNet structure. The input of the pathological image segmentation model is a high-magnification slice and a low-magnification slice of a pathological image with a size of 256*256*3. Each input image has features and receptive fields with different spatial resolutions.

3. The pathological image segmentation method as described in claim 1, characterized in that, The pathological image segmentation method includes the following steps: Step 1: Use an encoder to input high-magnification and low-magnification slices of the pathological image into the pathological image segmentation model for encoding; Step 2: The high-magnification slice features and low-magnification slice features of the coding layer, as well as the features of the fourth coding layer, are input into the multi-view feature fusion module using the decoder. Step 3: After fusing the two input feature maps with different magnifications using the multi-view feature fusion module, the feature maps are concatenated with the upper-layer features and then the fused features are input into the subsequent decoding layer.

4. The pathological image segmentation method as described in claim 3, characterized in that, Each decoding layer consists of an upsampling layer and two convolutional layers. The upsampling layer enlarges the original feature map by two times. Each of the two convolutional layers consists of a 3x3 convolutional layer, a BN layer, and a ReLU activation function. The feature map is then fed into the next decoding layer, and the same operation is performed in each layer. Finally, the fifth layer is connected to a segmentation layer, which outputs a 256*256 pixel segmentation map with predicted values ​​of 0 and 1. Finally, the segmentation map and the ground truth labels are used to calculate the DICE loss and train the model. The formula for calculating the dice loss is as follows: In the formula, the Dice coefficient is a set similarity metric used to calculate the similarity between two samples, with a value range of [0, 1]; True Positive, is judged as a positive sample and is indeed a positive sample; True Negative, is judged as a negative sample and is indeed a negative sample; False Positive, is judged as a positive sample but is actually a negative sample; False Negative, is judged as a negative sample but is actually a positive sample.

5. The pathological image segmentation method as described in claim 3, characterized in that, The multi-view feature fusion module uses various dilated convolutions to extract diverse features from low-magnification images, and then fuses them to achieve soft feature alignment and multi-view feature extraction.

6. The pathological image segmentation method as described in claim 5, characterized in that, The multi-view feature extraction includes: Multiple convolutions are used to extract features from low-magnification images. In the multi-view feature extraction structure, L and H represent low-magnification and high-magnification feature maps, respectively. The spur represents the relationship between upper and lower layers in the UNet structure. Rate = 1, 3, 5 represent dilated convolutions with Rate values ​​of 1, 3, and 5, respectively. Crop means cropping the image at the top left corner of the original image at a ratio of 1 / 16 of the total size. The decoding layer contains a 2x upsampling layer, a feature concatenation layer, and two 3x3 convolutional layers. Each convolutional layer is followed by ReLU and BN after each convolution. High-magnification images are processed through multiple dilated convolutional and average pooling layers to extract features from various receptive fields. These features are then concatenated across channels. A 1x1 convolutional layer is used to fuse the extracted features, which are then cropped according to their relative positions. The resulting data is then fed into a channel attention layer, where different weights are assigned to each channel. Features extracted from different convolutional layers are filtered and fed into the decoding layer. In the decoding layer, high- and low-magnification image features are concatenated and fused using two convolutional layers. The high- and low-magnification images are then upsampled and concatenated with features from the previous layer at high magnification. This upsampling is then fed into subsequent convolutional layers to extract features, and finally, the data is fed into the subsequent layers of the entire model.

7. A pathological image segmentation system implementing the pathological image segmentation method according to any one of claims 1 to 6, characterized in that, The pathological image segmentation system includes an encoder, a multi-view feature fusion module, and a decoder; The encoder is used to input high-magnification and low-magnification slices of pathological images into the model for encoding. The encoder adopts the ResNet50 structure, which contains a total of 5 stages. Each stage contains various convolutional, BN, activation layers and residual structures. The feature maps of the five different stages of ResNet50 are obtained as features of the five encoding layers of UNet. The decoder is used to input the high-magnification slice features and low-magnification slice features of the coding layer, as well as the features of the coding layer of the fourth layer, into the multi-view feature fusion module, starting from the fifth coding layer. The multi-view feature fusion module includes a feature soft alignment module and a multi-view convolution module, which are used to fuse the two input feature maps with different magnifications and then concatenate them with the upper-layer features on the channel side, and then input the fused features into the subsequent decoding layer.

8. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the pathological image segmentation method as described in any one of claims 1 to 6.

9. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the pathological image segmentation system as described in claim 7.

Images