A pathological image segmentation method and device
Through weak supervision and learning and twin network technology, the pathological images are divided, and the problem of low pathological diagnosis and subjectivity of pathological images in existing technologies has been solved, and automatic positioning and efficient diagnosis of cancerous areas have been realized.
Patent Information
- Application Number
- CN202210264678.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-17
- Publication Date
- 2025-01-28
- Estimated Expiration
- 2042-03-17
AI Technical Summary
The prior art has problems such as long time-consuming, subjective factors and high risk of misdiagnosis in pathological image diagnosis, which is difficult to effectively assist pathological diagnosis and improve diagnostic efficiency and objectivity.
Through weak supervision and learning methods, use image -grade labels with less labeling volume, divide the pathological images, generate class activation diagrams and regularize through twin networks, convert it into pseudo -pixel -level labels, optimize semantic segmentation networks to achieve pathological images to achieve pathological images. Quickly and accurate division.
The automatic positioning of the cancerous region in the pathological image is achieved, the diagnostic efficiency and the objectivity of the results are improved, and the subjectivity of artificial diagnosis and the risk of misdiagnosis are reduced.
Smart Images
Figure CN114565605B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning and image processing, and in particular to a pathological image segmentation method and device. Background Art
[0002] In recent years, with the rapid development of medical technology, the number of patient data in hospitals has increased dramatically, and a large amount of medical imaging data needs to be processed urgently. The current manual diagnosis method has a large workload for pathologists, which makes it difficult for pathologists to pay more attention to the patient's condition and perform surgery; moreover, due to the subjective factors in manual diagnosis, there will be differences of opinion between doctors, and misdiagnosis may also occur. Therefore, how to better utilize some information of histopathology, better assist diagnosis, improve the efficiency of pathological diagnosis, and make the diagnosis results more objective is a technical problem that needs to be solved urgently in the current medical field. Summary of the invention
[0003] The present application provides a pathological image segmentation method and device, which can use tissue pathology images with fewer image-level labels to quickly and accurately segment them through weakly supervised learning, thereby automatically locating cancerous areas in pathological images.
[0004] In a first aspect, the present application provides a pathological image segmentation method, the method comprising:
[0005] Acquire a pathological image of the tissue, preprocess the pathological image to obtain a preprocessed image, and divide the preprocessed image into a training set and a validation set according to a preset ratio using the preprocessed image as a sample;
[0006] Inputting the training set samples into a priori classification network to obtain predicted categories of the samples; calculating a binary cross entropy loss based on the predicted categories and the true categories of the training set samples as a priori classification loss;
[0007] Generate a class activation map according to the prior classification network, and regularize the class activation map using a twin network to obtain a corrected class activation map;
[0008] Converting the modified class activation map into a pseudo pixel-level label according to a preset conversion rule;
[0009] Inputting the training set samples into a preset semantic segmentation network to obtain the segmentation results of the training set samples;
[0010] The segmentation result and the pseudo pixel-level label are used to calculate the Dice loss to optimize the semantic segmentation network, and the pathological image is segmented by the optimized semantic segmentation network.
[0011] Furthermore, the predicted categories include negative and positive categories.
[0012] Further, the pathological image is preprocessed to obtain a preprocessed image, including:
[0013] The pathological image is randomly flipped horizontally and vertically according to a preset probability value to obtain a flipped preprocessed image;
[0014] as well as,
[0015] The pathological image is randomly rotated according to a preset angle to obtain a rotated preprocessed image.
[0016] Furthermore, the prior classification network is a ResNet 50 network, which specifically includes: 2 ordinary convolution residual modules, 2 hole convolution residual modules and a classification module:
[0017] Among them, the first ordinary convolution residual module in the ordinary convolution residual module is composed of 3 ordinary convolution residual components; the second ordinary convolution residual module in the ordinary convolution residual module is composed of 4 ordinary convolution residual components; wherein each of the ordinary convolution residual components includes 2 convolution layers and 1 maximum pooling layer, the size of the convolution kernel of the ordinary convolution layer is 3×3, the step size is 1, and the padding is 1; the step size of the maximum pooling layer is 2, and the size of the pooling layer kernel is 2×2;
[0018] Among them, the first atrous convolution residual module in the atrous convolution residual module is composed of 6 atrous convolution residual components; the second atrous convolution residual module in the atrous convolution residual module is composed of 3 atrous convolution residual components; wherein each of the atrous convolution residual components includes 2 atrous convolution layers and 1 maximum pooling layer, the convolution kernel size of the atrous convolution layer is 3×3, the step size is 1, and the padding is 1; the step size of the maximum pooling layer is 2, the size of the pooling layer kernel is 2×2, and the atrous rates are 2 and 4 respectively;
[0019] The classification module includes 1 global average pooling layer and 1 linear classification layer. The size of the training set sample feature map output by the global average pooling layer is 1×1, and the output node of the linear classification layer is 2, indicating that the prediction is positive or negative.
[0020] Furthermore, a binary cross entropy loss is calculated according to the predicted category and the true category of the training set sample as a priori classification loss, including:
[0021] The prior classification loss is calculated using the following formula:
[0022]
[0023] in,
[0024] Among them, L cls represents the prior classification loss, y i represents the category of the true label, Represents the category of the predicted label, and N represents the total number of training samples.
[0025] Further, generating a class activation map according to the prior classification network includes:
[0026] The value G of the k-th dimension of the vector after global average pooling k The calculation method is:
[0027]
[0028] Among them, (x, y) represents the coordinates of a pixel point on the training set sample feature map, H and W represent the height and width of the training set sample feature map respectively;
[0029] CAM activation map of category c c (x, y) is the multiplication of the point of each channel on the feature map of the training set sample and the weight of the corresponding category of the fully connected layer, calculated as:
[0030]
[0031] in, represents the weight of the kth feature vector to the cth category, also representing G k The importance of the final classification into category c.
[0032] Further, the twin network includes: a first branch and a second branch;
[0033] The first branch is used to perform data transformation on the class activation map first, and then perform feature extraction to obtain a first class activation map;
[0034] The second branch is used to first perform feature extraction on the class activation map and then perform data transformation to obtain a second class activation map.
[0035] Furthermore, the class activation map is regularized using a twin network to obtain a corrected class activation map, including:
[0036] The regularization loss is calculated for the first type of activation map and the second type of activation map generated as follows:
[0037] L eq =|N[f(I)]-f[N(I)]|2;
[0038] Where I represents the input class activation map, N represents the prior classification network, f represents the data transformation operation, N[f(I)] represents the first branch processing process, and f[N(I)] represents the second branch processing process;
[0039] The regularization loss adopts the bi-norm of the class activation map, specifically the distance from the difference vector of the class activation map to the origin in the vector space.
[0040] Further, converting the modified class activation map into a pseudo pixel-level label according to a preset conversion rule includes:
[0041] The area in the class activation map that is higher than the set threshold is taken as a pseudo label of 1, and the area in the class activation map that is lower than the set threshold is taken as a pseudo label of 0;
[0042] Specifically, the expression of the pseudo label is as follows:
[0043]
[0044] Among them, y i represents the pseudo label of the i-th pixel, threshold represents the preset threshold, CAM i Represents the heat value of the i-th pixel in the class activation map;
[0045] Accordingly, the segmentation result and the pseudo pixel-level label are used to calculate the Dice loss, including:
[0046] The segmentation loss is calculated as follows:
[0047]
[0048] Among them, p i represents the predicted value of the i-th pixel, L dice represents the segmentation loss.
[0049] In a second aspect, the present application provides a pathological image segmentation device, the device comprising:
[0050] A preprocessed image acquisition unit, used to acquire a pathological image of a tissue, preprocess the pathological image to obtain a preprocessed image, and divide the preprocessed image into a training set and a validation set according to a preset ratio using the preprocessed image as a sample;
[0051] A priori classification loss calculation unit is used to input the training set samples into the priori classification network to obtain the predicted category of the samples; and calculate the binary cross entropy loss according to the predicted category and the true category of the training set samples as the priori classification loss;
[0052] A modified class activation map determining unit, used to generate a class activation map according to the prior classification network, and regularize the class activation map using a twin network to obtain a modified class activation map;
[0053] A pseudo pixel level label conversion unit, used to convert the correction class activation map into a pseudo pixel level label according to a preset conversion rule;
[0054] A segmentation result determination unit, used for inputting the training set samples into a preset semantic segmentation network to obtain the segmentation results of the training set samples;
[0055] The semantic segmentation network optimization unit is used to calculate the Dice loss between the segmentation result and the pseudo pixel-level label to optimize the semantic segmentation network, and perform pathological image segmentation through the optimized semantic segmentation network.
[0056] The technical solution provided in this application can obtain an objective and accurate pathological image segmentation method through sample preprocessing and model training using a priori classification network and twin network, thereby achieving the purpose of quickly and accurately completing medical image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 is a flow chart of a pathological image segmentation method provided in Example 1 of the present application;
[0058] Figure 2 is a schematic diagram of a priori classification network provided in Example 1 of the present application;
[0059] Figure 3 This is a schematic diagram of the overall framework of weakly supervised segmentation with pseudo-label correction provided in Example 1 of the present application;
[0060] Figure 4 is a schematic diagram of a semantic segmentation network provided in Example 2 of the present application;
[0061] Figure 5 This is a structural block diagram of a pathological image segmentation device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION
[0062] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.
[0063] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0064] Embodiment 1
[0065] Figure 1 This is a flow chart of the pathological image segmentation method provided in the first embodiment of the present application. This embodiment can be applicable to the scenario of pathological image segmentation. The method can be executed by the pathological image segmentation device provided in the embodiment of the present application. The device can be implemented by software and / or hardware and can be integrated into an electronic device.
[0066] like Figure 1 As shown, the pathological image segmentation method includes:
[0067] S110, acquiring a pathological image of a tissue, preprocessing the pathological image to obtain a preprocessed image, and using the preprocessed image as a sample to divide the sample into a training set and a validation set according to a preset ratio.
[0068] Among them, in order to enable the prior classification network and the segmentation network to process various types of tissue pathology images and reduce the impact of tissue pathology images with different distributions generated by different machines, mechanisms, and staining methods, the present invention performs data preprocessing on the tissue pathology images.
[0069] In this embodiment, optionally, preprocessing the pathological image to obtain a preprocessed image includes:
[0070] The pathological image is randomly flipped horizontally and vertically according to a preset probability value to obtain a flipped preprocessed image;
[0071] as well as,
[0072] The pathological image is randomly rotated according to a preset angle to obtain a rotated preprocessed image.
[0073] The specific implementation of preprocessing here is:
[0074] Flipping along the x-axis with a probability of 0.5 is called vertical flipping, and flipping along the y-axis with a probability of 0.5 is called horizontal flipping. The flipped data is then rotated at a random angle in a counterclockwise direction. The degree of rotation is one of the following four: 0°, 90°, 180°, and 270°. If the degree of rotation is 0, no rotation occurs.
[0075] The method of standardizing images is Z-Score standardization, which converts the distribution of tissue pathology images into a standard normal distribution and accelerates the convergence of the network. The specific calculation method is as follows:
[0076]
[0077] Among them, x is the input image, x' is the output image, μ is the mean of the pixel values of the entire image, σ is the variance of the pixel values of the entire image, and the obtained x ′ It follows a standard normal distribution.
[0078] The preprocessed data set is divided into training set and validation set in a ratio of 8:2.
[0079] S120, inputting the training set samples into a priori classification network to obtain predicted categories of the samples; and calculating a binary cross entropy loss according to the predicted categories and the true categories of the training set samples as the priori classification loss.
[0080] Prior classification networks can be incorporating prior knowledge into the design of neural networks, which limits the application of neural networks to solving specific problems based on some knowledge of interest. If there is prior information and invariance, it should be attached to the network design, so that this information does not need to be learned and the network design is simplified.
[0081] In this solution, optionally, the prior classification network is a ResNet 50 network, and the ResNet 50 network specifically includes: 2 ordinary convolution residual modules, 2 hole convolution residual modules and a classification module:
[0082] Among them, the first ordinary convolution residual module in the ordinary convolution residual module is composed of 3 ordinary convolution residual components; the second ordinary convolution residual module in the ordinary convolution residual module is composed of 4 ordinary convolution residual components; wherein each of the ordinary convolution residual components includes 2 convolution layers and 1 maximum pooling layer, the size of the convolution kernel of the ordinary convolution layer is 3×3, the step size is 1, and the padding is 1; the step size of the maximum pooling layer is 2, and the size of the pooling layer kernel is 2×2;
[0083] Among them, the first atrous convolution residual module in the atrous convolution residual module is composed of 6 atrous convolution residual components; the second atrous convolution residual module in the atrous convolution residual module is composed of 3 atrous convolution residual components; wherein each of the atrous convolution residual components includes 2 atrous convolution layers and 1 maximum pooling layer, the convolution kernel size of the atrous convolution layer is 3×3, the step size is 1, and the padding is 1; the step size of the maximum pooling layer is 2, the size of the pooling layer kernel is 2×2, and the atrous rates are 2 and 4 respectively;
[0084] The classification module includes 1 global average pooling layer and 1 linear classification layer. The size of the training set sample feature map output by the global average pooling layer is 1×1, and the output node of the linear classification layer is 2, indicating that the prediction is positive or negative.
[0085] Resnet is the abbreviation of Residual Network. This series of networks is widely used in fields such as target classification and as part of the backbone classic neural network for computer vision tasks. Typical networks include resnet50, resnet101, etc.
[0086] Figure 2 is a schematic diagram of a priori classification network provided in Example 1 of the present application, such as Figure 2 As shown in the figure, the improved ResNet50 network includes two ordinary convolution residual modules for extracting features, two hole convolution residual modules for expanding the receptive field, and a classification module for outputting the category of the image and generating a class activation map.
[0087] In this embodiment, optionally, the predicted category includes negative and positive categories.
[0088] It can be understood that the negative category means the absence of pathological abnormalities, and the positive category means the presence of pathological abnormalities.
[0089] In a feasible embodiment, optionally, a binary cross entropy loss is calculated according to the predicted category and the true category of the training set sample as a priori classification loss, including:
[0090] The prior classification loss is calculated using the following formula:
[0091]
[0092] in,
[0093] Among them, L cls represents the prior classification loss, y i represents the category of the true label, Represents the category of the predicted label, and N represents the total number of training samples.
[0094] Through such a setting, this scheme can determine the optimization method of the prior classification network based on the prior classification loss.
[0095] S130, generating a class activation map according to the prior classification network, and regularizing the class activation map using a twin network to obtain a modified class activation map.
[0096] Among them, the feature map output by the prior classification network can be used to generate the class activation map.
[0097] In this solution, optionally, generating a class activation map according to the prior classification network includes:
[0098] The value G of the k-th dimension of the vector after global average pooling k The calculation method is:
[0099]
[0100] Among them, (x, y) represents the coordinates of a pixel point on the training set sample feature map, H and W represent the height and width of the training set sample feature map respectively;
[0101] CAM activation map of category c c (x, y) is the multiplication of the point of each channel on the feature map of the training set sample and the weight of the corresponding category of the fully connected layer, calculated as:
[0102]
[0103] in, represents the weight of the kth feature vector to the cth category, also representing G k The importance of the final classification into category c.
[0104] Specifically, for tissue pathology images, only the activation heat map of the positive category needs to be considered. Therefore, after the image is input into the prior classification network, the feature map output by the prior classification network is obtained, and then the feature map is globally averaged pooled. The obtained feature vector is fully connected with the linear layer to obtain the weight of the feature vector to the positive category in the linear layer, and multiplied with the feature map to obtain the activation map of the positive category.
[0105] Wherein, the twin network includes: a first branch and a second branch;
[0106] The first branch is used to perform data transformation on the class activation map first, and then perform feature extraction to obtain a first class activation map;
[0107] The second branch is used to first perform feature extraction on the class activation map and then perform data transformation to obtain a second class activation map.
[0108] On the basis of the above solution, optionally, the class activation map is regularized using a twin network to obtain a corrected class activation map, including:
[0109] The regularization loss is calculated for the first type of activation map and the second type of activation map generated as follows:
[0110] L eq =|N[f(I)]-f[N(I)]|2;
[0111] Where I represents the input class activation map, N represents the prior classification network, f represents the data transformation operation, N[f(I)] represents the first branch processing process, and f[N(I)] represents the second branch processing process;
[0112] The regularization loss adopts the bi-norm of the class activation map, specifically the distance from the difference vector of the class activation map to the origin in the vector space.
[0113] N[f(I)] means that the image is first subjected to data transformation and then input into the prior classification network. f[N(I)] means that the image is first input into the prior classification network and then subjected to data transformation. The regularization loss uses the binary norm of the activation map, that is, the distance from the difference vector of the activation map to the origin in the vector space.
[0114] Figure 3 : is a schematic diagram of the overall framework of weakly supervised segmentation with pseudo-label correction provided in Example 1 of the present application, such as Figure 3 As shown, the feature extraction form of the twin network includes two branches, the upper and lower branches, where the feature extraction network of each branch is a priori classification network. In one branch of the twin network, the image is first transformed and then input into the priori classification network. In the other branch of the twin network, it is first input into the feature network and then the feature map is transformed. The activation maps of the positive categories generated by the two branches are respectively regularized with the bi-norm, that is, the activation maps are subtracted, and then the distance from the difference vector to the origin in the vector space is calculated as the regularization loss. This operation is a correction to the original class activation map, so that the class activation map obtained in the priori classification network has similar properties to the feature map of the segmentation network, and the corrected class activation map is more robust and of higher quality.
[0115] S140, converting the modified class activation map into a pseudo pixel-level label according to a preset conversion rule.
[0116] The segmentation network may adopt any semantic segmentation network. In this embodiment, a pyramid scene parsing network PSPNet is adopted.
[0117] In this embodiment, optionally, converting the modified class activation map into a pseudo pixel-level label according to a preset conversion rule includes:
[0118] The area in the class activation map that is higher than the set threshold is taken as a pseudo label of 1, and the area in the class activation map that is lower than the set threshold is taken as a pseudo label of 0;
[0119] Specifically, the expression of the pseudo label is as follows:
[0120]
[0121] Among them, y i represents the pseudo label of the i-th pixel, threshold represents the preset threshold, CAM i Represents the heat value of the i-th pixel in the class activation map.
[0122] The pyramid scene parsing network is used to automatically segment tissue pathology images. The pyramid scene parsing network includes four residual modules, a pyramid pooling module and an aggregation module. The residual modules contain 3, 4, 6, and 3 residual components respectively. Each residual component contains 2 3×3 convolutional layers with a step size of 1 and a padding of 1, as well as a maximum pooling layer. The step size of the maximum pooling layer is 2 and the kernel size is 2×2; the pyramid pooling layer contains 4 branches, each of which contains an adaptive average pooling layer and a convolutional layer. After the feature map passes through the four branches, the sizes of the feature maps are 1×1, 2×2, 3×3 and 6×6 respectively; the aggregation module upsamples the four feature maps to the input size, concatenates them with the input, and finally passes through a convolutional layer to output pixel-level segmentation results.
[0123] S150, inputting the training set samples into a preset semantic segmentation network to obtain segmentation results of the training set samples.
[0124] Image semantic segmentation is an important part of image processing and machine vision technology in image understanding, and is also an important branch in the field of AI. Semantic segmentation is to classify each pixel in the image and determine the category of each point (such as background, person or car, etc.), so as to divide the area. At present, semantic segmentation has been widely used in scenes such as autonomous driving and drone landing point determination.
[0125] S160, calculating the Dice loss of the segmentation result and the pseudo pixel-level label to optimize the semantic segmentation network, and performing pathological image segmentation through the optimized semantic segmentation network.
[0126] In this embodiment, specifically, calculating the Dice loss between the segmentation result and the pseudo pixel-level label includes:
[0127] The segmentation loss is calculated as follows:
[0128]
[0129] Among them, p i represents the predicted value of the i-th pixel, L dice represents the segmentation loss.
[0130] The method of the present invention can achieve a Dice coefficient of 65.32% on the DigestPath dataset using data training with only image-level labels, which is 0.5% higher than the uncorrected class activation map. The activation map generated after regularization can well locate the position of the cancerous area, and is very detailed in depicting the boundaries and contours, which is very beneficial for subsequent segmentation tasks. Therefore, in the case where the pathology dataset does not have pixel-level labels, the classification network can be trained to obtain a high-quality CAM map. Among them, DigestPath is a colorectal cancer pathology image dataset. The present application includes but is not limited to this dataset and can support all pathology image datasets. Dice is a universal detection measurement indicator. Generally speaking, the Dice coefficient is obtained by comprehensively averaging the intersection-union ratios of all categories.
[0131] The technical solution provided in the embodiment of the present application generates a class activation map through a priori classification network, corrects the class activation heat map through a twin network, obtains pseudo labels and then inputs them into a segmentation network to obtain pixel-level segmentation results, thereby obtaining a weakly supervised tissue pathology image segmentation method. It can use tissue pathology images with image-level labels with a small number of annotations, and through a weakly supervised learning method, it can quickly and accurately segment them, thereby completing the automatic positioning of cancerous areas in pathology images.
[0132] Embodiment 2
[0133] This embodiment is a preferred embodiment provided based on the above embodiments. Figure 4 Schematic diagram of the semantic segmentation network provided in Example 2 of the present application. The implementation example of the weakly supervised histopathological image segmentation method based on pseudo-label correction provided in this solution includes the following steps:
[0134] Step 1: Read the tissue pathology image data, flip it along the x-axis with a probability of 0.5, called vertical flip, and flip it along the y-axis with a probability of 0.5, called horizontal flip. Then rotate the flipped data at a random angle. The direction of rotation is counterclockwise, and the degree of rotation is one of the following four: 0°, 90°, 180°, 270°. If the degree of rotation is 0, then no rotation occurs.
[0135] The method of standardizing images is Z-Score standardization, which converts the distribution of tissue pathology images into a standard normal distribution and accelerates the convergence of the network. The specific calculation method is as follows:
[0136]
[0137] Among them, x is the input image, x' is the output image, μ is the mean of the pixel values of the entire image, σ is the variance of the pixel values of the entire image, and the obtained x' obeys the standard normal distribution.
[0138] The preprocessed data set is divided into training set and validation set in a ratio of 8:2.
[0139] Step 2: Input the data in step 1 into the prior classification network. The prior classification network uses an improved ResNet 50 network including two ordinary convolution residual modules for feature extraction, two hole convolution residual modules for expanding the receptive field, and a classification module for outputting the category of the image and generating a class activation map.
[0140] Among them, the ordinary convolution residual module is composed of 3 and 4 ordinary convolution residual components respectively, where each ordinary convolution residual component includes 2 convolution layers and 1 maximum pooling layer, the size of the convolution kernel is 3×3, the step size is 1, the padding is 1, the step size of the maximum pooling layer is 2, and the kernel size is 2×2;
[0141] The dilated convolution residual module consists of 6 and 3 dilated convolution residual components respectively, where each dilated convolution residual component includes 2 dilated convolution layers and 1 maximum pooling layer. The size of the convolution kernel is 3×3, the stride is 1, and the padding is 1. The stride of the maximum pooling layer is 2, the kernel size is 2×2, and the dilation rates are 2 and 4 respectively.
[0142] The classification module includes a global average pooling layer and a linear classification layer. The size of the feature map output by the global average pooling layer is 1×1, and the output node of the linear classification layer is 2, indicating that the prediction is positive or negative;
[0143] The binary cross entropy loss is calculated based on the predicted category and the true category of the prior classification network as the prior classification loss. The calculation method is as follows:
[0144]
[0145]
[0146] Among them, L cls represents the prior classification loss, y i represents the category of the true label, Represents the category of the predicted label, and N represents the total number of training samples.
[0147] Step 3: For tissue pathology images, only the activation heat map of the positive category needs to be considered. Therefore, after the image is input into the prior classification network in step 2, the feature map output by the prior classification network is obtained, and then the feature map is globally averaged pooled. The value G of the kth dimension of the vector after global average pooling is k The calculation method is:
[0148]
[0149] Where (x, y) represents the coordinates of a pixel on the feature map, and H and W represent the height and width of the feature map respectively;
[0150] The obtained feature vector is fully connected with the linear layer to obtain the weight of the feature vector to the positive category in the linear layer, and multiplied with the feature map to obtain the activation map of the positive category.
[0151] The activation map CAM(x,y) of the positive category is the multiplication of the point of each channel on the feature map with the weight of the positive category of the fully connected layer, calculated as:
[0152] CAM(x,y)=∑ k w k ×f k (x,y);
[0153] where w k represents the weight of the kth feature vector to the positive category, also represents G k Importance of the final classification as positive.
[0154] Step 4: Use the twin network to correct the class activation map generated by the prior classification network, the overall framework of weak supervision segmentation corrected by pseudo-labels, the pseudo-label is the class activation map, and the feature extraction form of the twin network is used, which includes two branches, the upper and lower branches, where the feature extraction network of each branch is a prior classification network. In one branch of the twin network, the image is first transformed and then feature extracted. In the other branch of the twin network, feature extraction is performed first, and then the feature map is transformed. The class activation map is generated according to the method of step 4, and the regularization loss is calculated for the two generated class activation maps. The calculation method is as follows:
[0155] L eq =|N[f(I)]-f[N(I)]|2;
[0156] Where I represents the input image, N represents the prior classification network, f represents the data transformation operation, N[f(I)] represents that the image is first subjected to data transformation operation and then input into the prior classification network, f[N(I)] represents that the image is first input into the prior classification network and then subjected to data transformation operation, and the regularization loss adopts the bi-norm of the activation map, that is, the distance from the difference vector of the activation map to the origin in the vector space;
[0157] Step 5: Create a pseudo pixel-level label based on the modified class activation map generated in step 4. The area above the set threshold in the class activation map is taken as 1 as the pseudo label, and the area below the set threshold in the class activation map is taken as 0 as the pseudo label. The expression of the pseudo label is as follows:
[0158]
[0159] Among them, y i represents the pseudo label of the i-th pixel, threshold represents the preset threshold, CAM i Represents the heat value of the i-th pixel in the class activation map.
[0160] Step 6: Train the semantic segmentation network based on the pseudo pixel-level labels from step 5. The segmentation loss is calculated as follows:
[0161]
[0162] Among them, p i represents the predicted value of the i-th pixel, L dice represents the segmentation loss.
[0163] The semantic segmentation network structure uses a pyramid scene parsing network to automatically segment tissue pathology images. The pyramid scene parsing network includes four residual modules, a pyramid pooling module and an aggregation module. The residual modules contain 3, 4, 6, and 3 residual components respectively. Each residual component contains 2 3×3 convolutional layers with a step size of 1 and a padding of 1, as well as a maximum pooling layer. The step size of the maximum pooling layer is 2 and the kernel size is 2×2; the pyramid pooling layer contains 4 branches, each of which contains an adaptive average pooling layer and a convolutional layer. After the feature map passes through the four branches, the sizes of the feature maps are 1×1, 2×2, 3×3, and 6×6 respectively; the aggregation module upsamples the four feature maps to the input size, concatenates them with the input, and finally passes through a convolutional layer to output pixel-level segmentation results.
[0164] The method of the present invention can achieve a Dice coefficient of 65.32% on the DigestPath dataset using data training with only image-level labels, which is 0.5% higher than the uncorrected class activation map. The activation map generated after regularization can well locate the position of the cancerous area, and is very detailed in depicting the boundaries and contours, which is very beneficial for subsequent segmentation tasks. Therefore, in the case where the pathology dataset does not have pixel-level labels, the classification network can be trained to obtain a high-quality CAM map. Among them, DigestPath is a colorectal cancer pathology image dataset. The present application includes but is not limited to this dataset and can support all pathology image datasets. Dice is a universal detection measurement indicator. Generally speaking, the Dice coefficient is obtained by comprehensively averaging the intersection-union ratios of all categories.
[0165] The beneficial effect of the present invention is that: through the above technical solution, the present invention can use tissue pathology images with fewer image-level labels, and through weakly supervised learning, quickly and accurately segment them to complete the automatic positioning of cancerous areas in pathology images.
[0166] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.
[0167] Embodiment 3
[0168] Figure 5 This is a structural block diagram of a pathological image segmentation device provided in Embodiment 3 of the present invention. The device can execute the pathological image segmentation method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
[0169] The device may include:
[0170] A preprocessed image acquisition unit 510 is used to acquire a pathological image of a tissue, preprocess the pathological image to obtain a preprocessed image, and divide the preprocessed image into a training set and a validation set according to a preset ratio using the preprocessed image as a sample;
[0171] A priori classification loss calculation unit 520 is used to input the training set samples into the priori classification network to obtain the predicted category of the samples; and calculate the binary cross entropy loss according to the predicted category and the true category of the training set samples as the priori classification loss;
[0172] A modified class activation map determining unit 530 is used to generate a class activation map according to the prior classification network, and regularize the class activation map using a twin network to obtain a modified class activation map;
[0173] A pseudo pixel level label conversion unit 540, configured to convert the modified class activation map into a pseudo pixel level label according to a preset conversion rule;
[0174] The segmentation result determination unit 550 is used to input the training set samples into a preset semantic segmentation network to obtain the segmentation results of the training set samples;
[0175] The semantic segmentation network optimization unit 560 is used to calculate the Dice loss between the segmentation result and the pseudo pixel-level label to optimize the semantic segmentation network, and perform pathological image segmentation through the optimized semantic segmentation network.
[0176] The above-mentioned product can execute the pathological image segmentation method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.
[0177] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A pathological image segmentation method, characterized in that: include: Acquire a pathological image of the tissue, preprocess the pathological image to obtain a preprocessed image, and divide the preprocessed image into a training set and a validation set according to a preset ratio using the preprocessed image as a sample; Inputting the training set samples into a priori classification network to obtain predicted categories of the samples; Calculate a binary cross entropy loss based on the predicted category and the true category of the training set sample as a priori classification loss; Generate a class activation map according to the prior classification network, and regularize the class activation map using a twin network to obtain a corrected class activation map; Converting the modified class activation map into a pseudo pixel-level label according to a preset conversion rule; Inputting the training set samples into a preset semantic segmentation network to obtain the segmentation results of the training set samples; Calculating Dice loss between the segmentation result and the pseudo pixel-level label to optimize the semantic segmentation network, and performing pathological image segmentation through the optimized semantic segmentation network; The twin network includes: a first branch and a second branch; The first branch is used to perform data transformation on the class activation map first, and then perform feature extraction to obtain a first class activation map; The second branch is used to first perform feature extraction on the class activation map and then perform data transformation to obtain a second class activation map.
2. The method according to claim 1, characterized in that The predicted classes include negative and positive classes.
3. The method according to claim 1, characterized in that The pathological image is preprocessed to obtain a preprocessed image, including: The pathological image is randomly flipped horizontally and vertically according to a preset probability value to obtain a flipped preprocessed image; as well as, The pathological image is randomly rotated according to a preset angle to obtain a rotated preprocessed image.
4. The method according to claim 1, characterized in that: The prior classification network is a ResNet 50 network, which specifically includes: 2 ordinary convolution residual modules, 2 hole convolution residual modules and a classification module: Among them, the first ordinary convolution residual module in the ordinary convolution residual module is composed of 3 ordinary convolution residual components; the second ordinary convolution residual module in the ordinary convolution residual module is composed of 4 ordinary convolution residual components; wherein each of the ordinary convolution residual components includes 2 convolution layers and 1 maximum pooling layer, the size of the convolution kernel of the ordinary convolution layer is 3×3, the step size is 1, and the padding is 1; the step size of the maximum pooling layer is 2, and the size of the pooling layer kernel is 2×2; Among them, the first atrous convolution residual module in the atrous convolution residual module is composed of 6 atrous convolution residual components; the second atrous convolution residual module in the atrous convolution residual module is composed of 3 atrous convolution residual components; wherein each of the atrous convolution residual components includes 2 atrous convolution layers and 1 maximum pooling layer, the convolution kernel size of the atrous convolution layer is 3×3, the step size is 1, and the padding is 1; the step size of the maximum pooling layer is 2, the size of the pooling layer kernel is 2×2, and the atrous rates are 2 and 4 respectively; The classification module includes 1 global average pooling layer and 1 linear classification layer. The size of the training set sample feature map output by the global average pooling layer is 1×1, and the output node of the linear classification layer is 2, indicating that the prediction is positive or negative.
5. The method according to claim 1, characterized in that Calculating a binary cross entropy loss according to the predicted category and the true category of the training set sample as a priori classification loss includes: The prior classification loss is calculated using the following formula: in, Among them, L cls represents the prior classification loss, y i represents the category of the true label, Represents the category of the predicted label, and N represents the total number of training samples.
6. The method according to claim 4, characterized in that Generate a class activation map according to the prior classification network, including: The value G of the k-th dimension of the vector after global average pooling k The calculation method is: Where (x, y) represents the coordinates of a pixel on the training set sample feature map, H and W represent the height and width of the training set sample feature map respectively; CAM activation map of category c c (x, y) is the multiplication of the point of each channel on the feature map of the training set sample and the weight of the corresponding category of the fully connected layer, calculated as: in, represents the weight of the kth feature vector to the cth category, also representing G k The importance of the final classification into category c.
7. The method according to claim 1, characterized in that Regularizing the class activation map using a twin network to obtain a corrected class activation map includes: The regularization loss is calculated for the first type of activation map and the second type of activation map generated as follows: L eq =|N[f(I)]-f[N(I)]|2; Where I represents the input class activation map, N represents the prior classification network, f represents the data transformation operation, N[f(I)] represents the first branch processing process, and f[N(I)] represents the second branch processing process; The regularization loss adopts the bi-norm of the class activation map, specifically the distance from the difference vector of the class activation map to the origin in the vector space.
8. The method according to claim 1, characterized in that Converting the modified class activation map into a pseudo pixel-level label according to a preset conversion rule includes: The area in the class activation map that is higher than the set threshold is taken as a pseudo label of 1, and the area in the class activation map that is lower than the set threshold is taken as a pseudo label of 0; Specifically, the expression of the pseudo label is as follows: Among them, y i represents the pseudo label of the i-th pixel, threshold represents the preset threshold, CAM i Represents the heat value of the i-th pixel in the class activation map; Accordingly, the segmentation result and the pseudo pixel-level label are used to calculate the Dice loss, including: The segmentation loss is calculated as follows: Among them, p i represents the predicted value of the i-th pixel, L dice represents the segmentation loss.
9. A pathological image segmentation device, characterized in that: include: A preprocessed image acquisition unit, used to acquire a pathological image of a tissue, preprocess the pathological image to obtain a preprocessed image, and divide the preprocessed image into a training set and a validation set according to a preset ratio using the preprocessed image as a sample; A priori classification loss calculation unit, used for inputting the training set samples into the priori classification network to obtain the predicted category of the samples; Calculate a binary cross entropy loss based on the predicted category and the true category of the training set sample as a priori classification loss; A modified class activation map determining unit, used to generate a class activation map according to the prior classification network, and regularize the class activation map using a twin network to obtain a modified class activation map; A pseudo pixel level label conversion unit, used to convert the correction class activation map into a pseudo pixel level label according to a preset conversion rule; A segmentation result determination unit, used for inputting the training set samples into a preset semantic segmentation network to obtain the segmentation results of the training set samples; A semantic segmentation network optimization unit, used to calculate the Dice loss between the segmentation result and the pseudo pixel-level label to optimize the semantic segmentation network, and perform pathological image segmentation through the optimized semantic segmentation network; The twin network includes: a first branch and a second branch; The first branch is used to perform data transformation on the class activation map first, and then perform feature extraction to obtain a first class activation map; The second branch is used to first perform feature extraction on the class activation map and then perform data transformation to obtain a second class activation map.