Method and system for multi-component image segmentation in pathological images based on category separation

Abstract

The present invention discloses a method and system for multi-component segmentation of pathological images based on category separation, which relates to the technical field of image processing in computational pathology, including: acquiring a pathological image, inputting the pathological image into a pre-established category separation network model for processing, including a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module and a category image reconstruction decoder; the global feature extractor and the category-specific feature extractor respectively extract the global features and category-specific features of the pathological image, the category fusion module fuses the global features and category-specific features of the pathological image to obtain fused global features and category-specific features; the segmentation module performs segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image, thereby further improving the segmentation performance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing for computational pathology, and in particular to a method and system for multi-component segmentation of pathological images based on category separation. Background Art

[0002] In the field of pathology, microscopic identification of morphological features, molecular target expression, and tumor-infiltrating lymphocytes in pathological images is fundamental to disease diagnosis, treatment, and prognosis. Currently, with the rapid development of artificial intelligence (AI), increasing research is devoted to the development and clinical application of AI pathology models, in which accurate whole-slice image (WSI) segmentation is crucial. However, manually annotating these images is a tedious task, typically requiring a significant investment of time from pathologists with domain expertise.

[0003] U-Net is a highly efficient image segmentation model. It is a convolutional neural network (CNN) based on a decoder architecture and a skip connection mechanism, which minimizes the loss of spatial information, making it particularly valuable in medical imaging applications. To enhance the U-Net's ability to extract features at different stages, various improvements based on U-Net have emerged. These methods aggregate multi-stage features through skip connections to generate high-resolution segmentation maps, achieving excellent performance in medical image segmentation. However, due to the inherent limitations of convolutional operations, CNNs are unable to learn long-range relationships between pixels. Some methods attempt to address this issue by adding self-attention to the encoder, transferring salient features from the encoder to the decoder and suppressing irrelevant information to accurately reconstruct the segmentation map. While these methods alleviate this issue, CNN-based methods are still insufficient to capture long-range dependencies. Furthermore, pathological images are characterized by large local variations and uneven distribution of tissue components, making it particularly necessary to seek better methods for tissue component segmentation in the tumor microenvironment. Summary of the Invention

[0004] In order to address the deficiencies mentioned in the above background technology, the object of the present invention is to provide a method and system for multi-component segmentation of pathological images based on category separation.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a method for multi-component segmentation of pathological images based on category separation, the method comprising the following steps:

[0006] Acquiring a pathological image and inputting the pathological image into a pre-established category separation network model for processing, wherein the category separation network model includes a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module, and a category image reconstruction decoder;

[0007] The global feature extractor and the category-specific feature extractor extract global features and category-specific features of the pathological image respectively, and the category fusion module fuses the global features and category-specific features of the pathological image to obtain fused global features and category-specific features;

[0008] The segmentation module performs segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the pre-established category separation network model is obtained by jointly optimizing alignment loss, category intersection loss, segmentation loss function, contrast loss function and reconstruction loss.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the pathological image is obtained by constructing a pathological image training set, including pathological images I i and its corresponding multi-component category labels Represents the number of pathological images in the training set, and the global feature extractor E based on the U-Net encoder is constructed. g Extract pathological image I i Global characteristics of , by constructing M category-specific feature extractors based on U-Net encoders , extract pathological image I i Class-specific features of , M represents the number of component segmentation categories, Represents pixel multiplication.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: a process in which the category fusion module fuses the global features of the pathology image:

[0012] Will Send it to the category fusion module CF to get the fusion , and use KL divergence to global features and fused global features Alignment,The alignment loss is defined as:

[0013]

[0014] in, Indicates calculation of global features and fused global features The KL divergence between

[0015] The category fusion module consists of an intra-class transformation module, an inter-class transformation module, a concat layer, a linear layer, and a layer normalization layer. They are sent to the intra-class transformation module to obtain 、

[0016] , As Input1, with As Input2, it is sent to the inter-class transformation module to obtain the fused category-specific features. ,

[0017] Then, they are fed into the concat layer, the linear layer, and the layer normalization layer as Input2. As Input1, it is sent to the inter-class transformation module to obtain the fused global features. .

[0018] In combination with the first aspect, in some implementations of the first aspect, the method further includes: inputting the intra-class transformation module Flatten , then exchange the dimension Permute to , where C represents the channel, H represents the height, and W represents the width. After being fed into the linear layer, the position encoding of the learnable parameter PE is added. The process is expressed as

[0019]

[0020] in, represents the weight of the linear layer, After being sent to the layer normalization LN layer, it is sent to three linear layers to calculate the three components , , , each component divides the last dimension into R heads, that is ,

[0021] , , , R heads calculate the attention respectively as

[0022]

[0023] in, express The size of the last dimension, Represents the intra-class transformation IaT module component The rth head of Represents the intra-class transformation IaT module component The rth head of , T represents transpose, Represents the intra-class transformation IaT module component The rth head of the model, A1 represents the attention of the first head, A2 represents the attention of the second head of the intra-class transformation IaT module, and A r represents the attention of the r-th head of the intra-class transformation IaT module, represents the attention of the Rth head of the intra-class transformation IaT module, CL represents the concatenation operation before feeding into the linear layer, The features after being fed into the layer normalization layer, the linear layer and the GELU activation layer to form the FFN module are added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: inputting the inter-class transformation module Flattened to , and then swap the dimensions to , fed into the linear layer and then added with learnable parameters The position encoding is then fed into the normalization layer and the linear layer. The process is expressed as:

[0025]

[0026] in, 、 Represents the weight of the linear layer, The last dimension is divided into R heads, and we get ;enter Flattened to , and then swap the dimensions to , after being fed into the linear layer and the position encoding of the learnable parameter PE, the process is expressed as:

[0027]

[0028] in, represents the weight of the linear layer, After being fed into the normalization layer, they are fed into the linear layer to obtain two components. , , each component divides the last dimension into R heads, and we get

[0029] , , , R heads calculate the attention respectively as:

[0030]

[0031] in, Represents the inter-class transformation ItT module component The rth head of Represents the inter-class transformation ItT module component The rth head of Represents the inter-class transformation ItT module component The rth head of represents the attention of the first head of the inter-class transformation ItT module, Represents the attention of the second head of the inter-class transformation ItT module, represents the attention of the r-th head of the inter-class transformation ItT module, represents the attention of the Rth head of the inter-class transformation ItT module, The features after being fed into the layer normalization layer, the linear layer and the GELU activation layer to form the FFN module are added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

[0032] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: in the category-specific feature extractor, the pathological image I i With other categories Send in Extracted category cross features ,in, , calculate the category cross loss :

[0033]

[0034] in, calculate The L1 norm of .

[0035] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: a process in which the segmentation module performs segmentation based on global features of the pathological image to obtain a multi-component segmentation result:

[0036] Set the feature to global Feed into the segmentation module based on U-Net decoder , output pathological image I i The multi-component segmentation result P i , and calculate the segmentation loss :

[0037]

[0038] in, represents the cross entropy loss, represents the Dice similarity loss, is a multi-component category label The label value of the mth class of the tth pixel in , is the segmentation result The predicted probability of the mth class for the tth pixel in , where T is the number of pixels.

[0039] In conjunction with the first aspect, in certain implementations of the first aspect, the method further includes: a process in which the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result:

[0040] The fused class-specific features Feed into the class image reconstruction decoder Perform category image reconstruction and output the reconstructed image , and calculate the reconstruction loss :

[0041]

[0042] in, calculate The L2 norm of .

[0043] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a multi-component segmentation image system in a pathological image based on category separation, comprising:

[0044] An image processing unit, configured to acquire a pathological image and input the pathological image into a pre-established category separation network model for processing, wherein the category separation network model includes a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module, and a category image reconstruction decoder;

[0045] a feature fusion unit, configured for the global feature extractor and the category-specific feature extractor to extract global features and category-specific features of the pathological image respectively, and the category fusion module to fuse the global features and category-specific features of the pathological image to obtain fused global features and category-specific features;

[0046] The component segmentation unit is used for the segmentation module to perform segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image.

[0047] Beneficial effects of the present invention:

[0048] The present invention can further improve the segmentation performance and is applicable to multi-component segmentation of pathological images. The present invention achieves better results in multi-category segmentation tasks of pathological images. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0050] Figure 1 It is a schematic flow chart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram of the intra-class transformation IaT module of the present invention;

[0052] Figure 3 It is a schematic diagram of the inter-class transformation ItT module of the present invention;

[0053] Figure 4 This is a schematic diagram of the category fusion module CF of the present invention;

[0054] Figure 5 is a schematic diagram of the category separation network of the present invention;

[0055] Figure 6 2. It is a schematic diagram comparing the segmentation of the present invention and the existing method;

[0056] Figure 7 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] Example 1:

[0059] like Figure 1 As shown, a multi-component image segmentation method in a pathological image based on category separation includes the following steps:

[0060] S101: Acquire a pathological image and input the pathological image into a pre-established category separation network model for processing, wherein the category separation network model includes a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module, and a category image reconstruction decoder;

[0061] The pre-established category separation network model is obtained by jointly optimizing alignment loss, category intersection loss, segmentation loss function, contrast loss function and reconstruction loss;

[0062] The entire network is optimized using the total loss, which is expressed as:

[0063]

[0064] The pathological images are obtained by constructing a pathological image training set, which includes pathological images I i and its corresponding multi-component category labels Represents the number of pathological images in the training set, and the global feature extractor E based on the U-Net encoder is constructed. g Extract pathological image I i Global characteristics of , by constructing M category-specific feature extractors based on U-Net encoders , extract pathological image I i Class-specific features of , M represents the number of component segmentation categories, Indicates pixel multiplication;

[0065] S102: The global feature extractor and the category-specific feature extractor extract global features and category-specific features of the pathological image respectively, and the category fusion module fuses the global features and category-specific features of the pathological image to obtain fused global features and category-specific features;

[0066] The category fusion module will Send it to the ClassFusion (CF) module to get the fusion , and use KL divergence (Kullback-LeiblerDivergence) to global features and fused global features Alignment,The alignment loss is defined as:

[0067]

[0068] in, Indicates calculation of global features and fused global features The KL divergence between .

[0069] The category fusion module consists of an intra-class transformer (IaT) module, an inter-class transformer (ItT) module, a concat (channel concatenation), a linear layer (Linear), and a layer normalization (LN) layer. They are sent to the intra-class transformation module to obtain , As Input1, with As Input2, it is sent to the inter-class transformation module to obtain the fused category-specific features. , Then, they are fed into the concat layer, the linear layer, and the layer normalization layer as Input2. As Input1, it is sent to the inter-class transformation module to obtain the fused global features. .

[0070] The input of the intra-class transformation module Among them, C represents the channel, H represents the height, W represents the width, and the flattening is , and then exchange the dimension (Permute) to , sent to the linear layer (Linear) and then added with the position encoding of the learnable parameter PE, the process can be expressed as

[0071]

[0072] in, represents the weight of the linear layer, After being sent to the layer normalization LN layer, it is sent to three linear layers to calculate the three components , , , each component divides the last dimension into R heads, that is ,

[0073] , , , R heads calculate the attention respectively as

[0074]

[0075] in, express The size of the last dimension of , CL means concatenation operation first and then feeding into the linear layer, The features after being fed into the layer normalization layer, the FFN module composed of two linear layers and the GELU activation layer are then added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

[0076] The input of the inter-class transformation module Flattened to , and then swap the dimensions to , fed into the linear layer and then added with learnable parameters The position encoding is then fed into the normalization layer and the linear layer. The process is expressed as:

[0077]

[0078] in, 、 Represents the weight of the linear layer, The last dimension is divided into R heads, and we get ;enter Flattened to , and then swap the dimensions to , after being fed into the linear layer and the position encoding of the learnable parameter PE, the process is expressed as:

[0079]

[0080] in, represents the weight of the linear layer, After being fed into the normalization layer, they are fed into the linear layer to obtain two components. , , each component divides the last dimension into R heads, and we get

[0081] , , , R heads calculate the attention respectively as:

[0082]

[0083] The features after being fed into the layer normalization layer, the FFN module composed of two linear layers and the GELU activation layer are then added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

[0084] As a preferred solution of the multi-component image segmentation method based on category separation in the pathological image of the present invention, the pathological image I i With other categories Send in Extracted category cross features ,in, , calculate the category cross loss :

[0085]

[0086] in, calculate The L1 norm of .

[0087] S103: The segmentation module performs segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image.

[0088] The segmentation module performs segmentation based on the global features of the pathological image to obtain a multi-component segmentation result:

[0089] Set the feature to global Feed into the segmentation module based on U-Net decoder , output pathological image I i The multi-component segmentation result P i , and calculate the segmentation loss :

[0090]

[0091] in, represents the cross entropy loss, represents the Dice similarity loss, is a multi-component category label The label value of the mth class of the tth pixel in , is the segmentation result The predicted probability of the mth class for the tth pixel in , where T is the number of pixels.

[0092] The process of the category image reconstruction decoder performing category image reconstruction based on the fused category-specific features and multi-component segmentation results:

[0093] The fused class-specific features Feed into the class image reconstruction decoder Perform category image reconstruction and output the reconstructed image , and calculate the reconstruction loss :

[0094]

[0095] in, calculate The L2 norm of .

[0096] Specifically, the present invention will be further described below through examples:

[0097] In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0098] First, the data used in this paper was collected from an international public dataset (Semantic Segmentation of Breast Cancer, BCSS). This dataset includes whole-slide pathology images from 151 different patients, all stained with hematoxylin and eosin (HE). These whole-slide images were annotated by an experienced pathologist and include all regions within the image for five component categories: tumor, stroma, lymphocytic infiltrate, necrosis, and other components. 100 images served as the training set, and 51 images served as the test set. The whole-slide images were processed using the following steps: A 512×512 sliding window was used to crop the whole-slide images into non-overlapping partial-slide images. The cropped images comprised 32,000 partial-slide images in the training set, and 15,000 partial-slide images in the test set.

[0099] This paper uses the PyTorch deep learning framework, Torch version 1.13.0. All training and validation processes were performed on an NVIDIA GeForce RTX 3090 graphics card equipped with 24GB of video memory. During training, the neural network used a mini-batch approach to read data, with a batch size of 2. The Stochastic Gradient Descent (SGD) optimizer was used, with an initial learning rate of 0.01, an optimizer momentum of 0.9, and an optimizer regularization coefficient of 0.0001. The Poly method was used as the learning rate adjustment strategy.

[0100] The experimental results of the present invention are as follows:

[0101] In order to quantitatively evaluate the performance of the proposed method, two different evaluation metrics were selected to evaluate the performance of the neural network in the multi-component segmentation task of breast cancer: Dice similarity coefficient (DSC) and intersection over union (IoU). The DSC calculation formula is:

[0102]

[0103] in, is the number of true positive segmented pixels, is the number of false positive segmented pixels, is the number of false negative segmented pixels. mDSC calculates the average DSC of all categories. The formula for calculating the intersection-over-union ratio is:

[0104]

[0105] mIoU calculates the average IoU of all categories. The segmentation results of the present invention are compared with those of existing methods, as shown in Table 1. Table 1 compares the segmentation results of the present invention with those of other methods.

[0106] like Figure 2 The figure shows the schematic diagram of the intra-class transformation IaT module of the present invention. The input of the intra-class transformation IaT module is (C stands for channel, H stands for height, and W stands for width) Flattened to , and then exchange the dimension (Permute) to , sent to the linear layer (Linear) and then added with the position encoding of the learnable parameter PE, the process can be expressed as

[0107]

[0108] in, represents the weight of the linear layer, After being sent to the layer normalization LN layer, it is sent to three linear layers to calculate the three components , , , each component divides the last dimension into R heads, that is ,

[0109] , , , the rth head of the intra-class transformation IaT module calculates the attention as

[0110]

[0111] in, express The size of the last dimension, Represents the intra-class transformation IaT module component The rth head of Represents the intra-class transformation IaT module component The rth head of , T represents transpose, Represents the intra-class transformation IaT module component The rth head of the model, A1 represents the attention of the first head, A2 represents the attention of the second head of the intra-class transformation IaT module, and A r represents the attention of the r-th head of the intra-class transformation IaT module, represents the attention of the Rth head of the intra-class transformation IaT module, CL represents the concatenation operation before feeding into the linear layer, The features after being fed into the layer normalization layer, the linear layer and the GELU activation layer to form the FFN module are added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

[0112] like Figure 3 The figure shows the schematic diagram of the inter-class transformation ItT module of the present invention. The input of the inter-class transformation ItT module is Flattened to , and then swap the dimensions to , fed into the linear layer and then added with learnable parameters The position encoding is then fed into the layer normalization layer and the linear layer. The process can be expressed as

[0113]

[0114] in, 、 Represents the weight of the linear layer, The last dimension is divided into R heads, and we get ;enter Flattened to , and then swap the dimensions to , after being fed into the linear layer and the position encoding of the learnable parameter PE, the process is expressed as:

[0115]

[0116] in, represents the weight of the linear layer, After being fed into the normalization layer, they are fed into the linear layer to obtain two components. , , each component divides the last dimension into R heads, and we get

[0117] , , , the r-th head of the inter-class transformation ItT module calculates the attention as:

[0118]

[0119] in, Represents the inter-class transformation ItT module component The rth head of Represents the inter-class transformation ItT module component The rth head of Represents the inter-class transformation ItT module component The rth head of represents the attention of the first head of the inter-class transformation ItT module, Represents the attention of the second head of the inter-class transformation ItT module, represents the attention of the r-th head of the inter-class transformation ItT module, represents the attention of the Rth head of the inter-class transformation ItT module, The features after being fed into the layer normalization layer, the linear layer and the GELU activation layer to form the FFN module are added , and then perform the inverse transformation of swapping dimensions and flattening to obtain ,

[0120] The features after being fed into the layer normalization layer, the FFN module composed of two linear layers and the GELU activation layer are then added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

[0121] like Figure 4 As shown in FIG, it is a schematic diagram of the category fusion module CF of the present invention. The category fusion module CF is composed of an intra-class transformation IaT module, an inter-class transformation ItT module, concat (channel splicing), a linear layer (Linear) and a layer normalization (LN) layer. They are sent to the intra-class transformation module to obtain 、 . As Input1, with As Input2, it is sent to the inter-class transformation module to obtain the fused category-specific features. . Then, they are fed into the concat layer, the linear layer, and the layer normalization layer as Input2. As Input1, it is sent to the inter-class transformation module to obtain the fused global features. ;

[0122] like Figure 5 The figure shows a schematic diagram of the class separation network of the present invention; the class separation network consists of a global feature extractor, a class-specific feature extractor, a class fusion module, a segmentation module, and a class image reconstruction decoder. Global Feature Extractor , used to extract image I i Global characteristics of ; M class-specific feature extractors ( , M represents the number of component segmentation categories), used to extract image I i Class-specific features of ; Category fusion module CF, used to fuse global features and class-specific features ; Image I i With other categories Send in Extracted category cross features , used for category separation; segmentation module D g , used to convert global features As input, we get the multi-component segmentation result P i ; The fused class-specific features

[0123] Feed into the class image reconstruction decoder Perform category image reconstruction and output the reconstructed image .

[0124] like Figure 6 The following are segmentation results from different methods. The first row shows the original pathology slide image. The second row shows the corresponding gold standard. Following are segmentation results from DRD-UNet, TestFit, DETisSeg, PCSformer, HisynSeg, and the method presented in this paper. The last row shows the segmentation labels, indicating the regions of the five component types in the image: Tumor, Stroma, Lymphocytic Infiltrate, Necrosis, and Others.

[0125] Table 1 Experimental data comparison table

[0126]

[0127]

[0128] As shown in Table 1, compared with other existing methods, the method proposed in this invention outperforms other existing methods in terms of the average segmentation accuracy of the five categories, whether in terms of Dice similarity coefficient or intersection-over-union ratio. Compared with the best HisynSeg method among other methods, the method proposed in this invention improves the five-category mIoU by 1.39% and the five-category mDSC reaches 79.83%. Figure 6 It can be seen that other methods still have many incorrect segmentation situations. For example, DRD-UNet segmentation obtains the image of the second row and the first column, in which "others" is incorrectly segmented as stroma; TestFit segmentation obtains the image of the third row and the second column, in which "necrosis" is incorrectly segmented as tumor; DETisSeg segmentation obtains the image of the fourth row and the first column, in which part of "others" is incorrectly segmented as stroma; PCSformer segmentation obtains the image of the fifth row and the first column, in which part of "others" is incorrectly segmented as stroma; HisynSeg segmentation obtains the image of the sixth row and the first column, in which part of "others" is incorrectly segmented as stroma; the segmentation result of the method of the present invention is very close to the segmentation label of the last row.

[0129] In order to prove the benefits of each part of the present invention for image segmentation, ablation experiments are further conducted below. Basic Network is a global feature extractor With the segmentation module The UNet is constructed and used To constrain multi-class segmentation; basic network + category-specific feature extractor + category image reconstruction decoder Indicates that a category-specific feature extractor and a category image reconstruction decoder are added to the base network, and To constrain multi-class segmentation, To constrain category image reconstruction; basic network + category-specific feature extractor + category image reconstruction decoder Indicates that a category-specific feature extractor, category image reconstruction decoder, and category fusion module are added to the base network, and To constrain multi-class segmentation, To constrain the category image reconstruction, use To constrain category separation; basic network + category-specific feature extractor + category image reconstruction decoder + inter-class transformation module Indicates that a category-specific feature extractor, a category image reconstruction decoder, and an inter-class transformation module are added to the base network, and To constrain multi-class segmentation, To constrain the category image reconstruction, use To constrain category separation; basic network + category-specific feature extractor + category image reconstruction decoder + category fusion module Indicates that a category-specific feature extractor, category image reconstruction decoder, and category fusion module are added to the base network, and To constrain multi-class segmentation, To constrain the category image reconstruction, use To constrain the category separation; finally, the method of the present invention. As can be seen from Table 2, with each new module or loss function, the evaluation indicators mIoU and mDSC are improved.

[0130] Table 2 Ablation experiment

[0131]

[0132] Example 2: The second aspect, as Figure 7 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a multi-component segmentation image system in a pathological image based on category separation, comprising:

[0133] An image processing unit 11 is used to acquire a pathological image and input the pathological image into a pre-established category separation network model for processing, wherein the category separation network model includes a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module, and a category image reconstruction decoder;

[0134] A feature fusion unit 12, configured for the global feature extractor and the category-specific feature extractor to extract global features and category-specific features of the pathological image respectively, and the category fusion module to fuse the global features and category-specific features of the pathological image to obtain fused global features and category-specific features;

[0135] The component segmentation unit 13 is used for the segmentation module to perform segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder to perform category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image.

[0136] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0137] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the above-described method. The storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0138] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0139] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present disclosure. Various changes and improvements may be made to the present disclosure without departing from the spirit and scope of the present disclosure, and such changes and improvements shall fall within the scope of the present disclosure.

Claims

1. A multi-component image segmentation method in pathological images based on category separation, characterized in that: The method comprises the following steps: Acquiring a pathological image and inputting the pathological image into a pre-established category separation network model for processing, wherein the category separation network model includes a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module, and a category image reconstruction decoder; The global feature extractor and the category-specific feature extractor extract global features and category-specific features of the pathological image respectively, and the category fusion module fuses the global features and category-specific features of the pathological image to obtain fused global features and category-specific features; The process of the category fusion module fusing the global features of the pathological image: Will Send it to the category fusion module CF to get the fusion , and use KL divergence to global features and fused global features Alignment,The alignment loss is defined as: in, Indicates calculation of global features and fused global features The KL divergence between The category fusion module consists of an intra-class transformation module, an inter-class transformation module, a concat layer, a linear layer, and a layer normalization layer. They are sent to the intra-class transformation module to obtain 、 , As Input1, with As Input2, it is sent to the inter-class transformation module to obtain the fused category-specific features. , Then, they are fed into the concat layer, the linear layer, and the layer normalization layer as Input2. As Input1, it is sent to the inter-class transformation module to obtain the fused global features. ; The segmentation module performs segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image.

2. The method for multi-component segmentation of pathological images based on category separation according to claim 1, characterized in that: The pre-established category separation network model is obtained by jointly optimizing alignment loss, category intersection loss, segmentation loss function, contrast loss function and reconstruction loss.

3. The multi-component image segmentation method in pathological images based on category separation according to claim 1 is characterized in that: The pathological images are obtained by constructing a pathological image training set, which includes pathological images I i and its corresponding multi-component category labels Represents the number of pathological images in the training set, and the global feature extractor E based on the U-Net encoder is constructed. g Extract pathological image I i Global characteristics of , by constructing M category-specific feature extractors based on U-Net encoders , extract pathological image I i Class-specific features of , M represents the number of component segmentation categories, Represents pixel multiplication.

4. The method for multi-component segmentation of pathological images based on category separation according to claim 1, characterized in that: The input of the intra-class transformation module Flatten , then exchange the dimension Permute to , where C represents the channel, H represents the height, and W represents the width. After being fed into the linear layer, the position encoding of the learnable parameter PE is added. The process is expressed as in, represents the weight of the linear layer, After being sent to the layer normalization LN layer, it is sent to three linear layers to calculate the three components , , , each component divides the last dimension into R heads, that is , , , , R heads calculate the attention respectively as in, express The size of the last dimension, Represents the intra-class transformation IaT module component The rth head of Represents the intra-class transformation IaT module component The rth head of , T represents transpose, Represents the intra-class transformation IaT module component The rth head of the model, A1 represents the attention of the first head, A2 represents the attention of the second head of the intra-class transformation IaT module, and A r represents the attention of the r-th head of the intra-class transformation IaT module, represents the attention of the Rth head of the intra-class transformation IaT module, CL represents the concatenation operation before feeding into the linear layer, The features after being fed into the layer normalization layer, the linear layer and the GELU activation layer to form the FFN module are added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

5. The method for multi-component segmentation of pathological images based on category separation according to claim 4, characterized in that: The input of the inter-class transformation module Flattened to , and then swap the dimensions to , fed into the linear layer and then added with learnable parameters The position encoding is then fed into the normalization layer and the linear layer. The process is expressed as: in, 、 Represents the weight of the linear layer, The last dimension is divided into R heads, and we get ;enter Flattened to , and then swap the dimensions to , after being fed into the linear layer and the position encoding of the learnable parameter PE, the process is expressed as: in, represents the weight of the linear layer, After being fed into the normalization layer, they are fed into the linear layer to obtain two components. , , each component divides the last dimension into R heads, and we get , , , R heads calculate the attention respectively as: in, Represents the inter-class transformation ItT module component The rth head of Represents the inter-class transformation ItT module component The rth head of Represents the inter-class transformation ItT module component The rth head of represents the attention of the first head of the inter-class transformation ItT module, Represents the attention of the second head of the inter-class transformation ItT module, represents the attention of the r-th head of the inter-class transformation ItT module, represents the attention of the Rth head of the inter-class transformation ItT module, The features after being fed into the layer normalization layer, the linear layer and the GELU activation layer to form the FFN module are added , and then perform the inverse transformation of swapping dimensions and flattening to obtain .

6. The method for multi-component segmentation of pathological images based on category separation according to claim 1, characterized in that: In the class-specific feature extractor, the pathological image I i With other categories Send in Extracted category cross features ,in, , calculate the category cross loss : in, calculate The L1 norm of .

7. The method for multi-component segmentation of pathological images based on category separation according to claim 1, characterized in that: The segmentation module performs segmentation based on the global features of the pathological image to obtain a multi-component segmentation result: Set the feature to global Feed into the segmentation module based on U-Net decoder , output pathological image I i The multi-component segmentation result P i , and calculate the segmentation loss : in, represents the cross entropy loss, represents the Dice similarity loss, is a multi-component category label The label value of the mth class of the tth pixel in , is the segmentation result The predicted probability of the mth class for the tth pixel in , where T is the number of pixels.

8. The method for multi-component segmentation of pathological images based on category separation according to claim 1, characterized in that: The process of the category image reconstruction decoder performing category image reconstruction based on the fused category-specific features and multi-component segmentation results: The fused class-specific features Feed into the class image reconstruction decoder Perform category image reconstruction and output the reconstructed image , and calculate the reconstruction loss : in, calculate The L2 norm of .

9. A system for segmenting multiple components in pathological images based on category separation, which adopts the method for segmenting multiple components in pathological images based on category separation according to any one of claims 1 to 8, characterized in that: include: An image processing unit, configured to acquire a pathological image and input the pathological image into a pre-established category separation network model for processing, wherein the category separation network model includes a global feature extractor, a category-specific feature extractor, a category fusion module, a segmentation module, and a category image reconstruction decoder; a feature fusion unit, configured for the global feature extractor and the category-specific feature extractor to extract global features and category-specific features of the pathological image respectively, and the category fusion module to fuse the global features and category-specific features of the pathological image to obtain fused global features and category-specific features; The component segmentation unit is used for the segmentation module to perform segmentation based on the global features of the pathological image to obtain a multi-component segmentation result, and the category image reconstruction decoder performs category image reconstruction based on the fused category-specific features and the multi-component segmentation result to obtain a reconstructed image.

Images