A colorectal cancer immunohistochemistry image gland segmentation method, device, and medium

Abstract

The application discloses a colorectal cancer immunohistochemical image gland segmentation method, equipment and medium, which comprises the following steps: inputting an original colorectal cancer immunohistochemical image to be processed into a gland segmentation model to obtain a binary gland recognition result, and superimposing the gland recognition result and the original colorectal cancer immunohistochemical image to obtain a gland segmentation result; wherein the gland segmentation model comprises a down-sampling module, an up-sampling module and a skip connection module for connecting the down-sampling module and the up-sampling module; and the skip connection module is composed of a direct skip connection and a convolution module in parallel.

Description

Technical Field

[0001] The present invention relates to the technical field of digital pathology image processing, and in particular to a method, device, and medium for colorectal cancer immunohistochemical image glands in a small sample scenario. Background Art

[0002] In colorectal cancer tissue, pathological changes such as the morphology of glandular ducts are important predictive features, so the glandular segmentation results have important guiding significance for subsequent immunohistochemical scoring and prognostic analysis. The glandular structure in different stages of cancer and differentiation is often complex and changeable, which also places higher demands on the performance of the segmentation network. Existing studies have introduced complex structures into the segmentation network to improve the model's perception of complex scenes, but the data sets in medical scenarios are often very limited, and complex models often cannot be well trained and perform poorly. Existing studies often reduce the complexity of the model by reducing the depth of the network, which is contrary to the requirement of high-difficulty segmentation tasks to maintain the depth of the network to ensure the network's feature extraction and decision-making capabilities.

[0003] Therefore, it is urgent to propose a gland segmentation method for colorectal cancer immunohistochemical images in small sample scenarios to achieve accurate segmentation of glands in immunohistochemical images and provide more accurate segmentation results of the area of ​​interest for tasks such as immunohistochemical interpretation. Summary of the Invention

[0004] In view of this, the present invention provides a method, device, and medium for gland segmentation in colorectal cancer immunohistochemistry images in a small sample scenario.

[0005] In a first aspect, an embodiment of the present invention provides a method for gland segmentation in colorectal cancer immunohistochemistry images, the method comprising:

[0006] The original colorectal cancer immunohistochemical image to be processed is input into the gland segmentation model to obtain a binary gland recognition result, and the gland recognition result is superimposed on the original colorectal cancer immunohistochemical image to obtain a gland segmentation result;

[0007] The gland segmentation model includes a downsampling module, an upsampling module, and a jump connection module for connecting the downsampling module and the upsampling module; the jump connection module is composed of a direct jump connection and a convolution module in parallel.

[0008] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned colorectal cancer immunohistochemistry image gland segmentation method.

[0009] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the colorectal cancer immunohistochemical image gland segmentation method.

[0010] In a fourth aspect, an embodiment of the present application provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the colorectal cancer immunohistochemical image gland segmentation method.

[0011] Compared with the prior art, the present application has the following beneficial effects:

[0012] The present application provides a colorectal cancer immunohistochemical image gland segmentation method, which is aimed at the requirements of context information and local detail information for complex deformation gland segmentation tasks, and proposes to construct a minimum full-depth link feature fusion structure on the basis of UNet to realize full-dimensional feature utilization. The present application utilizes a skip connection module to better transmit detail information to realize better boundary processing capability, and inserts a multi-head self-attention mechanism module in the second down-sampling sub-module and the third down-sampling sub-module to process the gland structure and the internal cavity area with increased deformation, fully utilizes the strong prompt information of the specific staining mechanism of the immunohistochemical image on the negative and positive gland positions, and thus improves the accuracy of the immunohistochemical image gland segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 A schematic diagram of the gland segmentation model provided by the embodiment of the present application;

[0015] Figure 2 A schematic diagram of the minimum full-depth link feature fusion structure provided by the embodiment of the present application;

[0016] Figure 3 A structure diagram of the Self Attention module provided by the embodiment of the present application;

[0017] Figure 4 A structure diagram of the ECANet channel attention module provided by the embodiment of the present application;

[0018] Figure 5 A comparison diagram of the visual segmentation results of the method of the present application and the prior art research method provided by the embodiment of the present application;

[0019] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a method for gland segmentation in colorectal cancer immunohistochemistry images, the method comprising:

[0023] The original colorectal cancer immunohistochemical image to be processed is input into the gland segmentation model to obtain a binary gland recognition result, and the gland recognition result is superimposed on the original colorectal cancer immunohistochemical image to obtain a gland segmentation result;

[0024] The gland segmentation model includes a downsampling module, an upsampling module, and a skip connection module connecting the downsampling and upsampling modules. The skip connection module is composed of a direct skip connection and a convolution module in parallel. The parallel convolution modules in the skip connection module include a first 3×3 convolution layer, a first batch of normalization layers, a first Relu activation function, a second 3×3 convolution layer, a second batch of normalization layers, and a second Relu activation function, connected in sequence.

[0025] Furthermore, the downsampling module includes a first downsampling submodule, a second downsampling submodule, a third downsampling submodule, and a fourth downsampling submodule connected in sequence; the first downsampling submodule and the fourth downsampling submodule are composed of a ResNet module and an ECA module connected in sequence; the second downsampling submodule and the third downsampling submodule are composed of a ResNet module, a multi-head self-attention mechanism module, and an ECA module connected in sequence.

[0026] It should be noted that in this example, to address the receptive field requirements of glands with varying degrees of cancer and differentiation in immunohistochemistry images, the multi-head self-attention mechanism was introduced only in the second and third downsampling submodules. This effectively strikes a balance between spatial resolution and semantic information, achieving better results while avoiding complex computations and overfitting. Furthermore, the multi-head self-attention mechanism addresses the deformation and enlargement of glands due to cancer. This mechanism effectively compensates for the limited receptive field of traditional convolutional neural networks, which cannot effectively determine the internal cavity of the gland.

[0027] Furthermore, the upsampling module includes a first upsampling submodule, a second upsampling submodule, a third upsampling submodule, and a fourth upsampling submodule connected in sequence; wherein the first upsampling submodule, the second upsampling submodule, the third upsampling submodule, and the fourth upsampling submodule are all composed of an ECA module and a convolution module.

[0028] It should be noted that the ECA module can assign greater weight to specific staining channels. This mechanism can effectively identify glands that are negative and positive for specific staining. ECANet controls the local cross-channel information interaction by controlling the convolution kernel size. In this example, the convolution kernel size is set to 3.

[0029] Furthermore, the gland segmentation model specifically includes: a first downsampling submodule is connected to a first upsampling submodule through a first jump connection submodule; a second downsampling submodule is connected to a second upsampling submodule through a second jump connection submodule; a third downsampling submodule is connected to a third upsampling submodule through a third jump connection submodule; and a fourth downsampling submodule is connected to a fourth upsampling submodule through a fourth jump connection submodule; so that the gland recognition result output by the gland segmentation model includes all feature maps that have undergone odd-numbered convolutions and feature maps that have undergone even-numbered convolutions.

[0030] Furthermore, the feature dimension of the i-th skip connection submodule is the same as the feature dimension of the i-th downsampling submodule; in this example, the feature dimensions of each layer of the downsampling part are 64, 128, 256, and 512, respectively. The feature map of each layer is downsampled through the maximum pooling operation and dimensionally expanded through the first convolution block.

[0031] Further, the i-th skip connection submodule performs channel dimension concatenation with the original long skip connection feature map. In this example, the fused features of each layer of the decoding part are composed of three parts: the high-resolution feature map of the corresponding layer directly transmitted from the encoding part, the high-level feature after the specific convolution operation from the newly added skip connection path, and the feature map after the spatial resolution recovery of the next layer of the decoding part by the up-sampling operation. The three types of feature maps are concatenated in the channel dimension, and the number of channels after concatenation is the sum of the number of channels of the three.

[0032] It should be noted that the present application improves the UNet network based on the minimum full-depth link fusion structure, and uses the least links to realize the fusion of full-depth features through reasonable structure design, thereby obtaining a network structure design with efficient information transmission, and improving the accuracy of colorectal cancer immunohistochemical image gland segmentation in a small sample scene. The skip connection module is used to realize efficient transmission of information flow in the encoding stage to retain more important detail features of the segmentation task;

[0033] Further, the multi-head self-attention mechanism module includes a batch normalization layer (Batch Normalization) connected in sequence, attention mechanism calculation, a batch normalization layer, a Relu activation function, an Mlp layer realized by a 1x1 convolution, the input feature map is added to the feature map after attention mechanism calculation through a residual connection, and then the feature map after attention mechanism calculation is added to the last feature map for output to realize detail feature transmission of the segmentation network. The multi-head self-attention mechanism module is used for feature dimension reduction, relative position coding, and self-attention mechanism calculation. Specifically, it includes:

[0034] Feature dimension reduction, removing redundant information: on the basis of traditional self-attention, two projection operations are used to project the key K and value V from the original high-dimensional space to a low-dimensional space wherein the key K and value V are projections of the input X in the attention mechanism, d is the dimension of each key or value vector, n=HxW, H and W represent the input feature map size, k=hwxw<<n, h and w represent the size of the down-sampled feature map;

[0035] Relative position coding, enhancing position perception ability: the relative position between pixels i=(i x , i y ) and j=(j x , j y ) along the height and width dimensions is calculated as wherein wherein q iDenotes the query vector representing pixel i, which is used to capture the correlation between pixel i and other pixels and interacts with the query vector to calculate the attention weight. are the relative width j x -i x and relative height j y -i y Learnable embeddings of

[0036] Self-attention mechanism calculation:

[0037]

[0038] Furthermore, the ECA module is used for global information aggregation, feature recalibration, and output of weighted feature maps. Specifically, it includes:

[0039] Global information aggregation: Perform global average pooling on the initially extracted feature maps in the spatial dimension to generate a vector describing the global information of each channel;

[0040] Feature recalibration: Cross-channel information interaction is performed through one-dimensional convolution to learn channel weights;

[0041] Output weighted feature map: The weighted feature map processed by the channel attention module is output to guide the subsequent gland segmentation model to more effectively identify and distinguish specific stained negative and positive glands.

[0042] In summary, the present invention provides a method for gland segmentation in immunohistochemical images of colorectal cancer. This example proposes an optimized model design suitable for complex segmentation tasks in small sample scenarios. A minimum full-depth link feature fusion structure is designed, and the fusion of full-depth features is achieved by retaining the least links, which provides a new idea for the construction of segmentation networks in complex task scenarios with small sample data sets. In response to the requirements of the deformed and enlarged glandular structure in immunohistochemical images for the receptive field, a multi-head self-attention mechanism module is introduced to capture long-distance dependencies, and ResNet short jump connections are used to achieve efficient transmission of information flow. In addition, a sophisticated structural design is used to reduce the computational overhead of the complex attention mechanism while avoiding over-fitting. Combined with the specific staining mechanism of immunohistochemical images, the channel attention mechanism is introduced to enhance the model's attention to important features, which is a significant improvement compared to the existing classic models.

[0043] Example 1

[0044] 1) Simulation conditions

[0045] The experiment uses a server equipped with a CPU Intel(R)Core(TM)i7-13700KF Processor@5.40GHz, 64GB memory, and two GPURTX 4090 (24GB) graphics cards. The model is implemented based on the Pytorch deep learning framework, the Pytorch version is 1.8.1, and the Python version is 3.8. During training, the input image is uniformly scaled to 512×512, the batch size is controlled to 4, the Adam optimizer is used for parameter optimization, and the cosine annealing algorithm is used to adjust the learning rate. The initial learning rate is set to 1e-4, and the final learning rate is set to 1e-6. The loss function used is Binary CrossEntropyLoss, and the epoch of each training is set to 200. In order to avoid overfitting and enhance the adaptability and robustness of the model, random transformations such as random rotation, random horizontal or vertical flipping, color gamut jitter, and random cropping are used to transform the input image with a certain probability during the training phase. The present invention is verified using two data sets. Among them, the CRAG dataset consists of 213 H&E stained images from 38 whole-slice imaging samples, with a main resolution of 1512×1516 and accompanied by instance-level expert annotations. The dataset contains 173 training images and 40 test images with different cancer grades. The CRC colorectal cancer P53 protein immunohistochemical staining dataset comes from the Institute of Oncology, the Second Affiliated Hospital, School of Medicine, Zhejiang University. This study has been approved by the Human Research Ethics Committee of the Second Affiliated Hospital, School of Medicine, Zhejiang University, with the approval number (2020) Lun Shen Yan No. (481). It contains 180 immunohistochemical slice segmentation and annotation data with a resolution of 3000×3000. The training set contains 120 images (36 negative, 27 weakly positive, 27 positive, and 30 strongly positive), and the test set contains 60 images (18 negative, 12 weakly positive, 12 positive, and 18 strongly positive). This paper compares the proposed method with UNet, UNet++, ResUNet, UTNet and other methods on two datasets and conducts ablation experiments.

[0046] 2) Simulation results

[0047] In this paper, the proposed method is compared with UNet, UNet++, ResUNet, UTNet and other methods on two datasets and ablation experiments are conducted. The Dice and IoU parameters are used to evaluate the segmentation performance.

[0048] Table 1 Comparison of segmentation performance between the method of the present invention and existing research methods

[0049]

[0050] From Table 1 andFigure 5 It can be seen that the present invention has better segmentation results than other methods, greatly improving the pathological image segmentation performance in small sample scenarios, especially with stronger adaptability and robustness. It has achieved good gland segmentation results for lesion images of different levels and has better practical engineering application value.

[0051] Table 2 Ablation experiment results of the present invention

[0052]

[0053] As can be seen from Table 2, the modules and their modular layout design proposed in the present invention are effective in improving feature utilization efficiency, recovering detail information, and combining context information for segmentation decision making, which demonstrates the effectiveness of each module.

[0054] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned colorectal cancer immunohistochemical image gland segmentation method. Figure 6 As shown in FIG, a hardware structure diagram of any device with data processing capability in which the method for gland segmentation of colorectal cancer immunohistochemical images provided by the embodiment of the present invention is provided, except Figure 6 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0055] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned method for gland segmentation in colorectal cancer immunohistochemistry images. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0056] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0057] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for gland segmentation in colorectal cancer immunohistochemical images, characterized in that: The method comprises: The original colorectal cancer immunohistochemical image to be processed is input into the gland segmentation model to obtain a binary gland recognition result, and the gland recognition result is superimposed on the original colorectal cancer immunohistochemical image to obtain a gland segmentation result; The gland segmentation model includes a downsampling module, an upsampling module, and a skip connection module for connecting the downsampling module and the upsampling module; the skip connection module is composed of a direct skip connection and a convolution module in parallel; The downsampling module includes a first downsampling submodule, a second downsampling submodule, a third downsampling submodule, and a fourth downsampling submodule connected in sequence; the first downsampling submodule and the fourth downsampling submodule are composed of a ResNet module and an ECA module connected in sequence; the second downsampling submodule and the third downsampling submodule are composed of a ResNet module, a multi-head self-attention mechanism module, and an ECA module connected in sequence; The upsampling module includes a first upsampling submodule, a second upsampling submodule, a third upsampling submodule, and a fourth upsampling submodule connected in sequence; Among them, the gland segmentation model specifically includes: a first downsampling submodule is connected to a first upsampling submodule through a first jump connection submodule; a second downsampling submodule is connected to a second upsampling submodule through a second jump connection submodule; a third downsampling submodule is connected to a third upsampling submodule through a third jump connection submodule; and a fourth downsampling submodule is connected to a fourth upsampling submodule through a fourth jump connection submodule; so that the gland recognition result output by the gland segmentation model includes all feature maps that have undergone odd-numbered convolutions and feature maps that have undergone even-numbered convolutions.

2. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The first upsampling submodule, the second upsampling submodule, the third upsampling submodule, and the fourth upsampling submodule are all composed of an ECA module and a convolution module.

3. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The parallel convolution module in the jump connection module includes a first 3×3 convolution layer, a first batch of normalization layers, a first Relu activation function, a second 3×3 convolution layer, a second batch of normalization layers, and a second Relu activation function connected in sequence.

4. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The multi-head self-attention mechanism module is used for feature dimensionality reduction, relative position encoding, and self-attention mechanism calculation.

5. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The ECA module is used for global information aggregation, feature recalibration, and output weighted feature maps.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the colorectal cancer immunohistochemical image gland segmentation method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for gland segmentation in colorectal cancer immunohistochemical images as described in any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for gland segmentation in colorectal cancer immunohistochemical images according to any one of claims 1 to 5 is implemented.

Images