Colorectal cancer immunohistochemical image gland segmentation method, equipment and medium

Abstract

The invention discloses a colorectal cancer immunohistochemical image gland segmentation method, equipment and a medium, and the method comprises the steps: inputting a to-be-processed original colorectal cancer immunohistochemical image into a gland segmentation model, obtaining a binary gland recognition result, and superposing the gland recognition result and the original colorectal cancer immunohistochemical image to obtain a gland segmentation result; wherein the gland segmentation model comprises a down-sampling module, an up-sampling module and a jump connection module used for connecting the down-sampling module and the up-sampling module; and the jump connection module is formed by connecting direct jump connection with a convolution module in parallel.

Description

Technical Field

[0001] The present invention relates to the technical field of digital pathology image processing, and in particular to a method, a device and a medium for colorectal cancer immunohistochemical image glands in a small sample scenario. Background Art

[0002] In colorectal cancer tissue, pathological changes such as the morphology of glandular ducts are important predictive features, so the glandular segmentation results have important guiding significance for subsequent immunohistochemical scoring and prognostic analysis. The glandular structures at different stages of carcinogenesis and differentiation are often complex and changeable, which also puts higher requirements on the performance of the segmentation network. Existing studies have introduced complex structures into the segmentation network to improve the model's perception of complex scenes, but the data sets in medical scenarios are often very limited, and complex models often cannot be well trained and perform poorly. Existing studies often reduce model complexity by reducing network depth, which is contrary to the need for high-difficulty segmentation tasks to maintain network depth and ensure the network's feature extraction and decision-making capabilities.

[0003] Therefore, it is urgent to propose a gland segmentation method for colorectal cancer immunohistochemical images in small sample scenarios to achieve accurate segmentation of glands in immunohistochemical images and provide more accurate segmentation results of the area of ​​interest for tasks such as immunohistochemical interpretation. Summary of the invention

[0004] In view of this, the present invention provides a method, device, and medium for gland segmentation in colorectal cancer immunohistochemistry images in a small sample scenario.

[0005] In a first aspect, an embodiment of the present invention provides a method for gland segmentation in colorectal cancer immunohistochemistry images, the method comprising:

[0006] The original colorectal cancer immunohistochemical image to be processed is input into the gland segmentation model to obtain a binary gland recognition result, and the gland recognition result is superimposed on the original colorectal cancer immunohistochemical image to obtain a gland segmentation result;

[0007] The gland segmentation model includes a downsampling module, an upsampling module, and a jump connection module for connecting the downsampling module and the upsampling module; the jump connection module is composed of a direct jump connection and a convolution module in parallel.

[0008] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned colorectal cancer immunohistochemistry image gland segmentation method.

[0009] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the above-mentioned method for gland segmentation in colorectal cancer immunohistochemistry images is implemented.

[0010] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned colorectal cancer immunohistochemical image gland segmentation method.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] The present invention provides a method for gland segmentation in colorectal cancer immunohistochemical images. Aiming at the requirements of complex deformed glandular segmentation tasks for contextual information and local detail information, it proposes to build a minimum full-depth link feature fusion structure based on UNet to fully utilize full-dimensional features. The present invention uses a skip connection module to better transmit detail information to achieve better boundary processing capabilities. At the same time, a multi-head self-attention mechanism module is inserted into the second downsampling submodule and the third downsampling submodule to process the deformed and enlarged glandular structure and its internal cavity area, making full use of the strong prompt information of the specific staining mechanism of immunohistochemical images on the locations of negative and positive glands, thereby improving the accuracy of gland segmentation in immunohistochemical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0014] Figure 1 A schematic diagram of a gland segmentation model provided in an embodiment of the present invention;

[0015] Figure 2 A schematic diagram of a minimum full-depth link feature fusion structure provided by an embodiment of the present invention;

[0016] Figure 3 A structural diagram of a Self Attention module provided in an embodiment of the present invention;

[0017] Figure 4 A structural diagram of the ECANet channel attention module provided by an embodiment of the present invention;

[0018] Figure 5 A comparison diagram of the visual segmentation results of the method of the present invention and the existing research method provided in the embodiment of the present invention;

[0019] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a method for gland segmentation in colorectal cancer immunohistochemistry images, the method comprising:

[0023] The original colorectal cancer immunohistochemical image to be processed is input into the gland segmentation model to obtain a binary gland recognition result, and the gland recognition result is superimposed on the original colorectal cancer immunohistochemical image to obtain a gland segmentation result;

[0024] The gland segmentation model includes a downsampling module, an upsampling module, and a jump connection module for connecting the downsampling module and the upsampling module; the jump connection module is composed of a direct jump connection and a convolution module in parallel. The parallel convolution modules in the jump connection module include a first 3×3 convolution layer, a first batch of normalization layers, a first Relu activation function, a second 3×3 convolution layer, a second batch of normalization layers, and a second Relu activation function connected in sequence.

[0025] Furthermore, the downsampling module includes a first downsampling submodule, a second downsampling submodule, a third downsampling submodule, and a fourth downsampling submodule connected in sequence; the first downsampling submodule and the fourth downsampling submodule are composed of a ResNet module and an ECA module connected in sequence; the second downsampling submodule and the third downsampling submodule are composed of a ResNet module, a multi-head self-attention mechanism module, and an ECA module connected in sequence.

[0026] It should be noted that in this example, in response to the requirements for receptive fields in the cases of deformation and enlargement of glands with different degrees of cancer and differentiation in immunohistochemical images, only the second and third downsampling submodules introduced multi-head self-attention mechanism modules, which can effectively achieve a balance between spatial resolution and semantic information to obtain better results while avoiding complex calculations and over-fitting problems. At the same time, the multi-head self-attention mechanism module is used to solve the problem of glands that are deformed and enlarged due to cancer. This mechanism can effectively make up for the problem that the receptive field of traditional convolutional neural networks is insufficient and cannot make effective decisions on the internal cavity of the gland.

[0027] Furthermore, the upsampling module includes a first upsampling submodule, a second upsampling submodule, a third upsampling submodule, and a fourth upsampling submodule connected in sequence; wherein the first upsampling submodule, the second upsampling submodule, the third upsampling submodule, and the fourth upsampling submodule are all composed of an ECA module and a convolution module.

[0028] It should be noted that the ECA module can give greater weight to specific staining channels, and this mechanism can effectively identify negative and positive glands of specific staining. ECANet controls the local cross-channel information interaction by controlling the convolution kernel size. In this example, the convolution kernel size is set to 3.

[0029] Furthermore, the gland segmentation model specifically includes: a first downsampling submodule is connected to a first upsampling submodule through a first jump connection submodule; a second downsampling submodule is connected to a second upsampling submodule through a second jump connection submodule; a third downsampling submodule is connected to a third upsampling submodule through a third jump connection submodule; and a fourth downsampling submodule is connected to a fourth upsampling submodule through a fourth jump connection submodule; so that the gland recognition result output by the gland segmentation model includes all feature maps that have undergone odd number of convolutions and feature maps that have undergone even number of convolutions.

[0030] Furthermore, the feature dimension of the i-th skip connection submodule is the same as the feature dimension of the i-th downsampling submodule; in this example, the feature dimensions of each layer of the downsampling part are 64, 128, 256, and 512, respectively, and the feature map of each layer is downsampled through the maximum pooling operation and dimensionally expanded through the first convolution block.

[0031] Furthermore, the i-th jump connection submodule is concatenated with the feature map of the original long jump connection in the channel dimension through a concatenation operation. In this example, the fused features of each layer of the decoding part are composed of three parts: the high-resolution feature map of the corresponding layer directly transmitted from the encoding part, the high-level features from the newly added jump connection path, which are processed by a specific convolution operation, and the feature map of the next layer from the decoding part after the spatial resolution is restored through an upsampling operation. The above three types of feature maps are concatenated in the channel dimension, and the number of channels after concatenation is the sum of the number of channels of the three.

[0032] It should be noted that the present invention improves the UNet network based on the minimum full-depth link fusion structure, and realizes the fusion of full-depth features through reasonable structural design using the least links, thereby obtaining a network structure design for efficient information transmission, and improving the accuracy of gland segmentation in colorectal cancer immunohistochemistry images in small sample scenarios. The skip connection module is used to realize efficient transmission of information flow in the encoding stage to retain more important detail features of the segmentation task;

[0033] Furthermore, the multi-head self-attention mechanism module includes a batch normalization layer (BatchNormalization), attention mechanism calculation, batch normalization layer, Relu activation function, and Mlp layer implemented by 1×1 convolution, which are connected in sequence. The input feature map is added to the feature map calculated by the attention mechanism through the residual connection, and then the feature map calculated by the attention mechanism is added to the final feature map for output to realize the detailed feature transmission of the segmentation network. The multi-head self-attention mechanism module is used for feature dimensionality reduction, relative position encoding, and self-attention mechanism calculation. Specifically including:

[0034] Feature dimensionality reduction and removal of redundant information: Based on traditional self-attention, two projection operations are used to transform the key K and value V from the original high-dimensional space Projection to low-dimensional space Where the key K and value V are the projections of the input X in the attention mechanism, d is the dimension of each key or value vector, n = H × W, H and W represent the input feature map size, k = hw < < n, h and w represent the feature map size after downsampling;

[0035] Relative position encoding to enhance position awareness: pixel i along the height and width dimensions = (i x ,i y ) and j=(j x , j y ) is calculated as follows: Among them q iDenotes the query vector representing pixel i, which is used to capture the correlation of pixel i with other pixels and interacts with the query vector to calculate the attention weight. are the relative width j x -i x and relative height j y -i y Learnable embeddings of

[0036] Self-attention mechanism calculation:

[0037]

[0038] Furthermore, the ECA module is used for global information aggregation, feature recalibration, and output of weighted feature maps. Specifically, it includes:

[0039] Global information aggregation: Perform global average pooling on the initially extracted feature maps in the spatial dimension to generate a vector describing the global information of each channel;

[0040] Feature recalibration: Cross-channel information interaction is performed through one-dimensional convolution to learn channel weights;

[0041] Output weighted feature map: The weighted feature map processed by the channel attention module is output to guide the subsequent gland segmentation model to more effectively identify and distinguish specific stained negative and positive glands.

[0042] In summary, the present invention provides a gland segmentation method for colorectal cancer immunohistochemical images. This example proposes an optimized model design suitable for complex segmentation tasks in small sample scenarios. The minimum full-depth link feature fusion structure is designed, and the fusion of full-depth features is achieved by retaining the least links, which provides a new idea for the construction of segmentation networks in complex task scenarios with small sample data sets. In view of the demand for receptive fields of deformed and enlarged glandular structures in immunohistochemical images, a multi-head self-attention mechanism module is introduced to capture long-distance dependencies, and ResNet short jump connections are used to achieve efficient transmission of information flow. In addition, a sophisticated structural design is used to reduce the computational overhead of complex attention mechanisms while avoiding over-fitting. Combined with the specific staining mechanism of immunohistochemical images, the channel attention mechanism is introduced to enhance the model's attention to important features, which is a significant improvement compared to existing classic models.

[0043] Example 1

[0044] 1) Simulation conditions

[0045] The experiment uses a server with a CPU Intel (R) Core (TM) i7-13700KF Processor @ 5.40GHz, 64GB memory, and 2 GPU RTX 4090 (24GB) graphics cards. The model is implemented based on the Pytorch deep learning framework, the Pytorch version is 1.8.1, and the Python version is 3.8. During training, the input image is uniformly scaled to 512×512, the batch size is controlled to 4, the Adam optimizer is used for parameter optimization, and the cosine annealing algorithm is used to adjust the learning rate. The initial learning rate is set to 1e-4, the final learning rate is set to 1e-6, the loss function used is Binary CrossEntropyLoss, and the epoch of each training is set to 200. In order to avoid overfitting and enhance the adaptability and robustness of the model, random transformations such as random rotation, random horizontal or vertical flipping, color gamut jitter, and random cropping are used to transform the input image with a certain probability during the training stage. The present invention is verified using two data sets. Among them, the CRAG dataset consists of 213 H&E stained images from 38 whole-slice imaging samples, with a main resolution of 1512×1516 and instance-level expert annotations. The dataset contains 173 training images and 40 test images with different cancer grades. The CRC colorectal cancer P53 protein immunohistochemical staining dataset comes from the Institute of Oncology, the Second Affiliated Hospital, School of Medicine, Zhejiang University. This study has been approved by the Human Research Ethics Committee of the Second Affiliated Hospital, School of Medicine, Zhejiang University, with the approval number (2020) Lunshenyan No. (481). It contains 180 immunohistochemical slice segmentation and annotation data with a resolution of 3000×3000. The training set contains 120 images (36 negative, 27 weakly positive, 27 positive, and 30 strongly positive), and the test set contains 60 images (18 negative, 12 weakly positive, 12 positive, and 18 strongly positive). The present invention compares the method proposed in this paper with UNet, UNet++, ResUNet, UTNet and other methods on two datasets and conducts ablation experiments.

[0046] 2) Simulation results

[0047] The present invention compares the proposed method with UNet, UNet++, ResUNet, UTNet and other methods on two datasets and conducts ablation experiments, using the Dice and IoU parameters to evaluate the segmentation performance.

[0048] Table 1 Comparison of segmentation performance between the method of the present invention and the existing research methods

[0049]

[0050] From Table 1 and Figure 5 It can be seen that the present invention has better segmentation results than other methods, greatly improving the pathological image segmentation performance in small sample scenarios, especially with stronger adaptability and robustness. It has achieved better gland segmentation results for lesion images of different levels and has better practical engineering application value.

[0051] Table 2 Ablation experiment results of the present invention

[0052]

[0053] As can be seen from Table 2, the modules and their modular layout designs proposed in the present invention are effective in improving feature utilization efficiency, restoring detail information, and combining context information for segmentation decision making, which demonstrates the effectiveness of each module.

[0054] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned colorectal cancer immunohistochemical image gland segmentation method. Figure 6 As shown, it is a hardware structure diagram of any device with data processing capability in which the method for gland segmentation of colorectal cancer immunohistochemical images provided by the embodiment of the present invention is located, except Figure 6 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0055] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned colorectal cancer immunohistochemistry image gland segmentation method is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0056] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0057] It will be appreciated that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for gland segmentation in colorectal cancer immunohistochemical images, characterized in that: The method comprises: The original colorectal cancer immunohistochemical image to be processed is input into the gland segmentation model to obtain a binary gland recognition result, and the gland recognition result is superimposed on the original colorectal cancer immunohistochemical image to obtain a gland segmentation result; The gland segmentation model includes a downsampling module, an upsampling module, and a jump connection module for connecting the downsampling module and the upsampling module; the jump connection module is composed of a direct jump connection and a convolution module in parallel.

2. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The downsampling module includes a first downsampling submodule, a second downsampling submodule, a third downsampling submodule, and a fourth downsampling submodule connected in sequence; The first downsampling submodule and the fourth downsampling submodule are composed of a ResNet module and an ECA module connected in sequence; the second downsampling submodule and the third downsampling submodule are composed of a ResNet module, a multi-head self-attention mechanism module, and an ECA module connected in sequence.

3. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The upsampling module includes a first upsampling submodule, a second upsampling submodule, a third upsampling submodule, and a fourth upsampling submodule connected in sequence; Among them, the first upsampling submodule, the second upsampling submodule, the third upsampling submodule, and the fourth upsampling submodule are all composed of an ECA module and a convolution module.

4. A method for gland segmentation in colorectal cancer immunohistochemical images according to claim 2 or 3, characterized in that: The gland segmentation model specifically includes: The first downsampling submodule is connected to the first upsampling submodule through the first jump connection submodule; the second downsampling submodule is connected to the second upsampling submodule through the second jump connection submodule; the third downsampling submodule is connected to the third upsampling submodule through the third jump connection submodule; the fourth downsampling submodule is connected to the fourth upsampling submodule through the fourth jump connection submodule; so that the gland recognition result output by the gland segmentation model includes all feature maps that have undergone odd-numbered convolutions and feature maps that have undergone even-numbered convolutions.

5. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The parallel convolution module in the jump connection module includes a first 3×3 convolution layer, a first batch of normalization layers, a first Relu activation function, a second 3×3 convolution layer, a second batch of normalization layers, and a second Relu activation function connected in sequence.

6. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The multi-head self-attention mechanism module is used for feature dimension reduction, relative position encoding, and self-attention mechanism calculation.

7. The method for gland segmentation in colorectal cancer immunohistochemical images according to claim 1, characterized in that: The ECA module is used for global information aggregation, feature recalibration, and output weighted feature maps.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the colorectal cancer immunohistochemistry image gland segmentation method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for gland segmentation in colorectal cancer immunohistochemical images as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for gland segmentation in colorectal cancer immunohistochemical images described in any one of claims 1-7 is implemented.

Images