A pathological image classification method based on the interaction of cell and multi-scale tissue information

Abstract

The present invention discloses a pathological image classification method based on cell and multi-scale tissue information interaction, comprising the following steps: S1, segmenting the pathological image into cell nuclei and tissue blocks of different scales; S2, performing feature extraction on the cell nuclei and tissue blocks; S3, using the K-nearest neighbor algorithm to generate the connection relationship between the cell nuclei and tissue block edges, and modeling a pathological image into a cell map and three tissue maps of different scales; S4, constructing a multi-scale cell map and feature map information interaction model, performing convolution on the cell map, and performing attention interaction on the tissue map; S5, combining the tissue features after interactive learning with the cell features to classify the pathological image; this method can solve the problem that different data sets require tissue maps of different scales, as well as the problem of information loss and incomplete interaction of the same tissue map, effectively improving the performance of pathological image classification, and having high classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of biological technology, and in particular to a pathological image classification method based on cell and multi-scale tissue information interaction. Background Art

[0002] Medical pathology images are crucial for cancer diagnosis and analysis. They can determine the presence of cancer cells and the type and level of cancer. Unlike traditional natural images, medical pathology images have extremely high resolution, reaching millions of pixels. Manual automatic classification of pathology images is extremely labor-intensive and costly. Deep learning can help doctors automatically classify pathology images. Due to the high resolution of pathology images, traditional convolutional networks cannot directly process a complete image. Instead, a complete image can be simply cut into equal-sized patches, which are then trained end-to-end using GoogLeNet. This approach forcibly separates continuous semantic information by cutting patches, which results in the loss of continuous semantic information between patches during network learning. Furthermore, selecting the right patch size is challenging: small patches may not contain sufficient information, while large patches may contain redundant information. Furthermore, the situation varies across datasets. Therefore, for existing pathological image classification methods, the size of superpixel blocks has a great influence on the final decision for different data sets. Pixel blocks of a single scale cannot meet the general requirements of pathological image classification. Secondly, there is information loss in the combination of direct features and cell features of tissue images, and the classification accuracy is poor, which makes it difficult to meet people's usage needs. Summary of the Invention

[0003] The purpose of the present invention is to provide a pathological image classification method based on cell and multi-scale tissue information interaction. The pathological image classification method based on cell and multi-scale tissue information interaction can solve the problem that different data sets require tissue maps of different scales and the problem of information loss and incomplete interaction of the same tissue map, effectively improving the performance of pathological image classification and high classification accuracy.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A pathological image classification method based on cell and multi-scale tissue information interaction includes the following steps:

[0006] S1. Segment the pathological image into cell nuclei and tissue blocks of different scales;

[0007] S2, feature extraction of cell nuclei and tissue blocks;

[0008] S3. Use the K-nearest neighbor algorithm to generate the connection relationship between cell nuclei and tissue block edges, and model a pathological image into a cell map and three tissue maps at different scales;

[0009] S4. Construct a multi-scale cell map and feature map information interaction model, perform convolution on the cell map, and perform attention interaction on the tissue map;

[0010] S5. Combine the tissue features after interactive learning with the cell features to perform pathological image classification.

[0011] Preferably, the specific process of step S1 is:

[0012] S11. Segment cells from a pathological image using the HoverNet network to obtain cell nuclei.

[0013] S12. Use the SLIC superpixel segmentation algorithm to segment the same pathological image into three tissue blocks of different scales. Tissue maps of different scales contain information of different scales.

[0014] Preferably, the specific process of step S2 is:

[0015] S21, the image size of the segmented cell nucleus and tissue block is unified into 224*224;

[0016] S22. Input the cell nuclei and tissue blocks of uniform size into the resnet34 model pre-trained for natural images for feature extraction, and output the features of the cell nuclei and tissue blocks respectively. The output feature dimension is 512 dimensions, where the 512-dimensional feature represents each node feature of the cell nucleus or tissue nucleus.

[0017] Preferably, the specific process of step S3 is:

[0018] S31. Setting the parameter K of the K-nearest neighbor algorithm to 5, treating the cell nucleus or tissue mass as a node, calculating the Euclidean distance between the cell nucleus node and the tissue mass node and all remaining nodes, and finding the five nodes with the closest Euclidean distance;

[0019] S32. Connect each node to its five nearest nodes.

[0020] Preferably, the specific process of step S4 is:

[0021] S41, construct the network of cell branch and tissue branch respectively;

[0022] S42, the cell branch is responsible for the convolution of the cell map. The convolution method adopts SAGE convolution, and its convolution formula is:

[0023]

[0024] Among them, W1 and W2 are the matrix parameters that need to be learned, X i , refers to the features after integrating other node features, x i Refers to the node characteristics itself, x j Refers to the neighbor node characteristics, mean j∈N(i) Refers to the average weight;

[0025] S43, the organization branch inputs the organization block features into the Transformer encoder, uses the self-attention mechanism of the Transformer encoder to complete the information interaction of the organization block features, and outputs the organization features of the same size after interactive learning.

[0026] Preferably, the specific process of step S5 is: adding the interactive tissue block features of each scale to the corresponding cell node features, so that the features of the cell node are fused with the features of the tissue maps of different scales and receptive fields, and in the process of information aggregation, different scale tissue maps are given different weights and returned to the cell feature map, and finally the pathological image is classified by decision-making fusion information.

[0027] After adopting the above technical solution, the present invention has the following beneficial effects:

[0028] 1. The present invention converts a pathological image into a cell map and tissue maps of different scales, and uses the Transformer encoder to interact with information at the same scale. The information learned from the tissue map is combined with the cell map, and then end-to-end training is performed. This can solve the problem that different data sets require tissue maps of different scales, as well as the problem of information loss and incomplete interaction in the same tissue map, effectively improving the performance of pathological image classification with high classification accuracy.

[0029] 2. The present invention adopts a multi-scale tissue map design to solve the problem that pathological image classification of different data sets may require tissue maps of different scales. At the same time, it learns the weights of tissue map information at different scales combined with cell maps to effectively fuse information at different scales.

[0030] 3. The present invention uses the Transformer encoder to interact with tissue diagram information. For each scale of the tissue diagram, the Transformer encoder is used to interact with information, thereby realizing global information sharing and returning the global information of the tissue to the cell diagram. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 This is a display diagram showing the effect of making pathological images into cell maps and tissue maps;

[0033] Figure 3 It is a structural diagram of the information interaction model of the present invention;

[0034] Figure 4 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] Example

[0037] like Figures 1 to 4 As shown, a pathological image classification method based on cell and multi-scale tissue information interaction includes the following steps:

[0038] S1. Segment the pathological image into cell nuclei and tissue blocks of different scales;

[0039] The specific process of step S1 is:

[0040] S11. Segment cells from a pathological image using the HoverNet network to obtain cell nuclei.

[0041] S12. Use the SLIC superpixel segmentation algorithm to segment the same pathological image into three tissue blocks of different scales. Tissue maps of different scales contain information of different scales.

[0042] S2, feature extraction of cell nuclei and tissue blocks;

[0043] The specific process of step S2 is:

[0044] S21, the image size of the segmented cell nucleus and tissue block is unified into 224*224;

[0045] S22. Input the cell nuclei and tissue blocks of uniform size into the resnet34 model pre-trained for natural images for feature extraction, and output the features of the cell nuclei and tissue blocks respectively. The output feature dimension is 512 dimensions, where the 512-dimensional feature represents each node feature of the cell nucleus or tissue nucleus.

[0046] S3. Use the K-nearest neighbor algorithm to generate the connection relationship between cell nuclei and tissue block edges, and model a pathological image into a cell map and three tissue maps at different scales;

[0047] The specific process of step S3 is:

[0048] S31. Setting the parameter K of the K-nearest neighbor algorithm to 5, treating the cell nucleus or tissue mass as a node, calculating the Euclidean distance between the cell nucleus node and the tissue mass node and all remaining nodes, and finding the five nodes with the closest Euclidean distance;

[0049] S32. Connect each node to its five nearest nodes.

[0050] S4. Construct a multi-scale cell map and feature map information interaction model, perform convolution on the cell map, and perform attention interaction on the tissue map;

[0051] The specific process of step S4 is:

[0052] S41, construct the network of cell branch and tissue branch respectively;

[0053] S42, the cell branch is responsible for the convolution of the cell map. The convolution method adopts SAGE convolution, and its convolution formula is:

[0054]

[0055] Among them, W1 and W2 are the matrix parameters that need to be learned, X i , refers to the features after integrating other node features, x i Refers to the node characteristics itself, x j Refers to the neighbor node characteristics, mean j∈N(i) Refers to the average weight;

[0056] S43, the organization branch inputs the organization block features into the Transformer encoder, uses the self-attention mechanism of the Transformer encoder to complete the information interaction of the organization block features, and outputs the organization features of the same size after interactive learning.

[0057] S5. Combine the interactively learned tissue features with cell features to perform pathological image classification;

[0058] The specific process of step S5 is as follows: adding the interactive tissue block features of each scale to the corresponding cell node features, fusing the cell node features with the tissue map features of different scales and receptive fields, and assigning different weights to the tissue maps of different scales and returning them to the cell feature map during the information aggregation process, and finally classifying the pathological image through the decision-making fusion information.

[0059] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A pathological image classification method based on cell and multi-scale tissue information interaction, characterized in that: The following steps are involved: S1. Segment the pathological image into cell nuclei and tissue blocks of different scales; S2, feature extraction of cell nuclei and tissue blocks; S3. Use the K-nearest neighbor algorithm to generate the connection relationship between cell nuclei and tissue block edges, and model a pathological image into a cell map and three tissue maps at different scales; The specific process of step S3 is: S31. Setting the parameter K of the K-nearest neighbor algorithm to 5, treating the cell nucleus or tissue mass as a node, calculating the Euclidean distance between the cell nucleus node and the tissue mass node and all the remaining nodes, and finding the five nodes with the closest Euclidean distance; S32, connect each node to its five nearest nodes; S4. Construct a multi-scale cell map and feature map information interaction model, perform convolution on the cell map, and perform attention interaction on the tissue map; The specific process of step S4 is: S41, construct the network of cell branch and tissue branch respectively; S42, the cell branch is responsible for the convolution of the cell map. The convolution method adopts SAGE convolution, and its convolution formula is: Among them, W1 and W2 are the matrix parameters that need to be learned, X' i Refers to the features after integrating other node features, x i Refers to the node characteristics itself, x j Refers to the neighbor node characteristics, mean j∈N(i) Refers to the average weight; S43, the organization branch inputs the tissue block features into the Transformer encoder, uses the self-attention mechanism of the Transformer encoder to complete the information interaction of the tissue block features, and outputs the tissue features of the same size after interactive learning; S5. Combine the interactively learned tissue features with cell features to perform pathological image classification; The specific process of step S5 is as follows: adding the interactive tissue block features of each scale to the corresponding cell node features, fusing the cell node features with the tissue map features of different scales and receptive fields, and assigning different weights to the tissue maps of different scales and returning them to the cell feature map during the information aggregation process, and finally classifying the pathological image through the decision-making fusion information.

2. A pathological image classification method based on cell and multi-scale tissue information interaction as claimed in claim 1, characterized in that: The specific process of step S1 is: S11. Segment cells from a pathological image using the HoverNet network to obtain cell nuclei. S12. Use the SLIC superpixel segmentation algorithm to segment the same pathological image into three tissue blocks of different scales. Tissue maps of different scales contain information of different scales.

3. The pathological image classification method based on cell and multi-scale tissue information interaction according to claim 2, characterized in that: The specific process of step S2 is: S21, the image size of the segmented cell nucleus and tissue block is unified into 224*224; S22. Input the cell nuclei and tissue blocks of uniform size into the resnet34 model pre-trained for natural images for feature extraction, and output the features of the cell nuclei and tissue blocks respectively. The output feature dimension is 512 dimensions, where the 512-dimensional feature represents each node feature of the cell nucleus or tissue nucleus.

Images