Non-small cell lung cancer h&e staining image multiple tissue segmentation method, system and storage medium
By using threshold segmentation and Gaussian blur to generate tissue masks, combined with a bilinear convolution model and attention mechanism layer, the problem of high time and labor costs in the segmentation of various tissues in H&E staining images of non-small cell lung cancer is solved, and efficient automated segmentation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG GENERAL HOSPITAL
- Filing Date
- 2021-12-31
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies suffer from high time and labor costs in the process of segmenting various tissues in H&E staining images of non-small cell lung cancer.
Threshold segmentation and Gaussian blur techniques are used to generate tissue masks. Combined with a bilinear convolution model and attention mechanism layer, tissue type identification and coloring are performed by dividing the image into sub-image blocks. ResNet50 is used for pre-training parameter transfer to improve the model's generalization ability.
It significantly reduces computational load and manual intervention, improves the accuracy and efficiency of tissue type identification, and enables automated segmentation of various tissues in H&E-stained images of non-small cell lung cancer.
Smart Images
Figure CN114494288B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of digital image processing, specifically relating to a method, system, and storage medium for segmenting multiple tissues in hematoxylin-eosin (H&E) stained images of non-small cell lung cancer. Background Technology
[0002] Lung cancer has the highest mortality rate among all cancers, with non-small cell lung cancer accounting for approximately 75% of all lung cancers. Hematologic and epithelial (H&E) staining pathological image analysis is one of the routine methods and tools used by pathologists in the diagnosis of non-small cell lung cancer. Pathological WSI (Wastewater Injection) with its various tissue segmentations and area calculations is beneficial in assisting pathologists in the precise quantification and analysis of the disease, and has significant clinical and scientific value. Summary of the Invention
[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method, system and storage medium for segmenting multiple tissues in H&E stained images of non-small cell lung cancer. This invention can realize automated segmentation of multiple tissues in H&E stained images of non-small cell lung cancer, which can greatly reduce the time and labor costs of segmenting multiple tissues in H&E stained images of non-small cell lung cancer compared with manual processes.
[0004] To achieve the above objectives, the present invention adopts the following technical solution:
[0005] This invention provides a method for segmenting multiple tissues in H&E-stained images of non-small cell lung cancer, comprising the following steps.
[0006] Tissue type classification determines the distribution of tissue types in H&E-stained digital pathological images of non-small cell lung cancer. Tissue types are classified as: tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchi, blood vessels, necrotic areas, black carbon dust areas, and others.
[0007] Tissue type coloring: Based on the distribution of different tissue types in the H&E-stained digital pathological images of non-small cell lung cancer, pixel areas of different tissue types are colored to distinguish different tissue types and highlight the visual expression of different tissue types.
[0008] WSI pathology segmentation uses multiple tissue types to integrate the pixel regions of the stained tissue types in the H&E-stained digital pathology image of non-small cell lung cancer, thereby achieving tissue region localization and completing tissue segmentation.
[0009] As a preferred technical solution, the steps for classifying tissue types are as follows:
[0010] The image is divided into sub-image blocks. Threshold segmentation and Gaussian blurring techniques are used on the pathological WSI to generate a tissue mask. The tissue mask is a tissue region mask of the high-resolution pathological image. The background region is filtered out and only the tissue region is retained. According to the tissue region, a sliding window with overlapping regions is used to divide the image within the tissue region into several RGB sub-image blocks.
[0011] Sub-image patch recognition employs a pre-trained bilinear convolutional model to identify RGB sub-image patches. This model includes a ResNet50 layer, bilinear pooling layers, and an attention mechanism layer. The ResNet50 layer extracts end-to-end features from H&E-stained digital pathology images. The deep learning features extracted from the last convolutional layer are input into the bilinear pooling layer. Upon receiving these features, the bilinear pooling layer captures fine-grained features of non-small cell lung cancer tissue types and extracts highly localized features from the H&E-stained digital pathology images. The attention mechanism layer, upon receiving the fine-grained features extracted from the bilinear pooling layer, assigns attention weights to these features, allowing the model to focus its learning on more critical features and more discriminative local features, thereby improving the model's ability to identify tissue types.
[0012] As a preferred technical solution, the bilinear pooling layer is a bilinear structure, specifically:
[0013] y = b + w T x
[0014] y is the output of the model, b is the bias term, and w∈R n This is the corresponding transpose weight matrix, where n is the dimension of the input features;
[0015] The bilinear pooling layer is:
[0016] y = b + w T x+x T F T Fx
[0017] F∈R k×n It contains The weight matrix of the factors.
[0018] Preferably, the attention mechanism layer specifically comprises:
[0019]
[0020]
[0021] e i =wf i
[0022] c is the feature assigned attention weight, ai For attention weights, e i Let w be the attention score, and w be a learnable parameter.
[0023] As a preferred technical solution, the bilinear convolution model is pre-trained in the following manner:
[0024] Ten different tissue types were labeled on the H&E-stained digital pathology images of non-small cell lung cancer used for training: tumor epithelium, tumor stroma, background region, lymphocytes, normal tissue, bronchus, blood vessels, necrotic region, black carbon dust region, and others. Then, at a magnification of M times, several image patches of a set size were taken from the labeled region of each tissue type, and then the patches were scaled to the set size using bilinear interpolation as training samples. Finally, the training samples were used to train the pre-trained bilinear convolution model.
[0025] The pre-training process is used to assign initial parameters to the bilinear convolutional model. Pre-training is performed using publicly available tissue datasets to improve the model's performance on multiple tissue datasets for non-small cell lung cancer. ResNet50 is used for pre-training on multiple tissue datasets for colorectal cancer. The pre-trained parameters are then transferred to the non-small cell lung cancer classification model for parameter initialization. The convolutional layers are not frozen, and retraining is performed to improve the model's generalization ability and training efficiency on multiple tissue datasets for non-small cell lung cancer.
[0026] As a preferred technical solution, the steps for coloring the tissue type are as follows:
[0027] Based on the tissue type present in tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchi, blood vessels, necrotic areas, black carbon dust areas, and other areas, the sub-image blocks of the tissue type to be identified are divided into ten categories. The tissues present in tumor epithelial areas, tumor stroma areas, background areas, lymphocyte areas, normal tissue areas, bronchial areas, blood vessel areas, necrotic areas, black carbon dust contamination areas, and other areas are represented by red, orange, white, dark blue, green, yellow, purple, black, brown, and light blue, respectively.
[0028] As a preferred technical solution, the steps of the pathological WSI multiple tissue segmentation are as follows:
[0029] Error elimination: Eliminates the influence of coloring conflicts in overlapping areas of sub-image blocks;
[0030] The tissue regions are summarized, and the red tumor epithelial region, orange tumor stroma region, white background region, dark blue lymphocyte region, green normal tissue region, yellow bronchial region, purple blood vessel region, black necrotic region, and brown black carbon dust contamination region are mapped onto the tissue mask obtained in the sub-image block segmentation module according to their spatial coordinate relationships.
[0031] As a preferred technical solution, a sliding window is used to traverse the selected range, and each traversed sub-image block is input into a trained bilinear convolutional model to generate a prediction probability. The tissue type with the highest prediction probability is selected as the tissue type of the image block.
[0032] Another aspect of the present invention provides a multi-tissue segmentation system for H&E staining images of non-small cell lung cancer, which is applied to the aforementioned multi-tissue segmentation method for H&E staining images of non-small cell lung cancer, including a tissue type classification module, a tissue type coloring module, and a pathological WSI multi-tissue segmentation module;
[0033] The tissue type classification module is used to determine the distribution of tissue types in H&E-stained digital pathological images of non-small cell lung cancer. Tissue types are classified as: tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchus, blood vessels, necrotic area, black carbon dust area, and others.
[0034] The tissue type coloring module is used to color the pixel regions of different tissue types according to the distribution of different tissue types in the H&E staining digital pathological image of non-small cell lung cancer, so as to distinguish different tissue types and highlight the visual expression of different tissue types.
[0035] The WSI pathology module is used to integrate the pixel regions of the colored tissue types in the H&E-stained digital pathology image of non-small cell lung cancer according to the distribution of tissue types, thereby achieving tissue region localization and completing tissue segmentation.
[0036] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the aforementioned method for multiple tissue segmentation of H&E staining images of non-small cell lung cancer.
[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0038] 1. In the process of dividing the image into sub-image blocks, this invention employs techniques such as threshold segmentation and Gaussian blurring for pathological WSI to generate a tissue mask. This tissue mask is a tissue region mask for high-resolution pathological images. Compared with other studies, this tissue mask can filter out most of the background area, retaining only the tissue region, significantly reducing the computational load. The algorithm divides the image within the tissue region into several 224*224 sub-image blocks using a sliding window containing overlapping areas. For overlapping areas, this invention develops an error elimination stage to address the staining error caused by overlapping staining in these areas.
[0039] 2. The sub-image patch recognition developed in this invention, compared with other studies, introduces a bilinear pooling layer and an attention mechanism layer, combining the two. This allows the invention to possess both the powerful feature extraction capabilities of the bilinear pooling layer and the advantage of the attention mechanism layer in focusing on key features. The bilinear pooling layer breaks through the limitations of traditional deep learning models, which are limited to linear transformations. It can capture nonlinear features and extract highly localized, fine-grained image features, thus better handling fine-grained image classification problems. The attention mechanism layer is differentiable; a differentiable attention mechanism layer can calculate gradients through neural networks and obtain attention weights through forward propagation and backward feedback. The fine-grained features extracted from the bilinear pooling layer are input into the attention mechanism layer, thereby assigning attention weights to the extracted features. This allows the model to focus its learning on more critical features, concentrating on more discriminative local features, and improving the model's ability to identify tissue types.
[0040] 3. This invention utilizes existing research results and uses publicly available colorectal cancer multi-tissue datasets for pre-training. The pre-trained parameters are then transferred to the bilinear convolutional model in the sub-image patch recognition module of this invention, which accelerates the training of the lung cancer fine-grained multi-tissue classification model and significantly improves the accuracy of the lung cancer multi-tissue fine-grained classification model. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is an overall flowchart of various tissue segmentation methods for H&E staining images of non-small cell lung cancer in this embodiment of the invention;
[0043] Figures 2(a) and 2(b) are structural diagrams of the bilinear convolution model according to an embodiment of the present invention;
[0044] Figure 3 This is a WSI segmentation diagram of the pathology results of this invention;
[0045] Figure 4 This is a schematic diagram of the structure of the multi-tissue segmentation system for H&E staining images of non-small cell lung cancer according to the present invention;
[0046] Figure 5 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation
[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0049] like Figure 1 As shown in this embodiment, various tissue segmentation methods for H&E-stained images of non-small cell lung cancer include the following steps.
[0050] S1. Tissue type classification to determine the distribution of tissue types in H&E-stained digital pathology images of non-small cell lung cancer;
[0051] S2. Tissue type coloring: Based on the distribution of different tissue types in the H&E-stained digital pathological images of non-small cell lung cancer, the pixel areas of different tissue types are colored to distinguish different tissue types and highlight the visual expression of different tissue types.
[0052] S3, Pathology WSI Multiple Tissue Segmentation: Based on the distribution of tissue types in the H&E-stained digital pathology image of non-small cell lung cancer, the pixel regions of the stained tissue types are integrated together to achieve tissue region localization and complete tissue segmentation.
[0053] In this invention, the H&E-stained digital pathological image of non-small cell lung cancer is segmented into several sub-image blocks. A designed bilinear convolution model is then used to classify the tissues in each sub-image block. Finally, the results are summarized to obtain the distribution of various tissues in the H&E-stained digital pathological image of non-small cell lung cancer, as detailed below:
[0054] The H&E-stained digital pathological image of non-small cell lung cancer was processed to obtain a tissue mask of H&E-stained digital pathological image that is one-sixteenth the size of the original image.
[0055] Find the largest bounding rectangle of the mask. This rectangle is the range within which sub-image blocks are selected for cutting. Within this range, a sliding window is used to cut sub-image blocks, thereby filtering out most of the background.
[0056] The sliding window iterates through the selected area, and each sub-image patch is input into the trained bilinear convolution model to generate prediction probabilities. The tissue type with the highest prediction probability is selected as the tissue type of the image patch.
[0057] As shown in Figures 2(a) and 2(b), the bilinear convolutional model developed for the classification of multiple tissues in H&E staining images of non-small cell lung cancer in this embodiment is as follows:
[0058] 1) The first part consists of one CNN layer, one max pooling layer, four BLOCK1 modules, and eleven BLOCK2 modules; the BLOCK1 module consists of three CNN layers and one residual CNN layer; the BLOCK2 module consists of three CNN layers and residual connections.
[0059] 2) The second part is a bilinear pooling layer, which is used to receive the feature vector output from the first part;
[0060] 3) The third part is the attention mechanism layer, which assigns attention weights to the feature vectors output by the second part;
[0061] 4) Subsequently, the feature vectors processed by the Dropout layer are input into the fully connected layer, and the SoftMax function is used to perform ten classifications.
[0062] Furthermore, in step S1, the tissue type classification step specifically includes:
[0063] S1.1 Divide the image into sub-image blocks. Use threshold segmentation and Gaussian blurring techniques on the pathological WSI to generate a tissue mask. This tissue mask is a tissue region mask of the high-resolution pathological image. It filters out most of the background region and only retains the tissue region. Based on the tissue region, use a sliding window with overlapping regions to divide the image within the tissue region into several sub-image blocks of size 224*224.
[0064] S1.2 Sub-image patch recognition: To recognize sub-image patches, training samples must first be obtained and a bilinear convolution model trained for classification. In this embodiment, pathology experts have marked 10 different tissue types on the H&E-stained digital pathology images of non-small cell lung cancer used for training. These are tumor epithelium, tumor stroma, background region, lymphocytes, normal tissue, bronchus, blood vessels, necrotic region, black carbon dust region, and other regions. Then, several 224*224 sub-image patches are selected from the marked tissue regions as training samples. The training samples are then input into the proposed bilinear convolution model for model training, and deep learning features are extracted to form feature vectors.
[0065] Furthermore, the bilinear convolutional model includes: ResNet50, bilinear pooling layers, and attention mechanism layers;
[0066] The ResNet50 can extract end-to-end features from H&E stained digital pathology images. In this invention, the features extracted from the last convolutional layer are input into a bilinear pooling layer.
[0067] The bilinear pooling layer is a bilinear structure that breaks through the limitations of traditional deep learning models, which are confined to linear transformations. It can capture nonlinear features and extract highly localized, fine-grained image features, thus better handling fine-grained image classification problems. The deep learning features extracted by RensNet50 are input into the bilinear pooling layer to capture fine-grained features of non-small cell lung cancer tissue types and extract highly localized features from H&E-stained digital pathology images. The linear structure is specifically as follows:
[0068] y = b + w T x
[0069] y is the output of the model, b is the bias term, and w∈R n is the corresponding transpose weight matrix, and n is the dimension of the input features.
[0070] The bilinear pooling layer is:
[0071] y = b + w T x+x T F T Fx
[0072] F∈R k×n It contains The weight matrix of the factors.
[0073] The attention mechanism layer is a soft attention mechanism, which is differentiable. A differentiable attention mechanism can calculate gradients through a neural network and obtain attention weights through forward propagation and backward feedback. Fine-grained features extracted from the bilinear pooling layer are input into the attention mechanism layer, thereby assigning attention weights to the extracted features. This allows the model to focus its learning on more critical features, concentrating on more discriminative local features, thus improving the model's ability to identify tissue types. Specifically, the attention mechanism layer is as follows:
[0074]
[0075]
[0076] e i =wf i
[0077] c is the feature assigned attention weight, a i For attention weights, e i Let w be the attention score, and w be a learnable parameter.
[0078] The bilinear convolution model is trained in the following manner:
[0079] Ten different tissue types have been labeled on the H&E-stained digital pathology images of non-small cell lung cancer used for training by professional pathologists. These types are tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchus, blood vessels, necrotic area, black carbon dust area, and others. Then, at a magnification of 40 times, several 512*512 image patches are taken from the labeled areas of each tissue type. These patches are then scaled to 224*224 using bilinear interpolation as training samples. Finally, the training samples are used to train the pre-trained bilinear convolutional model.
[0080] The pre-training process is used to assign initial parameters to the bilinear convolutional model. Pre-training with publicly available tissue datasets improves the model's performance on various non-small cell lung cancer tissue datasets. ResNet50 is used for pre-training on a colorectal cancer tissue dataset. The pre-trained parameters are then transferred to the non-small cell lung cancer tissue classification model for parameter initialization. Convolutional layers are not frozen, and retraining is performed, thereby improving the model's generalization ability and training efficiency on various non-small cell lung cancer tissue datasets.
[0081] Using this bilinear convolution model, the tissue type is determined to exist in tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchi, blood vessels, necrotic areas, black carbon dust areas, and other areas. The sub-image blocks of the tissue type to be identified are then further subdivided into ten classes to complete the tissue type classification.
[0082] Based on tissue type, the tissues present in the tumor epithelial region, tumor stroma region, background region, lymphocyte region, normal tissue region, bronchial region, vascular region, necrotic region, black carbon dust contamination region, and other regions are represented by red, orange, white, dark blue, green, yellow, purple, black, brown, and light blue, respectively.
[0083] Furthermore, the tissue type coloring in step S2 is specifically as follows:
[0084] Based on the tissue type present in tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchi, blood vessels, necrotic areas, black carbon dust areas, and other areas, the sub-image blocks of the tissue type to be identified are divided into ten categories. The tissues present in tumor epithelium, tumor stroma, background area, lymphocyte area, normal tissue area, bronchial area, blood vessel area, necrotic area, black carbon dust contaminated area, and other areas are represented by red, orange, white, dark blue, green, yellow, purple, black, brown, and light blue, respectively.
[0085] Furthermore, in step S3, the pathological WSI involves multiple tissue segmentations, specifically:
[0086] S31. Error elimination: Eliminate the influence of coloring conflicts in overlapping areas of sub-image blocks;
[0087] S32. Tissue region summarization: The red tumor epithelial region, orange tumor stroma region, white background region, dark blue lymphocyte region, green normal tissue region, yellow bronchial region, purple blood vessel region, black necrotic region, and brown black carbon dust contamination region are mapped onto the tissue mask obtained in the sub-image block segmentation module according to their spatial coordinate relationships.
[0088] In summary, the implementation steps for performing various tissue segmentation in H&E-stained digital pathological images of non-small cell lung cancer in this embodiment are as follows:
[0089] 1) Divide the H&E-stained digital pathology image of lung cancer into several sub-image blocks of size 224*224;
[0090] 2) Identify tissue types, including dividing training samples, extracting deep learning features, training a bilinear convolutional model, and identifying tissue types;
[0091] 3) Tissue staining: Based on the tissue type classification results, stain the tissue areas;
[0092] 4) Error elimination: Use the classifier to compare the predicted probability values, select the tissue category with the highest probability value as the tissue category of the overlapping part, and color it.
[0093] 5) Organize the regions and map them onto the organization mask according to the spatial coordinate relationship.
[0094] The final pathological WSI segmentation result image is as follows Figure 3 As shown.
[0095] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.
[0096] Based on the same concept as the multi-tissue segmentation method for H&E-stained images of non-small cell lung cancer in the above embodiments, the present invention also provides a multi-tissue segmentation system for H&E-stained images of non-small cell lung cancer. This system can be used to perform the above-described multi-tissue segmentation method for H&E-stained images of non-small cell lung cancer. For ease of explanation, the structural schematic diagram of the embodiment of the multi-tissue segmentation system for H&E-stained images of non-small cell lung cancer only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.
[0097] like Figure 4 As shown, another embodiment of the present invention provides a multi-tissue segmentation system for H&E staining images of non-small cell lung cancer, including a tissue type classification module, a tissue type staining module, and a pathological WSI multi-tissue segmentation module;
[0098] The tissue type classification module is used to determine the distribution of tissue types in H&E-stained digital pathology images of non-small cell lung cancer.
[0099] Furthermore, the tissue type classification module includes: a sub-image block segmentation module and a sub-image block recognition module;
[0100] The sub-image block segmentation module uses techniques such as threshold segmentation and Gaussian blurring on the pathological WSI to generate a tissue mask. This tissue mask is a tissue region mask for high-resolution pathological images, filtering out most of the background area and retaining only the tissue region. Based on the tissue region, a sliding window with overlapping areas is used to divide the image within the tissue region into several RGB sub-image blocks;
[0101] The sub-image recognition module uses the bilinear convolution model designed in this embodiment to recognize image blocks after training.
[0102] The tissue type coloring module is used to color the pixel regions of different tissue types according to the distribution of different tissue types in the H&E staining digital pathological image of non-small cell lung cancer, so as to distinguish different tissue types and highlight the visual expression of different tissue types.
[0103] The WSI pathology module is used to integrate the pixel regions of the colored tissue types in the H&E-stained digital pathology image of non-small cell lung cancer according to the distribution of tissue types, so as to realize tissue region localization and complete tissue segmentation.
[0104] Furthermore, the pathological WSI multi-tissue segmentation module includes: an error elimination module and a tissue region summarization module;
[0105] The error elimination module is used to eliminate the influence of coloring conflicts in the overlapping areas of sub-image blocks;
[0106] The tissue region summarization module is used to map the red tumor epithelial region, orange tumor stroma region, white background region, dark blue lymphocyte region, green normal tissue region, yellow bronchial region, purple blood vessel region, black necrotic region, and brown carbon dust contamination region onto the tissue mask obtained in the sub-image block segmentation module according to spatial coordinate relationships.
[0107] It should be noted that the multi-tissue segmentation system for H&E-stained images of non-small cell lung cancer of the present invention corresponds one-to-one with the multi-tissue segmentation method for H&E-stained images of non-small cell lung cancer of the present invention. The technical features and beneficial effects described in the embodiments of the multi-tissue segmentation method for H&E-stained images of non-small cell lung cancer of the above invention are applicable to the embodiments of the multi-tissue segmentation system for H&E-stained images of non-small cell lung cancer. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.
[0108] Furthermore, in the embodiments of the non-small cell lung cancer H&E staining image multi-tissue segmentation system described above, the logical division of each program module is merely illustrative. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the non-small cell lung cancer H&E staining image multi-tissue segmentation system can be divided into different program modules to complete all or part of the functions described above.
[0109] like Figure 5 As shown, in one embodiment, a computer-readable storage medium is provided, storing a program in a memory. When the program is executed by a processor, it implements the aforementioned method for multiple tissue segmentation of H&E staining images of non-small cell lung cancer, specifically:
[0110] Tissue type classification to determine the distribution of tissue types in H&E-stained digital pathology images of non-small cell lung cancer;
[0111] Tissue type coloring: Based on the distribution of different tissue types in the H&E-stained digital pathological images of non-small cell lung cancer, pixel areas of different tissue types are colored to distinguish different tissue types and highlight the visual expression of different tissue types.
[0112] WSI pathology segmentation uses multiple tissue types to integrate the pixel regions of the stained tissue types in the H&E-stained digital pathology image of non-small cell lung cancer, thereby achieving tissue region localization and completing tissue segmentation.
[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to computer program instructions. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0115] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for multiple tissue segmentation of non-small cell lung cancer H&E staining images, characterized in that, Includes the following steps Tissue type classification determines the distribution of tissue types in H&E-stained digital pathological images of non-small cell lung cancer. Tissue types are classified as: tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchi, blood vessels, necrotic areas, black carbon dust areas, and others. Tissue type coloring: Based on the distribution of different tissue types in the H&E-stained digital pathological images of non-small cell lung cancer, pixel areas of different tissue types are colored to distinguish different tissue types and highlight the visual expression of different tissue types. WSI pathology segmentation uses multiple tissue types to integrate the pixel regions of the stained tissue types in the H&E-stained digital pathology image of non-small cell lung cancer, thereby locating the tissue regions and completing tissue segmentation. The specific steps for classifying organization types are as follows: The image is divided into sub-image blocks. Threshold segmentation and Gaussian blurring techniques are used on the pathological WSI to generate a tissue mask. The tissue mask is a tissue region mask of the high-resolution pathological image. The background region is filtered out and only the tissue region is retained. According to the tissue region, a sliding window with overlapping regions is used to divide the image within the tissue region into several RGB sub-image blocks. Sub-image patch recognition employs a pre-trained bilinear convolutional model to identify RGB sub-image patches. The bilinear convolutional model includes ResNet50, bilinear pooling layers, and an attention mechanism layer. The ResNet50 is used to extract end-to-end features from H&E-stained digital pathology images. The deep learning features extracted by the last convolutional layer are input into the bilinear pooling layer. After receiving the deep learning features, the bilinear pooling layer captures fine-grained features of non-small cell lung cancer tissue types and extracts highly local features of H&E-stained digital pathology images. After receiving the fine-grained features extracted from the bilinear pooling layer, the attention mechanism layer extracts the features and assigns them attention weights, thereby allowing the model to focus its learning on more critical features and concentrate on more discriminative local features, thus improving the model's ability to identify organizational types. The bilinear pooling layer is a bilinear structure, and the linear structure is specifically as follows: is an output of the model, is a bias term, is a corresponding transpose weight matrix, is a dimension of the input features; The bilinear pooling layer is: It contains The weight matrix of the factors; The attention mechanism layer is specifically as follows: , These are features that are assigned attention weights. For attention weights, Let w be the attention score, and w be a learnable parameter.
2. The method for segmenting multiple tissues in H&E-stained images of non-small cell lung cancer according to claim 1, characterized in that, The bilinear convolutional model is pre-trained in the following manner: Ten different tissue types were labeled on the H&E-stained digital pathology images of non-small cell lung cancer used for training: tumor epithelium, tumor stroma, background region, lymphocytes, normal tissue, bronchus, blood vessels, necrotic region, black carbon dust region, and others. Then, at a magnification of M times, several image patches of a set size were taken from the labeled region of each tissue type, and then the patches were scaled to the set size using bilinear interpolation as training samples. Finally, the training samples were used to train the pre-trained bilinear convolution model. The pre-training process is used to assign initial parameters to the bilinear convolutional model. Pre-training is performed using publicly available tissue datasets to improve the model's performance on multiple tissue datasets for non-small cell lung cancer. ResNet50 is used for pre-training on multiple tissue datasets for colorectal cancer. The pre-trained parameters are then transferred to the non-small cell lung cancer classification model for parameter initialization. The convolutional layers are not frozen, and retraining is performed to improve the model's generalization ability and training efficiency on multiple tissue datasets for non-small cell lung cancer.
3. The method for segmenting multiple tissues in H&E staining images of non-small cell lung cancer according to claim 1, characterized in that, The specific steps for coloring the tissue type are as follows: Based on the tissue type present in tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchi, blood vessels, necrotic areas, black carbon dust areas, and other areas, the sub-image blocks of the tissue type to be identified are divided into ten categories. The tissues present in tumor epithelial areas, tumor stroma areas, background areas, lymphocyte areas, normal tissue areas, bronchial areas, blood vessel areas, necrotic areas, black carbon dust contamination areas, and other areas are represented by red, orange, white, dark blue, green, yellow, purple, black, brown, and light blue, respectively.
4. The method for segmenting multiple tissues in H&E staining images of non-small cell lung cancer according to claim 3, characterized in that, The specific steps for multiple tissue segmentation in the pathological WSI are as follows: Error elimination: Eliminates the influence of coloring conflicts in overlapping areas of sub-image blocks; The tissue regions are summarized, and the red tumor epithelial region, orange tumor stroma region, white background region, dark blue lymphocyte region, green normal tissue region, yellow bronchial region, purple blood vessel region, black necrotic region, and brown black carbon dust contamination region are mapped onto the tissue mask obtained in the sub-image block segmentation module according to their spatial coordinate relationships.
5. The method for segmenting multiple tissues in H&E-stained images of non-small cell lung cancer according to claim 2, characterized in that, The sliding window iterates through the selected area, and each sub-image patch is input into the trained bilinear convolution model to generate prediction probabilities. The tissue type with the highest prediction probability is selected as the tissue type of the image patch.
6. A multi-tissue segmentation system for H&E staining images of non-small cell lung cancer, characterized in that, A multi-tissue segmentation method for non-small cell lung cancer H&E staining images according to any one of claims 1-5, including a tissue type classification module, a tissue type coloring module, and a pathological WSI multi-tissue segmentation module; The tissue type classification module is used to determine the distribution of tissue types in H&E-stained digital pathological images of non-small cell lung cancer. Tissue types are classified as: tumor epithelium, tumor stroma, background area, lymphocytes, normal tissue, bronchus, blood vessels, necrotic area, black carbon dust area, and others. The tissue type coloring module is used to color the pixel regions of different tissue types according to the distribution of different tissue types in the H&E staining digital pathological image of non-small cell lung cancer, so as to distinguish different tissue types and highlight the visual expression of different tissue types. The WSI pathology module is used to integrate the pixel regions of the colored tissue types in the H&E-stained digital pathology image of non-small cell lung cancer according to the distribution of tissue types, thereby achieving tissue region localization and completing tissue segmentation.
7. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the multiple tissue segmentation method for H&E staining images of non-small cell lung cancer as described in any one of claims 1-5.