H&e stained histopathology image cell nucleus segmentation and classification system, method, device and medium

Abstract

The application discloses a kind of H&E staining histopathological image cell nucleus segmentation and classification system, method, equipment and medium, system includes multi-task deep learning module, multi-task related attention mechanism module, cost-sensitive learning strategy module, two-step watershed algorithm post-processing module;The multi-task deep learning module is used to carry out cell nucleus semantic segmentation, instance segmentation and cell nucleus classification respectively;The multi-task related attention mechanism module is used to fuse the feature representation of cell nucleus segmentation module, cell nucleus instance segmentation module and cell nucleus classification module, to utilize global context information;The cost-sensitive learning strategy module is used to effectively solve the problem that the model performance is reduced due to the serious imbalance of the number of different cell types samples;The two-step watershed algorithm post-processing module is used to more accurately segment the cell nucleus with great difference in size and morphology.The application can realize H&E staining histopathology image cell nucleus segmentation and classification task.

Description

Technical Field

[0001] This invention belongs to the technical field of digital image processing, specifically relating to a system, method, device, and medium for segmenting and classifying cell nuclei in H&E-stained tissue pathological images. Background Technology

[0002] Cancer seriously endangers human health, and pathological diagnosis is the "gold standard" for determining the malignancy of tumors. H&E staining is a crucial step in preparing histopathological sections, revealing the tumor's tissue structure and cellular components. Cell nuclear segmentation and classification are necessary prerequisites and fundamental principles for digital pathological image analysis. Existing cell nuclear segmentation methods include region-based image segmentation, which divides the image into multiple regions based on its spatial characteristics. Common region-based image segmentation methods include watershed methods and split-merge methods. In region-based image segmentation, pixels within a region share similar gradient features, texture features, and pixel grayscale value features. The target pixel in these methods can be a small region of multiple pixels with numerous similar features, or it can be a single pixel. Different target pixel features will result in different pixels, affecting the final performance of the region growing method. Region growing methods generally achieve better segmentation results when the pixel grayscale values ​​of the image are uniformly distributed and have little difference. However, this method has certain drawbacks: First, it requires pre-setting similarity features during segmentation, which introduces a degree of uncertainty; second, it is highly sensitive to noise; and third, when the image has complex texture information and a large number of pixels, the segmentation rate drops significantly, and the accuracy of image segmentation is also affected. In clinical diagnosis, even small proportions of cells play a crucial role. Therefore, accurately segmenting and classifying cell nuclei of various categories, especially those representing smaller proportions in sample images, has significant clinical and scientific value. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a cell nucleus segmentation and classification system, method, device, and medium for H&E-stained tissue pathological images. This invention comprises a multi-task feature fusion framework consisting of three independent encoders and decoders. Furthermore, a cost-sensitive learning strategy is introduced to effectively address the problem of imbalanced sample numbers for different cell types in the learning samples.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] This invention provides a cell nucleus segmentation and classification system for H&E-stained tissue pathological images, including a multi-task deep learning module, a multi-task related attention mechanism module, a cost-sensitive learning strategy module, and a two-step watershed algorithm post-processing module.

[0006] The multi-task deep learning module includes three sub-task modules: a cell nucleus segmentation module, a cell nucleus instance segmentation module, and a cell nucleus classification module. The input images of the cell nucleus segmentation module, the cell nucleus instance segmentation module, and the cell nucleus classification module are processed through an encoding structure, a multi-task related attention mechanism module, and a decoding structure to obtain cell nucleus segmentation results, instance segmentation results, and cell nucleus classification results, respectively.

[0007] The cell nucleus segmentation module is used to distinguish whether the pixels of the input image belong to the cell nucleus or not.

[0008] The cell nucleus instance segmentation module is used to generate horizontal and vertical distance maps for instance segmentation, which helps to perform instance segmentation on the semantic segmentation results of the previous module;

[0009] The cell nucleus classification module is used to obtain the classification map corresponding to the histopathological image, which helps to complete the cell nucleus classification based on the instance segmentation results of the first two modules; the prediction result of the cell nucleus classification module corresponds to the category with the highest pixel ratio in the cell nucleus instance;

[0010] The multi-task related attention mechanism module is used to fuse the feature representations of the cell nucleus segmentation module, the cell nucleus instance segmentation module, and the cell nucleus classification module to realize the utilization of global context information.

[0011] The cost-sensitive learning strategy module is used to effectively solve the problem of model performance degradation caused by a severe imbalance in the number of samples of different cell types.

[0012] The two-step watershed algorithm post-processing module is used to more accurately segment cell nuclei that differ greatly in size, color, and shape.

[0013] As a preferred technical solution, the multi-task related attention mechanism module utilizes a feature interaction mechanism based on multi-task branches, specifically as follows:

[0014] The three sub-task modules first pass through their respective encoders and then connect to the multi-task related attention mechanism module;

[0015] The three sub-task modules output to their respective decoders and execute their respective tasks. Each of the three sub-task modules has its own parameters, while the parameters in the multi-task related attention mechanism module are shared by all sub-task modules. Due to the role of the multi-task related attention mechanism module, each sub-task module not only learns its own task features but also learns the relevant task features of other sub-task modules.

[0016] As a preferred technical solution, the cost-sensitive learning strategy module mitigates the impact of sample imbalance and improves the classification accuracy of cell nuclei, which have a small percentage of pixels, by embedding a cost-sensitive matrix into the loss function. Specifically:

[0017] For image pixel classification problems, a cost-sensitive penalty matrix is ​​applied to each pixel in each patch. Therefore, the existing cost-sensitive matrix W is first associated with each pixel in the input sample image. The input patch image of size (h1, h2) is defined as follows:

[0018]

[0019] And y p This represents the actual value label corresponding to P;

[0020]

[0021] Among them, y P (i,j)∈[0,M], correlation mapping matrix M p Defined as:

[0022]

[0023] The cost-sensitive loss function is defined as:

[0024]

[0025] Let and y represent the predicted result and the true label, respectively, and λ represent the weight of the cost-sensitive loss. p (y i ,·) represents the cost-sensitive confusion matrix after associating the true labels with the cost-sensitive labels, L base and Defined as:

[0026]

[0027]

[0028] Because w(k,k) is 0 in the cost-sensitive penalty matrix W, and M p (yi The operation result for the pixel of category ,·) is 0, therefore, using L ce As a complement to the cost-sensitive loss function, finally, L cs It can be used for cost-sensitive penalties for class imbalance.

[0029] As a preferred technical solution, the two-step watershed algorithm post-processing module includes the following two steps:

[0030] The first step of the watershed algorithm is to obtain the edge features in the horizontal and vertical directions using the gradient method, and then superimpose the edge features obtained in the two directions to obtain the complete edge features. Combined with the edge structure features, a label-controlled watershed method is introduced to obtain the cell nucleus instance segmentation results.

[0031] The second step of the watershed algorithm: First, merge the units that were incorrectly segmented by the first step of the watershed algorithm. Then, calculate the distance transformation map of the units in the merged result. In the distance transformation calculation process, the pixel value is larger the farther away from the background. Combining the distance transformation map, the second step of the label control watershed algorithm is used to achieve more accurate instance segmentation for all types of units.

[0032] The cell nucleus classification branch is applied to the cell nucleus instance segmentation results, and each instance unit is voted on according to the category attribute to obtain the segmentation and classification results.

[0033] Another aspect of the present invention provides a classification method for a cell nucleus segmentation and classification system for H&E-stained tissue pathological images, comprising the following steps:

[0034] S1. Obtain H&E stained tissue pathological images and input the pathological images into the multi-task deep learning module; the pathological images are first encoded by the multi-task deep learning module for cell semantic segmentation, cell nucleus instance segmentation and cell nucleus classification.

[0035] S2, The encoding structure output features of the three subtasks of the multi-task deep learning module are sequentially passed through the multi-task related attention mechanism module;

[0036] S3. The outputs of the three subtasks of the multi-task deep learning module are processed by the multi-task related attention mechanism module and then enter their respective decoding structures to obtain the output results of cell nucleus segmentation, instance segmentation and cell nucleus classification respectively.

[0037] S4. The cell kernel segmentation result and instance segmentation result of the multi-task deep learning module are obtained by using the first watershed algorithm to obtain the first watershed segmentation result;

[0038] S5. The cell nucleus classification results of the multi-task deep learning module are further combined with the first watershed segmentation results. By statistically analyzing the cell nucleus classification results in the cells segmented by the first watershed, the cell nucleus type is identified, and a cell nucleus segmentation and recognition map is obtained.

[0039] S6. For cells that are incorrectly segmented in the cell nucleus segmentation and recognition graph, adjacent cells of the same type are merged, and the distance transformation graph of the merged result is calculated. A second label-controlled watershed algorithm is then performed to obtain the corrected cell nucleus segmentation and recognition result.

[0040] As a preferred technical solution, the first step of the watershed algorithm is specifically as follows:

[0041] The gradient method is used to obtain the edge features in the horizontal and vertical directions. The edge features obtained in the two directions are superimposed to obtain the complete edge features. Combined with the edge structure features, the label control watershed method is introduced to obtain the cell nucleus instance segmentation results.

[0042] As a preferred technical solution, the second watershed algorithm is specifically as follows:

[0043] First, merge the incorrectly segmented units from the first step of the watershed algorithm. Then, calculate the distance transformation map of the units in the merged result. During the distance transformation calculation, the pixel value is larger for pixels farther from the background. Combining the distance transformation map, the second step of the label control watershed algorithm is used to achieve more accurate instance segmentation for all types of units.

[0044] In another aspect, the present invention provides an electronic device, the electronic device comprising:

[0045] At least one processor; and,

[0046] A memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores computer program instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the classification method of the H&E stained tissue pathology image cell nuclear segmentation and classification system.

[0048] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the classification method of the H&E stained tissue pathological image cell nucleus segmentation and classification system.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] This invention employs multi-task deep learning, multi-task related attention mechanism, cost-sensitive learning strategy, and two-step watershed algorithm post-processing techniques to solve the technical problems of low segmentation accuracy and low recognition accuracy of cell nuclei in tissue pathology images, thereby achieving the technical effect of accurate segmentation and recognition of cell nuclei in tissue pathology images. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the structure of the H&E-stained tissue pathological image cell nucleus segmentation and classification system according to an embodiment of the present invention.

[0053] Figure 2 This is a flowchart of the method for segmenting and classifying cell nuclei in H&E-stained tissue pathological images according to an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0056] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0057] like Figure 1As shown, this embodiment of the H&E-stained tissue pathology image cell nucleus segmentation and classification system includes a multi-task deep learning module, a multi-task related attention mechanism module, a cost-sensitive learning strategy module, and a two-step watershed algorithm post-processing module. In this system, the multi-task deep learning module consists of three branches for performing cell nucleus semantic segmentation, instance segmentation, and cell nucleus classification, respectively. These three branches are the cell nucleus segmentation (NS) branch, the cell nucleus instantiation (NI) branch, and the cell nucleus classification (NC) branch. Each branch performs its respective function, learning the shape, edge, and category features of the nuclei in the pathological image. The NS branch distinguishes whether pixels in the input image belong to cell nuclei. This method segments the cell nuclei in the histopathological image, obtaining a coarse semantic segmentation result. However, tumor cell nuclei are often clustered, resulting in a large number of adhered and overlapping nuclei. Semantic segmentation cannot effectively segment these cell nucleus clusters. The NI branch generates horizontal and vertical distance maps for instance segmentation, which helps to perform instance segmentation on the semantic segmentation results of the previous branch. The NC branch obtains the classification map corresponding to the histopathological image, which helps to complete cell nucleus classification on the instance segmentation results of the first two branches. The prediction result of the NC branch corresponds to the cell nucleus category with the highest pixel ratio.

[0058] To effectively integrate these features, this application introduces a multi-task-related attention strategy between the encoder and decoder structures for each of the three branches. After the decoder completes, the NS, NI, and NC branches perform nucleus semantic segmentation, nucleus instantiation, and nucleus classification, respectively. Finally, a post-processing module with a two-step watershed is proposed to alleviate over-segmentation of nuclei.

[0059] The multi-task related attention module is used to effectively integrate three independent branch tasks, effectively fuse and learn the feature representations of different sub-task branches, and realize the utilization of global context information;

[0060] The cost-sensitive learning strategy module is used to effectively solve the problem of model performance degradation caused by a severe imbalance in the number of samples of different cell types.

[0061] The two-step watershed algorithm post-processing module can more accurately segment cell nuclei that differ greatly in size, color, and shape.

[0062] Furthermore, the multi-task deep learning module includes a cell nucleus segmentation module, a cell nucleus instance segmentation module, and a cell nucleus classification module; the cell nucleus segmentation module is used to distinguish whether pixels in the input image belong to the cell nucleus or not.

[0063] The cell nucleus instance segmentation module is used to generate horizontal and vertical distance maps for instance segmentation, which helps to perform instance segmentation on the semantic segmentation results of the previous module;

[0064] The cell nucleus classification module is used to obtain the classification map corresponding to the histopathological image. This helps to complete the cell nucleus classification based on the instance segmentation results of the first two modules. The prediction result of the cell nucleus classification module corresponds to the category with the highest pixel ratio in the cell nucleus instance.

[0065] It is understood that the data samples used in the entire experiment of this invention come from the open-source data of the Colorectal Pathology Image Cell Nuclear Segmentation and Phenotyping (CoNSeP) and the Multi-Organ Cell Nuclear Segmentation and Classification Challenge (MoNuSAC), and all experiments are retrospective experiments, not experiments performed on living organisms.

[0066] Furthermore, the multi-task related attention mechanism module is implemented based on the feature interaction mechanism of multi-task branches in the following way:

[0067] Firstly, the three subtask branches are first processed by their respective encoders and then connected to the attention mechanism module.

[0068] Subsequently, the attention mechanism module outputs to the decoder and executes its respective task. Each of the three sub-task branches has its own parameters, while the parameters within the attention mechanism module are shared by all task branches. Due to the multi-task related attention mechanism, each branch not only learns its own task features but also considers the features of other branches. Therefore, compared to commonly used multi-task learning methods, the multi-task related attention mechanism can minimize interference between different sub-tasks while fusing the feature representations of each sub-task. The attention mechanism can acquire contextual information through long-term dependencies. By introducing a cross-attention mechanism to construct the multi-task related attention mechanism model, each pixel can acquire the contextual information of its pixels along its cross paths. Assuming the width of the input image is w and the height is h, the parameters of each cross-attention module are w + h - 1. When the operation is executed twice consecutively, each pixel will acquire information from all pixels in the entire image. Therefore, this invention connects two closely linked cross-attention modules to construct a multi-task related attention mechanism, which can minimize computational cost while utilizing global contextual information.

[0069] Furthermore, the cost-sensitive learning strategy module effectively addresses the problem of severe imbalance in the number of samples from different cell nucleus types. This module is implemented in the following way:

[0070] Imbalanced data is a common problem in object detection and classification tasks, and the imbalance in the number of different cell types is particularly prominent in histopathological image datasets. Cell type imbalance leads to a decline in segmentation and classification performance, especially for cells with a smaller number of cells. To address this issue, this application proposes a novel cost-sensitive loss function for the segmentation and classification of multiple cell nuclei. By embedding a cost-sensitive matrix into the loss function, the impact of sample imbalance can be mitigated, improving the classification accuracy of cell nuclei with a smaller percentage of pixels.

[0071] For multi-class tasks, assuming the number of classes is M, the cost sensitivity matrix W(M×M) can be defined as follows:

[0072]

[0073] w(j,k) is the loss when class j is misclassified as class k. The diagonal elements of matrix W are 0, and the values ​​of the off-diagonal elements on the cost-sensitive matrix are set according to the proportion of the number of samples.

[0074] The above analysis is based on the image level or patch level. For image pixel classification problems, this application applies a cost-sensitive penalty matrix to each pixel in each patch. Therefore, this application first associates an existing cost-sensitive matrix W with each pixel in the input sample image. A patch image with input size (h1, h2) can be defined as:

[0075]

[0076] And y p This represents the actual value label corresponding to P.

[0077]

[0078] Among them, y P (i,j)∈[0,M]. The correlation mapping matrix M p Defined as:

[0079]

[0080] The cost-sensitive loss function can be defined as:

[0081]

[0082] Let and y represent the predicted result and the true label, respectively, and λ represent the weight of the cost-sensitive loss. p (y i ,·) represents the cost-sensitive confusion matrix after associating the true labels with the cost-sensitive labels, L base and Defined as:

[0083]

[0084]

[0085] Because w(k,k) is 0 in the cost-sensitive penalty matrix W, and M p (y i The operation result corresponding to the category pixel (,·) is 0. Therefore, this application uses L. ce As a complement to the cost-sensitive loss function. Finally, L cs It can be used for cost-sensitive penalties for class imbalance.

[0086] Furthermore, the two-step watershed algorithm post-processing module can improve the performance of cell nucleus segmentation and classification. This module is implemented in the following way:

[0087] After the decoder stage, the nucleus instance segmentation branch obtains horizontal and vertical distance maps based on the nucleus instances. This application uses a gradient method to obtain edge features in the horizontal and vertical directions, and then superimposes the edge features obtained in both directions to obtain complete edge features. Combining these edge structure features, a label-controlled watershed method is introduced to obtain the nucleus instance segmentation result. However, the label-controlled watershed method based solely on horizontal and vertical distances cannot achieve accurate instance segmentation for all cell types because different cell types vary greatly in size, color, and morphology. For example, macrophages are easily over-segmented into multiple instances. Therefore, this application employs a label-controlled watershed method based on distance transformation, which optimizes the segmentation result of the previous step. In the second step, this application merges incorrectly segmented units and then calculates the distance transformation map of the units in the merged result. During the distance transformation calculation, pixels farther from the background have larger values. Combining the distance transformation map, the label-controlled watershed algorithm in the second step achieves more accurate instance segmentation for all types of units. Finally, the nucleus classification branch is applied to the nucleus instance segmentation result, and each instance unit is voted on according to its category attribute to obtain the segmentation and classification results. In this way, the results of cell nucleus instance segmentation and classification are obtained.

[0088] The technical solution of the present invention will be further explained below with reference to specific experiments:

[0089] like Figure 2 As shown, this invention provides a method for cell nucleus segmentation and classification in H&E-stained tissue pathological images, including several steps such as input image, encoder, multi-task related attention, decoder, post-processing, and output results, specifically:

[0090] First, a 256×256 patch is input into the encoder module of this application. The encoder module is used to obtain high-level semantic information. The first to fourth layers of the encoder module correspond to the first to fourth residual blocks in ResNet-34. Each residual block contains batch normalization and modified linear unit activation functions.

[0091] Furthermore, the multi-task fusion attention mechanism is used for the encoder-decoder of multi-task branches. The three sub-task branches first pass through their respective encoders and then connect to the attention mechanism module. Due to the integration mechanism of the multi-task fusion attention, each branch not only learns its own task features but also considers the features of other branches.

[0092] Furthermore, they output to the decoder and perform their respective tasks. Each of the three subtasks has its own parameters, while the parameters in the attention module are shared by all task branches.

[0093] Furthermore, after the decoder is completed, the NS branch, NI branch, and NC branch perform cell nucleus semantic segmentation, nucleus instantiation, and cell nucleus classification, respectively.

[0094] Furthermore, the two-step watershed algorithm post-processing module is performed in two steps:

[0095] Firstly, this application employs a label-controlled watershed method based on distance transformation, such as... Figure 2 As shown, it will optimize the segmentation results of the previous step.

[0096] The second step involves merging incorrectly segmented units and then calculating the distance transformation map of the merged units. During the distance transformation calculation, pixels farther from the background have larger values. Combining the distance transformation map, the second-step label-controlled watershed algorithm is used to achieve more accurate instance segmentation for all types of units. Finally, the nucleus classification branch is applied to the nucleus instance segmentation results, and each instance unit is voted on based on its category attribute to obtain the segmentation and classification results.

[0097] Please see Figure 3 In one embodiment, an electronic device is provided for a method of segmenting and classifying cell nuclei in H&E-stained tissue pathological images. The electronic device 300 may include a first processor 301, a first memory 302, and a bus. It may also include a computer program stored in the first memory 302 and executable on the first processor 301, such as an H&E-stained tissue pathological image cell nucleus segmentation and classification program 303.

[0098] The first memory 302 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 302 can be an internal storage unit of the electronic device 300, such as the portable hard drive of the electronic device 300. In other embodiments, the first memory 302 can also be an external storage device of the electronic device 300, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 300. Furthermore, the first memory 302 can include both internal and external storage units of the electronic device 300. The first memory 302 can be used not only to store application software and various types of data installed on the electronic device 300, such as the code of the H&E stained tissue pathology image cell nucleus segmentation and classification program 303, but also to temporarily store data that has been output or will be output.

[0099] In some embodiments, the first processor 301 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 301 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (such as federated learning defense programs) stored in the first memory 302, and calls data stored in the first memory 302 to perform various functions of the electronic device 300 and process data.

[0100] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 300, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0101] The H&E-stained tissue pathology image cell nucleus segmentation and classification program 303 stored in the first memory 302 of the electronic device 300 is a combination of multiple instructions. When run in the first processor 301, it can achieve the following:

[0102] S1. Obtain H&E stained tissue pathological images and input the pathological images into the multi-task deep learning module; the pathological images are first encoded by the multi-task deep learning module for cell semantic segmentation, cell nucleus instance segmentation and cell nucleus classification.

[0103] S2, The encoding structure output features of the three subtasks of the multi-task deep learning module are sequentially passed through the multi-task related attention mechanism module;

[0104] S3. The outputs of the three subtasks of the multi-task deep learning module are processed by the multi-task related attention mechanism module and then enter their respective decoding structures to obtain the output results of cell nucleus segmentation, instance segmentation and cell nucleus classification respectively.

[0105] S4. The cell kernel segmentation result and instance segmentation result of the multi-task deep learning module are obtained by using the first watershed algorithm to obtain the first watershed segmentation result;

[0106] S5. The cell nucleus classification results of the multi-task deep learning module are further combined with the first watershed segmentation results. By statistically analyzing the cell nucleus classification results in the cells segmented by the first watershed, the cell nucleus type is identified, and a cell nucleus segmentation and recognition map is obtained.

[0107] S6. For cells that are incorrectly segmented in the cell nucleus segmentation and recognition graph, adjacent cells of the same type are merged, and the distance transformation graph of the merged result is calculated. A second label-controlled watershed algorithm is then performed to obtain the corrected cell nucleus segmentation and recognition result.

[0108] Furthermore, if the modules / units integrated in the electronic device 300 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0109] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0110] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A cell nucleus segmentation and classification system for H&E-stained tissue pathological images, characterized in that, It includes a multi-task deep learning module, a multi-task related attention mechanism module, a cost-sensitive learning strategy module, and a two-step watershed algorithm post-processing module; The multi-task deep learning module comprises three sub-task modules: a cell nucleus segmentation module, a cell nucleus instance segmentation module, and a cell nucleus classification module. The input images for these modules are processed through an encoding structure, a multi-task related attention mechanism module, and a decoding structure to obtain cell nucleus segmentation results, instance segmentation results, and cell nucleus classification results, respectively. The multi-task related attention mechanism module employs a feature interaction mechanism based on multi-task branches, specifically: The three sub-task modules first pass through their respective encoders and then connect to the multi-task related attention mechanism module; The three sub-task modules output to their respective decoders and execute their respective tasks. Each of the three sub-task modules has its own parameters, while the parameters in the multi-task related attention mechanism module are shared by all sub-task modules. Due to the role of the multi-task related attention mechanism module, each sub-task module not only learns its own task features, but also learns the relevant task features of other sub-task modules. The cell nucleus segmentation module is used to distinguish whether the pixels of the input image belong to the cell nucleus or not. The cell nucleus instance segmentation module is used to generate horizontal and vertical distance maps for instance segmentation, which helps to perform instance segmentation on the semantic segmentation results of the previous module; The cell nucleus classification module is used to obtain the classification map corresponding to the histopathological image, which helps to complete the cell nucleus classification based on the instance segmentation results of the first two modules; the prediction result of the cell nucleus classification module corresponds to the category with the highest pixel ratio in the cell nucleus instance; The multi-task related attention mechanism module is used to fuse the feature representations of the cell nucleus segmentation module, the cell nucleus instance segmentation module, and the cell nucleus classification module to realize the utilization of global context information. The cost-sensitive learning strategy module effectively addresses the problem of model performance degradation caused by a severe imbalance in the number of samples from different cell types. Specifically, by embedding a cost-sensitive matrix into the loss function, the module mitigates the impact of sample imbalance and improves the classification accuracy of cell nuclei, which have a smaller proportion of pixels. For image pixel classification problems, a cost-sensitive penalty matrix is ​​applied to each pixel in each patch. Therefore, the existing cost-sensitive matrix is ​​used first. Associated with each pixel in the input sample image, the input size is The patch image is defined as: And y p This represents the actual value label corresponding to P; in, Association mapping matrix M p Defined as: The cost-sensitive loss function is defined as: and These represent the predicted result and the true label, respectively. The weights representing cost-sensitive losses, This represents the cost-sensitive confusion matrix after associating real labels with cost-sensitive labels. and Defined as: because In a cost-sensitive penalty matrix The value in the middle is 0. and The operation result corresponding to the category pixel is 0, therefore, using As a supplement to the cost-sensitive loss function, finally, It can be used for cost-sensitive penalties related to class imbalance; The two-step watershed algorithm post-processing module is used to more accurately segment cell nuclei that differ greatly in size, color, and morphology; the two-step watershed algorithm post-processing module includes the following two steps: The first step of the watershed algorithm is to obtain the edge features in the horizontal and vertical directions using the gradient method, and then superimpose the edge features obtained in the two directions to obtain the complete edge features. Combined with the edge structure features, a label-controlled watershed method is introduced to obtain the cell nucleus instance segmentation results. The second step, the watershed algorithm, first merges the incorrectly segmented units from the first step. Then, it calculates the distance transformation map of the units in the merged result. During the distance transformation calculation, the pixel value is larger for pixels farther from the background. Combining the distance transformation map, the second step, the marker-controlled watershed algorithm, is used to achieve more accurate instance segmentation for all types of units.

2. The H&E-stained tissue pathological image cell nuclear segmentation and classification system according to claim 1, characterized in that, The cell nucleus classification branch is applied to the cell nucleus instance segmentation results, and each instance unit is voted on according to the category attribute to obtain the segmentation and classification results.

3. The classification method of the H&E-stained tissue pathological image cell nuclear segmentation and classification system according to any one of claims 1-2, characterized in that, Includes the following steps: S1. Obtain H&E stained tissue pathological images and input the pathological images into the multi-task deep learning module; the pathological images are first encoded by the multi-task deep learning module for cell semantic segmentation, cell nucleus instance segmentation and cell nucleus classification. S2, The encoding structure output features of the three subtasks of the multi-task deep learning module are sequentially passed through the multi-task related attention mechanism module; S3. The outputs of the three subtasks of the multi-task deep learning module are processed by the multi-task related attention mechanism module and then enter their respective decoding structures to obtain the output results of cell nucleus segmentation, instance segmentation and cell nucleus classification respectively. S4. The cell kernel segmentation result and instance segmentation result of the multi-task deep learning module are obtained by using the first watershed algorithm to obtain the first watershed segmentation result; S5. The cell nucleus classification results of the multi-task deep learning module are further combined with the first watershed segmentation results. By statistically analyzing the cell nucleus classification results in the cells segmented by the first watershed, the cell nucleus type is identified, and a cell nucleus segmentation and recognition map is obtained. S6. For cells that are incorrectly segmented in the cell nucleus segmentation and recognition graph, adjacent cells of the same type are merged, and the distance transformation graph of the merged result is calculated. A second label-controlled watershed algorithm is then performed to obtain the corrected cell nucleus segmentation and recognition result.

4. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, the computer program instructions being executed by the at least one processor to enable the at least one processor to perform the classification method of the H&E stained histopathological image cell nuclear segmentation and classification system as described in any one of claims 1-2.

5. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the classification method of the H&E stained tissue pathological image cell nucleus segmentation and classification system according to any one of claims 1-2.

Images