Method and device for classifying tissue regions of immunohistochemistry images based on multi-scale stacking

Abstract

The application provides an immunohistochemical image tissue region classification method and device based on multi-scale stacking, which comprises the following steps: constructing an image sequence based on an immunohistochemical image, using a pre-trained classification model to classify and mark the tissue region of each image in the image sequence to obtain a classification sequence; transforming each prediction result image in the classification sequence into the same scale size and performing image stacking to obtain an immunohistochemical mask array; correcting the tissue region of the immunohistochemical mask array to obtain a corrected immunohistochemical mask array; and mapping the corrected immunohistochemical mask array to the immunohistochemical image to obtain an immunohistochemical image classification result. In the scheme, the immunohistochemical mask array is obtained through multi-scale stacking, and the pixel position in the to-be-corrected tissue region in the immunohistochemical mask array is reclassified through calculation of a dynamic correction weight, so that the fuzzy region of the initial classification is corrected in a targeted manner, and the classification accuracy of the to-be-corrected region is significantly improved.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to a method and apparatus for classifying tissue regions in immunohistochemical images based on multi-scale stacking. Background Technology

[0002] Immunohistochemistry (IHC) pathological image analysis, as a key technique in biomedical research and clinical pathological diagnosis, plays an irreplaceable role in cancer subtyping, efficacy evaluation, and prognosis. Especially in quantitative pathological scenarios such as HER2, PR, ER, and Ki-67 expression analysis in breast cancer, it provides direct molecular and morphological evidence for clinical decision-making and can be extended to immunohistochemical pathological image analysis tasks for multiple types of cancer, including lung cancer and gastric cancer. In IHC pathological image processing, the most commonly used tissue staining techniques include HE (hematoxylin and eosin) staining and DAB (diaminophenylbutanedione) staining. HE-stained images, with their relatively stable color characteristics and clear tissue boundaries, can achieve tissue localization through traditional image processing techniques such as color space conversion and morphological manipulation. However, DAB-stained images are often affected by factors such as uneven staining intensity and tissue folding artifacts, resulting in blurred tissue boundaries. This makes it difficult to effectively distinguish highly similar control tissues (i.e., reference tissues used to verify the effectiveness of staining operations) from foreground tissues (target tissues to be analyzed), posing a significant challenge to accurate tissue localization.

[0003] Current techniques for foreground tissue localization in DAB-stained IHC images have significant shortcomings, failing to meet the high accuracy and practicality requirements of clinical diagnosis. Firstly, traditional methods rely excessively on manual feature engineering, making them unable to adapt to complex scenarios such as uneven staining and artifact interference in DAB-stained images, hindering efficient extraction of foreground tissue regions and resulting in low localization efficiency and accuracy. Secondly, anchor-frame-based detection models, such as the YOLO series, are prone to missed or false detections in dense tissue regions and are limited by the fixed shape of the anchor frames, failing to accurately segment morphologically variable foreground tissues. Thirdly, other deep learning segmentation models (such as panoramic and instance segmentation models) are prone to edge fragmentation when facing differences in image scale and tissue heterogeneity at different scanning magnifications, failing to fully preserve the morphological features of foreground tissues. Fourthly, existing methods generally lack comprehensive post-processing mechanisms for multi-scale prediction consistency, tissue classification reliability, regional morphological features, and spatial distribution. Even if the basic model has a certain segmentation capability, the final output segmentation results still suffer from misclassification and small-area noise interference, making it difficult to meet the stringent standards of clinical diagnosis.

[0004] Therefore, given the unique challenges of DAB-stained IHC images and the limitations of existing technologies, there is an urgent need to develop an immunohistochemical image foreground tissue localization method that integrates deep learning model training, lightweight deployment, and multi-dimensional post-processing. This method aims to overcome current technical bottlenecks, achieve high-precision localization and segmentation of foreground tissues, and simultaneously meet the needs of real-time performance and lightweight deployment in clinical settings, thus providing technical support for precise pathological quantitative analysis via immunohistochemistry. Summary of the Invention

[0005] This application provides a method and apparatus for classifying tissue regions in immunohistochemistry images based on multi-scale stacking. By transforming all predicted result images in the classification sequence to the same scale and stacking them, an immunohistochemistry mask array is obtained. Then, by calculating dynamic correction weights, the pixel positions in the tissue regions to be corrected within the immunohistochemistry mask array are reclassified, thereby specifically correcting the blurred regions of the initial classification and significantly improving the classification accuracy of the regions to be corrected.

[0006] In a first aspect, embodiments of this application provide a method for classifying tissue regions in immunohistochemical images based on multi-scale stacking, the method comprising:

[0007] An image sequence is constructed based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue.

[0008] Each predicted image in the classification sequence is transformed to the same scale and the images are stacked to obtain an immunohistochemistry mask array. The classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted images are calculated. Based on the classification consistency and pixel confidence, the dynamic correction weight of each pixel position in different predicted images is calculated.

[0009] The immunohistochemistry mask array is divided into the tissue region to be corrected and the background region. The foreground tissue score and control tissue score of each pixel position in the tissue region to be corrected are calculated using the dynamic correction weight of the corresponding pixel position. Based on the foreground tissue score and control tissue score of each pixel position, each pixel position in the tissue region to be corrected is reclassified to obtain the corrected tissue region.

[0010] The corrected tissue region and background region are integrated as a corrected immunohistochemical mask array. The corrected immunohistochemical mask array is then mapped onto the immunohistochemical image to obtain the immunohistochemical image classification result.

[0011] Secondly, embodiments of this application provide a multi-scale stacked immunohistochemical image tissue region classification device, comprising:

[0012] The module constructs an image sequence based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue.

[0013] The stacking module is used to transform each predicted result image in the classification sequence to the same scale and stack the images to obtain an immunohistochemistry mask array. It calculates the classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted result images, and calculates the dynamic correction weight of each pixel position in different predicted result images based on the classification consistency and pixel confidence.

[0014] The correction module is used to divide the immunohistochemistry mask array into the tissue region to be corrected and the background region. It calculates the foreground tissue score and control tissue score of each pixel position in the tissue region to be corrected using the dynamic correction weight of the corresponding pixel position. Based on the foreground tissue score and control tissue score of each pixel position, it reclassifies each pixel position in the tissue region to be corrected to obtain the corrected tissue region.

[0015] The output module integrates the corrected tissue region and the background region as a corrected immunohistochemical mask array, and maps the corrected immunohistochemical mask array onto the immunohistochemical image to obtain the immunohistochemical image classification result.

[0016] The main contributions and innovations of this invention are as follows:

[0017] This application's embodiments construct image sequences containing representations at different scales based on immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image within the sequence, thereby obtaining a classification sequence that covers detailed information at different scales and avoids missed classifications of tissue regions at a single scale. Misclassification; This application embodiment transforms all predicted result images in the classification sequence to the same scale and stacks them to obtain an immunohistochemical mask array. It then calculates dynamically corrected weights to reclassify pixel positions within the tissue region to be corrected, thereby specifically correcting the blurred areas of the initial classification and significantly improving the classification accuracy of the region to be corrected. This application embodiment uses connected component analysis for secondary correction before mapping, presets a minimum effective area, and corrects the connected components of small regions containing only control tissue to the background, thus avoiding interference from invalid small regions. Correcting small regions containing only control tissue to the background reduces redundant annotations in the control region, making the final mask array cleaner and improving the purity and readability of the classification results. This solution, through dynamic quantization processing of the prediction model trained by Mask2Former, supports model server deployment for various application scenarios at FP16 and INT8 levels, facilitating rapid prediction processing. After INT8 quantization, the model weight file space is compressed by 89% with an accuracy loss of <3%.

[0018] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of a method for classifying tissue regions in immunohistochemical images based on multi-scale stacking, according to an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of an immunohistochemical image according to an embodiment of this application;

[0022] Figure 3 These are schematic diagrams of prediction results at different scales according to embodiments of this application;

[0023] Figure 4 This is a schematic diagram of an immunohistochemical image classification result according to an embodiment of this application;

[0024] Figure 5 This is a structural block diagram of a data entry device according to an embodiment of this application;

[0025] Figure 6This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0028] Example 1

[0029] This application provides a method and apparatus for classifying tissue regions in immunohistochemical images based on multi-scale stacking. It obtains an immunohistochemical mask array by transforming all predicted result images in the classification sequence to the same scale and stacking them. Then, it reclassifies the pixel positions within the tissue region to be corrected in the immunohistochemical mask array by calculating dynamic correction weights, thereby specifically correcting the blurred regions of the initial classification and significantly improving the classification accuracy of the region to be corrected. Specifically, refer to... Figure 1 The method includes:

[0030] An image sequence is constructed based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue.

[0031] Each predicted image in the classification sequence is transformed to the same scale and the images are stacked to obtain an immunohistochemistry mask array. The classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted images are calculated. Then, the pixel confidence of each pixel position is calculated based on the classification consistency of each pixel position. The dynamic correction weight of each pixel position is calculated based on the classification consistency and pixel confidence.

[0032] The immunohistochemistry mask array is divided into the tissue region to be corrected and the background region. The foreground tissue score and control tissue score of each pixel in the tissue region to be corrected are calculated using the dynamic correction weight of the corresponding pixel position. The corrected tissue region is obtained by reclassifying each pixel position based on the foreground tissue score and control tissue score of each pixel position.

[0033] The corrected tissue region and background region are integrated as a corrected immunohistochemical mask array. The corrected immunohistochemical mask array is then mapped onto the immunohistochemical image to obtain the immunohistochemical image classification result.

[0034] In the current embodiment, the immunohistochemical image is continuously downsampled to obtain different scale representations of the immunohistochemical image. Specifically, the immunohistochemical image is continuously downsampled three times to obtain four images, including the immunohistochemical image.

[0035] In this embodiment, the classification model is the Mask2Former model, and the pre-training process of the classification model is as follows:

[0036] Multiple immunohistochemical images, including four commonly used staining types—ER, PR, HER-2, and Ki-67—were acquired. The open-source pathological image analysis tool QuPath was used to generate labeled mask images by adding WSI-level annotations to foreground and control tissues. A Groovy script was then used to export the labeled immunohistochemical images and combine them with the corresponding labeled mask images as a data sample set. A schematic diagram of the immunohistochemical images is shown below. Figure 2 As shown.

[0037] Specifically, the image scale in the data sample set was uniformly adjusted to 2048×2048 pixels, and the background pixel value of the mask image was set to 0, the foreground tissue pixel value was set to 1, and the control tissue pixel value was set to 2. Finally, the data sample set was randomly divided into training set and validation set in a ratio of 8:2.

[0038] The Mask2Former model uses the Swin Transformer Large as the backbone network for segmentation and employs the Swin-L (IN21k) model benchmark trained on the Cityscapes image semantic segmentation task dataset as pre-training weights. The pre-defined training and validation sets are registered as the immunohistochemistry image datasets IHC_train and IHC_val, consistent with the semantic segmentation task dataset configuration. Training uses the default hyperparameter configuration, setting NUM_CLASSES to 3 classes (including the background class), NUM_GPUS to 1, SOLVER.IMS_PER_BATCH to 4, and DATALOADER.NUM_WORKERS to 1. The maximum number of training iterations MAX_ITER is set to 90,000 EPOCHS by default to ensure the model can fully learn the data features.

[0039] Specifically, this solution ensures the accuracy and completeness of labeled data by performing high-precision annotation on the WSI of the entire field of view slices. Simultaneously, through collaboration with a custom Groovy script, it supports the synchronous output of labels for multiple tissue categories, such as control tissues and foreground tissues, enabling batch processing of annotation results and dataset export. This guarantees a high degree of consistency between training data and real annotations, improving annotation efficiency and data quality while reducing annotation costs. Furthermore, this solution dynamically quantizes the prediction model trained by Mask2Former, supporting model deployment at FP16 and INT8 levels for various application scenarios, facilitating rapid prediction processing. After INT8 quantization, the model weight file size is compressed by 89% with an accuracy loss of <3%.

[0040] To ensure lightweight deployment of the model across different scenarios, during the quantization process of the classification model, the `torch.onnx.export` module was first used to convert the `.pth` format weight file of the Mask2Former semantic segmentation model trained on the tissue segmentation model to `.onnx` format. By constructing a quantization model wrapper, the input of the original model was converted to standard tensor input (NCHW format), supporting dynamic size input prediction, resulting in an FP32 quantized model. Compared to the original model, the MIoU, MACC, and MDice metrics decreased by less than 3%. Secondly, further quantization was performed using different methods. The `onnxconverter_common` module was used to convert the quantized FP32 model to FP16 accuracy, reducing the model size by 50%, with the MIoU, MACC, and MDice metrics decreasing by only less than 0.05% (experimental data are shown in Table 1). This supports FP16-level server deployment for handling complex cases requiring expert-level accuracy. Using the onnxruntime module, the FP32 model is converted to INT8 precision. Only fully connected layers are quantized, skipping the quantization of edge-sensitive convolutional layers. The INT8 model achieves an 89.4% reduction in size, while the MACC value decreases by only 0.02%, and the MIoU and MDice values ​​surpass those of FP32. This supports INT8-level server deployment, meeting the rapid analysis needs of most cases while reducing hardware procurement costs. Multi-level quantization provides lightweight deployment solutions for servers in different scenarios.

[0041] Table 1. Comparison of experimental results for different segmentation methods

[0042]

[0043] As shown in Table 1, the MIoU, MACC, and MDice metrics of this invention are all superior to the three existing mainstream segmentation models: YOLOv8 (based on YOLOv8n-seg), YOLO11 (based on YOLO11n-seg), and YOLO12 (based on YOLO12n-seg). After dynamic quantization of the Mask2Fromer model, the accuracy loss is within an acceptable range of 2%-3%. INT8 quantization is even better than the FP16 quantization scheme, with less accuracy loss. The addition of post-processing further improves the important segmentation metrics after quantization, increasing MIoU by 0.37% and MDice by 0.68%.

[0044] In the current embodiment, the immunohistochemical image is continuously downsampled to obtain different scale representations of the immunohistochemical image. In the step of transforming each predicted result image in the classification sequence to the same scale size, the scale of each predicted result image is transformed to the scale size of the immunohistochemical image by bilinear interpolation.

[0045] For example, if the immunohistochemistry image is a high-resolution image with a scale of 4096×4096, then by downsampling three times consecutively, three images with scales of 2048×2048, 1024×1024, and 512×512 will be obtained respectively. These three images, together with the immunohistochemistry image, form an image sequence. Each image in the image sequence is classified and labeled to obtain multiple predicted result images, which together form a classification sequence. A schematic diagram of the predicted result images at different scales is shown below. Figure 3 As shown.

[0046] For example, since the size of the immunohistochemical image is 4096×4096, the three predicted images with sizes of 2048×2048, 1024×1024 and 512×512 are processed by bilinear interpolation so that the size of each predicted image is 4096×4096.

[0047] In other words, this method stacks four prediction result images with a scale of 4096×4096 to obtain an immunohistochemistry mask array.

[0048] In the current embodiment, classification confidence represents the reliability of the classification result of a pixel location, and classification consistency represents the consistency of the pixel location across multiple prediction result images. A classification confidence closer to 1 indicates a higher reliability of the classification result for the corresponding pixel location, and a classification consistency closer to 1 indicates a higher consistency of the corresponding pixel location across multiple prediction result images. The foreground tissue is the target tissue to be analyzed in the experiment, and the control tissue is a reference tissue used to verify the effectiveness of the staining operation. Non-tissue regions in the prediction result images are marked as background regions. The formula for calculating the classification consistency of each pixel location in the immunohistochemical mask array across all prediction result images is:

[0049]

[0050] in, pixel position The classification consistency, where N is the number of predicted images. pixel position The number of times a region is classified as background in all predicted images. pixel position The number of times it is classified as foreground tissue in all predicted images. pixel position The number of times the tissue was classified as a control tissue in all predicted result images.

[0051] In other words, this scheme uses the probability that each pixel location is classified as a background region in all predicted result images as the background classification probability, the probability that each pixel location is classified as a foreground tissue in all predicted result images as the foreground classification probability, the probability that each pixel location is classified as a control tissue in all predicted result images as the control classification probability, and the maximum value among the background classification probability, foreground classification probability, and control classification probability as the classification consistency of the corresponding pixel location.

[0052] In the current embodiment, during the calculation of classification confidence for each pixel location, if the pixel location is classified as foreground tissue only in the tissue region classification results of all predicted result images, then the classification confidence for the corresponding pixel location is expressed as follows: ,in, pixel position Classification confidence, Let N be the foreground classification probability, and N be the number of predicted result images. If a pixel location is classified as control tissue only in the tissue region classification results of all predicted result images, then the classification confidence of the corresponding pixel location is expressed as: ,in, Pixel position Classification confidence, Let be the classification probability; if a pixel location is classified as both foreground and control tissue in the tissue region classification results of all predicted images, and the foreground classification probability is greater than or equal to the control classification probability, then the classification confidence of the corresponding pixel location is expressed as: If a pixel location is classified as both foreground and control tissue in the tissue region classification results of all predicted images, and the probability of classification as control tissue is greater than that as foreground tissue, then the classification confidence of the corresponding pixel location is expressed as follows: If a pixel location is classified as either foreground tissue or control tissue in all the tissue region classification results of the predicted images, then the classification confidence of the corresponding pixel location is 1.

[0053] In this embodiment, the predicted result image is 4, so N=4.

[0054] Specifically, since the primary purpose of observing immunohistochemical images is to observe the target tissue to analyze the patient's condition, in order to avoid over-reliance on control tissues in subsequent reclassification and thus ignoring the possibility of diseased cells, the confidence level of a pixel is reduced by 20% when the pixel location is classified as control tissue only in the tissue region classification results of all predicted images, or when the pixel location is classified as both foreground and control tissues in the tissue region classification results of all predicted images, and the probability of control classification is greater than the probability of foreground classification.

[0055] Specifically, to avoid misclassification of foreground tissues and serious consequences such as misdiagnosis, when a pixel location is classified as both foreground tissue and control tissue in the tissue region classification results of all predicted images, and the foreground classification probability is greater than or equal to the control classification probability, only 30% confidence level is retained. At this time, the classification reliability of the corresponding pixel location is low, and the pixel location will be reclassified in subsequent processes.

[0056] Specifically, if a pixel location is classified as either foreground tissue or control tissue in the tissue region classification results of all predicted result images, it indicates that the classification result of the corresponding pixel location is very accurate, and the classification confidence is directly set to 1.

[0057] Specifically, since the misclassification rate of the background region is almost 0, the model only has a chance of making mistakes on the foreground tissue or the control tissue. Therefore, the set of pixel locations that are classified as tissue regions on any predicted result image is taken as the tissue region to be corrected, and the region other than the tissue region to be corrected is the background region.

[0058] In the current embodiment, an initial weight is preset for each predicted result image, and the formula for calculating the dynamic correction weight of each pixel position in different predicted result images is as follows:

[0059]

[0060] in, pixel position Dynamically adjusted weights on the predicted image i, where i is the index of the predicted image. The initial weights for the predicted image i, pixel position Classification consistency, pixel position The classification confidence level.

[0061] Furthermore, if the pixel position If the predicted image i is classified as foreground tissue, the corresponding dynamic correction weight is updated by multiplying the enhancement coefficient by the corresponding dynamic correction weight, where the enhancement coefficient is greater than 1.

[0062] Specifically, when pixel position In the predicted image i, it is classified as foreground tissue, i.e. Then ×1.2 is used as the new dynamic correction weight, and the enhancement coefficient in this scheme is 1.2.

[0063] Specifically, this scheme uses the classification accuracy of the classification model on the validation set for immunohistochemical images of different scales as the initial weight of the corresponding prediction result image.

[0064] For example, the initial weight of the predicted image at a scale of 4096×4096 is 0.80, the initial weight at a scale of 2048×2048 is 0.85, the initial weight at a scale of 1024×1024 is 0.75, and the initial weight at a scale of 512×512 is 0.82, as expressed by the formula:

[0065]

[0066] In the current embodiment, the formulas for calculating the foreground tissue score and the control tissue score at each pixel location are as follows:

[0067]

[0068]

[0069] in, pixel position The foreground organization score, where i is the image index of the predicted result. pixel position Dynamically adjusted weights on the predicted image i. For indicator functions, in In the middle, if but Select 1 if the value is 1, otherwise select 0. Indicates pixel position It is classified as a foreground region in the predicted image i; pixel position The control tissue score, in In the middle, if but Select 1 if the value is 1, otherwise select 0. Indicates pixel position It is classified as the control region in the predicted image i.

[0070] Furthermore, in the step of reclassifying each pixel location within the area to be corrected based on the foreground organization score and the control organization score of each pixel location, the sum of the foreground organization score and the control organization score of each pixel location is calculated as the total pixel score. If the proportion of the foreground organization score of a pixel location in the total pixel score is greater than a first threshold, the corresponding pixel location is classified as foreground organization; if the proportion of the control organization score of a pixel location in the total pixel score is greater than the first threshold, the corresponding pixel location is classified as control organization; if the difference between the foreground organization score and the control organization score of a pixel location is less than or equal to a second threshold, the corresponding pixel location is classified as control organization; if the foreground organization score is greater than the control organization score and the difference between the foreground organization score and the control organization score is greater than the second threshold, the corresponding pixel location is classified as foreground organization; if the control organization score is greater than the foreground organization score and the difference between the foreground organization score and the control organization score is greater than the second threshold, the corresponding pixel location is classified as control organization.

[0071] Specifically, in this scheme, the first threshold is 35%, and the second threshold is 10%. That is, for any pixel location within the area to be corrected, if... If the value is greater than 0.35, the pixel location is classified as foreground organization. If the value is greater than 0.35, the pixel location is classified as a control group. If the difference between the foreground tissue score and the control tissue score is less than 10%, then the pixel location is classified as control tissue. > and If the pixel location is classified as foreground organization, then... > and If so, the pixel location is classified as a control tissue.

[0072] In the current embodiment, before mapping the modified immunohistochemistry mask array to the immunohistochemistry image, the modified immunohistochemistry mask array is modified a second time through connected component analysis. During the connected component analysis, a minimum effective area is preset based on the area of ​​the immunohistochemistry image. Connected components are extracted from the modified immunohistochemistry mask array, and connected components with an area smaller than the minimum effective area are obtained as small region connected components. Each pixel position in the small region connected component that only includes the classification results of the control tissue is modified into the background region.

[0073] Furthermore, the formula for the minimum effective area is expressed as:

[0074]

[0075] in, The minimum effective area is defined by H, where H is the width of the immunohistochemical image and W is the height of the immunohistochemical image. Both H and W are expressed in pixels.

[0076] Specifically, all connected components in the modified immunohistochemical mask array are extracted through connected component analysis, and those with areas smaller than 1 are selected. The connected components are used as small region connected components. If the small region connected component contains foreground tissue, the classification result of each pixel position in the small region connected component is retained to avoid accidentally deleting small target regions. If the small region connected component only contains control tissue, the classification result of each pixel position in the small region connected component is marked as background region.

[0077] Furthermore, during the connected component analysis, if the corrected immunohistochemistry mask array only includes control tissue, the largest connected component that only includes control tissue is obtained as the missed connected component. The centroid pixel position of the missed connected component is obtained. If the category of the centroid pixel position in any predicted result image is foreground tissue, then each pixel position in the missed connected component is corrected to foreground tissue. Through the above techniques, the missed foreground tissue caused by low-resolution regions can be corrected, making the final corrected immunohistochemistry mask array more accurate.

[0078] Furthermore, in the connected component analysis process, connected components that simultaneously include foreground and control tissues are considered as multi-category connected components. The total area of ​​the multi-category connected components includes the area of ​​foreground tissue and the area of ​​control tissue. The compactness of the corresponding multi-category connected components is calculated based on the perimeter and total area of ​​the multi-category connected components. Then, the aspect ratio of the corresponding multi-category connected components is calculated based on the width and height of the bounding rectangle of the multi-category connected components. If the multi-category connected components satisfy the first and second conditions, then each pixel position in the multi-category connected components is secondarily corrected to foreground tissue; otherwise, each pixel position in the multi-category connected components is secondarily corrected to control tissue. The first condition is: the proportion of the foreground tissue area in the multi-category connected components is greater than 0.75; the second condition is: the compactness of the multi-category connected components is greater than 0.7 and the aspect ratio is less than 1.5, and the proportion of the foreground tissue area in the multi-category connected components is greater than 0.65.

[0079] Specifically, the formulas for calculating the total area, compactness, and aspect ratio of multi-class connected domains are as follows:

[0080]

[0081]

[0082]

[0083] Where A is the total area of ​​multi-class connected components. Foreground organization area, For control tissue area, For compactness, P is the perimeter of the multi-class connected domain. Aspect ratio, The width of the bounding rectangle of the multi-class connected domain. The height of the outer rectangle of a multi-class connected domain.

[0084] Specifically, when the compactness is greater than 0.7, it indicates that the compactness of the multi-class connected components is high and close to a circle. When the aspect ratio is less than 1.5, it indicates that the aspect ratio of the multi-class connected components is close to 1.

[0085] Furthermore, during the connected component analysis, connected components with an area greater than a set threshold are considered valid connected components. The average centroid distance of the valid connected components is calculated. If the average centroid distance is greater than or equal to a preset position correction threshold, the valid connected components located in the left half of the immunohistochemical image are corrected to control tissue, and the valid connected components located in the right half of the immunohistochemical image are corrected to foreground tissue. If the average centroid distance is less than the preset position correction threshold, no secondary correction is performed on the valid connected components.

[0086] Specifically, the formula for calculating the average centroid distance is:

[0087]

[0088] in, Let be the average centroid distance, n be the number of effective connected components, and i and j be the indices of the effective connected components. This represents the x-coordinate of the centroid of the i-th valid connected component. This represents the ordinate of the centroid of the i-th valid connected component. This represents the x-coordinate of the centroid of the j-th valid connected component. This represents the ordinate of the centroid of the j-th valid connected component.

[0089] Specifically, the formula for calculating the position correction threshold is:

[0090]

[0091] Where T is the position correction threshold. This represents the maximum height and width of the immunohistochemical image.

[0092] Specifically, the placement of immunohistochemical images follows strict requirements: control tissue should be placed on the left side of the image, and foreground tissue on the right side. When it is assumed that the effective connected components conform to a left-right distribution, a secondary correction can be made based on the position of the effective connected components. If the effective connected components are densely distributed, then no secondary correction is performed on the effective connected components to avoid misjudgment.

[0093] In the current embodiment, a schematic diagram of the immunohistochemical image classification results obtained by mapping the modified immunohistochemical mask array onto the immunohistochemical image is shown below. Figure 4 As shown, the modified immunohistochemistry mask array is in the form of a mask. The mask information in the modified immunohistochemistry mask array is transferred to the immunohistochemistry image through pixel position mapping or direct overlay, thereby completing the accurate classification of the immunohistochemistry image.

[0094] Example 2

[0095] Based on the same concept, referencing Figure 5 This application also proposes a multi-scale stacked immunohistochemical image tissue region classification device, comprising:

[0096] The module constructs an image sequence based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue.

[0097] The stacking module is used to transform each predicted result image in the classification sequence to the same scale and stack the images to obtain an immunohistochemistry mask array. It calculates the classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted result images, and calculates the dynamic correction weight of each pixel position in different predicted result images based on the classification consistency and pixel confidence.

[0098] The correction module is used to divide the immunohistochemistry mask array into the tissue region to be corrected and the background region. It calculates the foreground tissue score and control tissue score of each pixel position in the tissue region to be corrected using the dynamic correction weight of the corresponding pixel position. Based on the foreground tissue score and control tissue score of each pixel position, it reclassifies each pixel position in the tissue region to be corrected to obtain the corrected tissue region.

[0099] The output module integrates the corrected tissue region and the background region as a corrected immunohistochemical mask array, and maps the corrected immunohistochemical mask array onto the immunohistochemical image to obtain the immunohistochemical image classification result.

[0100] Example 3

[0101] This embodiment also provides an electronic device, see reference. Figure 6 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0102] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0103] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0104] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0105] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the multi-scale stacked immunohistochemical image tissue region classifications in the above embodiments.

[0106] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0107] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0108] The input / output device 408 is used to input or output information. In this embodiment, the input information may be different scale representations of the immunohistochemical image, and the output information may be a predicted image, a corrected immunohistochemical mask array, etc.

[0109] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program:

[0110] An image sequence is constructed based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue.

[0111] Each predicted image in the classification sequence is transformed to the same scale and the images are stacked to obtain an immunohistochemistry mask array. The classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted images are calculated. Based on the classification consistency and pixel confidence, the dynamic correction weight of each pixel position in different predicted images is calculated.

[0112] The immunohistochemistry mask array is divided into the tissue region to be corrected and the background region. The foreground tissue score and control tissue score of each pixel position in the tissue region to be corrected are calculated using the dynamic correction weight of the corresponding pixel position. Based on the foreground tissue score and control tissue score of each pixel position, each pixel position in the tissue region to be corrected is reclassified to obtain the corrected tissue region.

[0113] The corrected tissue region and background region are integrated as a corrected immunohistochemical mask array. The corrected immunohistochemical mask array is then mapped onto the immunohistochemical image to obtain the immunohistochemical image classification result.

[0114] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0115] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0116] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 6 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0117] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for classifying tissue regions in immunohistochemical images based on multi-scale stacking, characterized in that, Includes the following steps: An image sequence is constructed based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue. Each predicted image in the classification sequence is transformed to the same scale and the images are stacked to obtain an immunohistochemistry mask array. The classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted images are calculated. Based on the classification consistency and pixel confidence, the dynamic correction weight of each pixel position in different predicted images is calculated. The immunohistochemical mask array is segmented into a region of tissue to be corrected and a background region. The foreground tissue score and control tissue score for each pixel in the region of tissue to be corrected are calculated using dynamic correction weights at corresponding pixel positions. Based on these scores, each pixel in the region of tissue to be corrected is reclassified to obtain the corrected tissue region. Specifically, the sum of the foreground and control tissue scores for each pixel is calculated as the total pixel score. If the proportion of the foreground tissue score in the total pixel score is greater than a first threshold, the corresponding pixel is classified as foreground tissue; if the proportion of the control tissue score is greater than the first threshold, the corresponding pixel is classified as control tissue; if the difference between the foreground and control tissue scores is less than or equal to a second threshold, the corresponding pixel is classified as control tissue; if the foreground tissue score is greater than the control tissue score and the difference between the foreground and control tissue scores is greater than the second threshold, the corresponding pixel is classified as foreground tissue; if the control tissue score is greater than the foreground tissue score and the difference between the foreground and control tissue scores is greater than the second threshold, the corresponding pixel is classified as control tissue. The corrected tissue region and background region are integrated as a corrected immunohistochemical mask array. The corrected immunohistochemical mask array is then mapped onto the immunohistochemical image to obtain the immunohistochemical image classification result.

2. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 1, characterized in that, The immunohistochemical image is continuously downsampled to obtain different scale representations of the immunohistochemical image. In the step of transforming each predicted result image in the classification sequence to the same scale size, the scale of each predicted result image is transformed to the scale size of the immunohistochemical image by bilinear interpolation.

3. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 1, characterized in that, The foreground tissue is the target tissue to be analyzed in the experiment, and the control tissue is the reference tissue used to verify the effectiveness of the staining operation. Non-tissue areas in the predicted image are marked as background areas. The probability that each pixel position is classified as a background area in all predicted images is used as the background classification probability. The probability that each pixel position is classified as a foreground tissue in all predicted images is used as the foreground classification probability. The probability that each pixel position is classified as a control tissue in all predicted images is used as the control classification probability. The maximum value among the background classification probability, foreground classification probability, and control classification probability is used as the classification consistency of the corresponding pixel position.

4. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 3, characterized in that, In calculating the classification confidence score for each pixel location, if a pixel location is classified as foreground tissue only in the tissue region classification results of all predicted images, then the classification confidence score for that pixel location is expressed as follows: ; If a pixel location is classified as control tissue only in the tissue region classification results of all predicted images, then the classification confidence of the corresponding pixel location is expressed as follows: ; If a pixel location is classified as both foreground and control tissue in the tissue region classification results of all predicted images, and the foreground classification probability is greater than or equal to the control classification probability, then the classification confidence of the corresponding pixel location is expressed as follows: ; If a pixel location is classified as both foreground and control tissue in the tissue region classification results of all predicted images, and the probability of classification as control tissue is greater than that as foreground tissue, then the classification confidence of the corresponding pixel location is expressed as follows: ; If a pixel location is classified as either foreground tissue or control tissue in all predicted tissue region classification results, then the classification confidence of the corresponding pixel location is 1. pixel position Classification confidence, Foreground classification probability, Let N be the number of predicted images, representing the classification probability. pixel position The number of times it is classified as foreground tissue in all predicted images. pixel position The number of times the tissue was classified as a control tissue in all predicted outcome images.

5. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 1, characterized in that, An initial weight is preset for each predicted image. The formula for calculating the dynamic adjustment weight of each pixel position in different predicted images is as follows: ； in, pixel position Dynamically adjusted weights on the predicted image i, where i is the index of the predicted image. The initial weights for the predicted image i, pixel position Classification consistency, pixel position The classification confidence level.

6. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 5, characterized in that, If pixel position If the predicted image i is classified as foreground tissue, the corresponding dynamic correction weight is updated by multiplying the enhancement coefficient by the corresponding dynamic correction weight, where the enhancement coefficient is greater than 1.

7. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 1, characterized in that, The formulas for calculating the foreground tissue score and the control tissue score at each pixel location are as follows: ； ； in, pixel position The foreground organization score, where i is the image index of the predicted result. pixel position Dynamically adjusted weights on the predicted image i. For indicator functions, in In the middle, if but Select 1 if the value is 1, otherwise select 0. Indicates pixel position It is classified as a foreground region in the predicted image i; pixel position The control tissue score, in In the middle, if but Select 1 if the value is 1, otherwise select 0. Indicates pixel position It is classified as the control region in the predicted image i.

8. The method for classifying tissue regions in immunohistochemical images based on multi-scale stacking according to claim 1, characterized in that, Before mapping the modified immunohistochemistry mask array to the immunohistochemistry image, the modified immunohistochemistry mask array is modified a second time through connected component analysis. During the connected component analysis, a minimum effective area is preset based on the area of ​​the immunohistochemistry image. Connected components are extracted from the modified immunohistochemistry mask array, and connected components with an area smaller than the minimum effective area are obtained as small region connected components. Each pixel position in the small region connected component that only includes the classification results of the control tissue is modified into the background region.

9. A tissue region classification device based on multi-scale stacking immunohistochemical images, characterized in that, include: The module constructs an image sequence based on immunohistochemical images. The image sequence includes different scale representations of the immunohistochemical images. A pre-trained classification model is used to classify and label the tissue regions of each image in the image sequence to obtain a classification sequence. The classification sequence includes prediction result images corresponding to immunohistochemical images at different scale representations. The tissue regions include foreground tissue and control tissue. The stacking module is used to transform each predicted result image in the classification sequence to the same scale and stack the images to obtain an immunohistochemistry mask array. It calculates the classification consistency and classification confidence of each pixel position in the immunohistochemistry mask array in all predicted result images, and calculates the dynamic correction weight of each pixel position in different predicted result images based on the classification consistency and pixel confidence. The correction module is used to segment the immunohistochemical mask array into a region of tissue to be corrected and a background region. It calculates the foreground tissue score and control tissue score for each pixel in the region of tissue to be corrected using dynamic correction weights at corresponding pixel positions. Based on these scores, it reclassifies each pixel in the region of tissue to be corrected to obtain the corrected region. Specifically, the sum of the foreground and control tissue scores for each pixel is calculated as the total pixel score. If the proportion of the foreground tissue score in the total pixel score is greater than a first threshold, the corresponding pixel is classified as foreground. Organization: If the proportion of the control organization score at a pixel location to the total pixel score is greater than a first threshold, then the corresponding pixel location is classified as a control organization; if the difference between the foreground organization score and the control organization score at a pixel location is less than or equal to a second threshold, then the corresponding pixel location is classified as a control organization; if the foreground organization score is greater than the control organization score and the difference between the foreground organization score and the control organization score is greater than a second threshold, then the corresponding pixel location is classified as a foreground organization; if the control organization score is greater than the foreground organization score and the difference between the foreground organization score and the control organization score is greater than a second threshold, then the corresponding pixel location is classified as a control organization. The output module integrates the corrected tissue region and the background region as a corrected immunohistochemical mask array, and maps the corrected immunohistochemical mask array onto the immunohistochemical image to obtain the immunohistochemical image classification result.

Images