An image registration method to assist in the diagnosis of heterogeneity in multi-stained pathological sections

Abstract

This invention belongs to the field of medical image pathology analysis and discloses an image registration method to assist in the diagnosis of heterogeneity in multi-stained pathological sections. The method first creates a large number of keypoints on a whole clinical slide image to mark the features of various stained biomarkers on the slide image; then, it retrieves information on the similarity of biomarker features around the keypoints and the consistency of the spatial distribution of the keypoints to obtain matching landmarks; finally, based on the matching landmarks, it uses a combination of rigid and affine transformation models to calibrate local regions of the whole slide image to obtain co-localization regions of homologous but different stained biomarkers in the slide tissue. Taking the registration of a whole breast cancer tissue slide image as an example, the method uses the biomarker-positive area contained in the registered continuous co-localization regions to analyze the Ki-67 biomarker index of breast cancer tissue to assist pathologists in the routine diagnosis of heterogeneity.

Description

Technical Field

[0001] This invention belongs to the field of medical image pathology analysis, and in particular relates to an image registration method to assist in the diagnosis of heterogeneity in multi-stained pathological sections. Background Technology

[0002] Multistaining pathological section heterogeneity refers to the phenomenon where, during multiple mitotic divisions, changes in cell molecular biology or gene expression can lead to the proliferation of different cell subpopulations within a tumor, resulting in tumor drug resistance. Clinical diagnosis of breast cancer has entered the era of molecular subtyping. Currently, pathology departments rely heavily on accurate molecular subtyping assessments to provide diagnoses of breast cancer. However, heterogeneity can lead to inaccurate molecular subtyping results, thus affecting the development of reliable treatment plans.

[0003] To avoid potential treatment failure, the combined interpretation of different stained biomarkers in consecutive whole-section images provides a suitable solution for tumor treatment. Taking H&E, ER, and Ki-67 stained breast cancer sections as an example, H&E sections are typically prepared using 4-micron-thick tissue sections stained with hematoxylin and eosin. The nuclei of breast cancer tumor cells are stained blue, while the cytoplasm appears red, providing intuitive morphological knowledge of the tumor tissue for pathological diagnosis. ER (estrogen receptor) and Ki-67 are both immunohistochemical staining methods (IHC) and are important immunohistochemical indicators for deciding breast cancer treatment. Fully positive ER sections serve as a control group for assessing Ki-67 positivity levels, effectively reflecting the Ki-67 index. By eliminating irrelevant background cells with non-positive expression in ER sections, the retained ER-positive biomarkers are used as a benchmark to assess Ki-67 heterogeneity levels. Ki-67 positivity levels are closely related to cell proliferation and can reflect the malignancy of tumor cells; higher heterogeneity levels are considered to lead to breast cancer proliferation and metastasis, or to indicate that tumor cells have spread to other parts of the body.

[0004] However, during clinical pathology examinations, pathologists often need to overcome color differences, rotation, and deformation issues between slides to manually locate local co-localized areas in order to assess the level of Ki-67 heterogeneity. This leads to a surge in the workload of pathology departments and severely limits the clinical application of large-scale pathological slides in breast cancer treatment. Furthermore, because different pathologists often need to focus on limited local areas when manually registering multi-stained slides, this experience-based approach may result in differences in assessing the Ki-67 heterogeneity level of the same patient. Therefore, it is essential to develop more objective auxiliary heterogeneity assessment methods for pathological examinations. Summary of the Invention

[0005] To address the practical problems encountered in routine diagnosis and to assist pathologists in analyzing the heterogeneity of multi-stained pathological sections, this invention discloses an image registration method to assist in the diagnosis of heterogeneity in multi-stained pathological sections.

[0006] The image registration method for assisting in the diagnosis of heterogeneity in multi-stained pathological sections disclosed in this invention is implemented using the following technical approach:

[0007] Step 1: Use whole slice images with different staining as the images to be registered, and mark key points on the images to be registered;

[0008] Step 2: Using a key point screening scheme, matching landmarks between images to be registered are obtained by iterating the feature similarity and spatial distribution consistency between different key points.

[0009] Step 3: Use matching landmarks to calculate the pixel coordinates of homologous biomarkers and the image rotation angle between the images to be registered, which are used to locate local colocalization regions and calibrate multichromosome biomarkers, respectively.

[0010] Step four: The entire tumor tissue is divided into blocks to obtain a series of calibrated colocalization image blocks. The ratio of the positive area of ​​different stained biomarkers in the colocalization image blocks is calculated to help analyze heterogeneity.

[0011] Preferably, step one includes the following:

[0012] The whole slice image represents a tumor tissue slice obtained by staining with H&E (hematoxylin-eosinstaining) and IHC (immunohistochemistry staining).

[0013] The images to be registered can be composed of various combinations, including whole pathological slide images stained with H&E and IHC, or whole pathological slide images stained with different types of IHC.

[0014] The images to be registered also include the number of images to be registered, which can consist of multiple whole pathological slide images, including but not limited to 2 or 3 images;

[0015] The key point marking method can be obtained using a feature point extraction algorithm, including the SIFT feature extraction algorithm and the FAST corner detection algorithm.

[0016] The keypoint marking method can also directly use pixel coordinates as keypoints.

[0017] Furthermore, the method of directly using pixel coordinates as key points involves first reducing the size of the slice image to be registered to 128x128 pixels, and then enlarging all pixel coordinates of the stained area of ​​the slice image to the target image size range to use pixel coordinates as key points.

[0018] As a further improvement to step two, the key point screening scheme in step two includes the following sub-steps:

[0019] (2-1) Use feature similarity descriptors to encode the similarity of biomarker features around key points to obtain the feature vector of key points;

[0020] (2-2) Calculate the similarity of biomarkers around keypoints in terms of Euclidean distance between each pair of feature vectors, and use the most similar paired feature points as coarse matching landmarks between images to be registered.

[0021] (2-3) Use the spatial distribution consistency rule to delete inaccurate matching landmarks that may exist in the initial matching landmarks, and retain accurate matching landmarks as fine matching landmarks;

[0022] (2-4) Based on the fine-matched landmarks, the double-circle mapping algorithm is used to guide the iterative operation process, and the registration landmarks are iterated from the unpaired key points to obtain more potential fine-matched landmarks as the final matching result.

[0023] Preferably, in step (2-1), the biomarker features include: biomarker staining features, texture structure features, and pixel energy features.

[0024] Preferably, in step (2-1), the feature vector of the key point is obtained by encoding multiple biomarker features using a key point descriptor algorithm, wherein the key point descriptor algorithm includes: BRIEF and SURF descriptor algorithms.

[0025] Furthermore, in step (2-3), the algorithm for the spatial distribution consistency rule includes:

[0026] Calculate the Euclidean distance between all matching landmarks, and solve for the sum of the Euclidean distances from each landmark to other landmarks as a quantitative index of the feature space distribution. The quantitative index follows the formula:

[0027]

[0028]

[0029] Where D is the Euclidean distance from the matched landmarks to other landmarks in the first slice image to be matched, d represents the Euclidean distance from the matched landmarks to other landmarks in the second slice image to be matched, and i represents the number of matched landmarks. and ε represents the iteration termination threshold when two adjacent iteration points are connected. j represents the iteration number and ε represents the iteration termination threshold. Ideally, the spatial distribution of matching landmarks on the first and second images is completely consistent, and the iteration termination threshold is 0. For non-ideal scenarios where the entire slice image is deformed, the iteration termination threshold approaches 0.

[0030] Furthermore, in steps (2-3), the finely matched landmarks simultaneously satisfy the conditions of the highest level of biomarker similarity and key point spatial distribution consistency.

[0031] Further, in steps (2-3), the dual-circle mapping algorithm uses the positional distance relationship between matching landmarks and unmatched feature points in the images to be registered to map the correspondence of unmatched key points, thereby guiding the matching of more potential landmarks. The dual-circle mapping algorithm includes the following steps:

[0032] Use at least three pairs of finely matched landmarks, with the coordinates of the paired landmarks represented as: [C1(x1,y1),C'1(x'1,y'1)], [C2(x2,y2),C'2(x'2,y'2)], [C3(x3,y3),C'3(x'3,y'3)];

[0033] Calculate the Euclidean distance from the unmatched keypoints to the matching landmarks on one of the whole slice images, and the distances are denoted as S1, S2, and S3;

[0034] Using distances S1 and S2 as radii and the matched landmarks C'1(x'1,y'1) and C'2(x'2,y'2) as centers, establish circle equations on another whole slice image, and solve for the coordinates of the intersection point of the two circles. The circle equations are as follows:

[0035] (x-x'1) 2 +(y-y'1) 2 =S1 2

[0036] (x-x'2) 2 +(y-y'2) 2 =S2 2

[0037] The coordinates of the intersection point of the circles are represented as O. α (x α ,y α ), O β (x β ,y β );

[0038] A discriminant for the intersection point coordinates is established using distance S3, and the discriminant is defined as follows:

[0039]

[0040] Where S4 represents C'3(x'3,y'3) and O α (x α ,y α Euclidean distance, S5 represents the distance between C'3(x'3,y'3) and O. β (x β ,y β Euclidean distance.

[0041] Further, in step (2-4), the iterative calculation process includes: using double-circle mapping to map unmatched key points based on fine-matched landmarks, then performing similarity comparison to obtain coarse-matched landmarks and constraining the coarse-matched landmarks to simultaneously satisfy the spatial distribution consistency relationship to obtain fine-matched landmarks, iterating repeatedly until no new fine-matched landmarks are generated, and all the fine-matched landmarks obtained at this time are taken as the final matching result.

[0042] As a further improvement to step three, the calculation of the pixel coordinates of homologous biomarkers and the image rotation angle employs affine transformation and rigid transformation models, respectively. Specifically, the coordinates are first located and a co-localization region is extracted. Then, the rotation angle is corrected using the center of the extracted co-localization region as the rotation center, ensuring that the biomarkers contained within the co-localization region are in the same Cartesian coordinate system, thus completing the biomarker registration task.

[0043] As a further improvement to step four, the auxiliary analysis of tumor heterogeneity should include the following sub-steps:

[0044] (4-1) Divide the whole slice image stained with HE into 500x500 micrometer image blocks, register each block on two IHC stained whole slice images to obtain continuous corresponding colocalization regions, which are used to form multiple sets of registration blocks.

[0045] (4-2) Calculate the ratio of the biomarker-positive area contained in each pair of colocalized small regions on two IHC-stained whole-section images and draw a heat map to assist in tumor heterogeneity analysis.

[0046] Further, in step (4-2), the method for calculating the ratio of the positive areas of the biomarkers includes:

[0047] The color segmentation algorithm is used to segment the positive biomarker regions in the co-localization blocks of IHC1 and IHC2 using a color gradient threshold. The specific formula for the color segmentation algorithm is as follows:

[0048] P=abs(RG)+abs(RB)+abs(GB)>T1 Q=abs(RG)+abs(RB)+abs(GB)>T2

[0049] Where P and Q are binary decision maps, representing the segmentation results of the positive regions of IHC1 and IHC2 respectively, and R, G, and B correspond to the pixel values ​​of the three dimensions of the slice image. T1 = 180 and T2 = 170 are set to represent the color gradient thresholds of IHC1 and IHC2 respectively.

[0050] Calculate the ratio of the area of ​​IHC1 positive region to the area of ​​IHC2 positive region, using the following formula:

[0051] K i =sum(P i ) / sum(Q i (i = 1, 2, 3, ...)

[0052] Among them, K i represents the ratio of positive region areas, i represents the number of co-localized patches, and sum represents the summation of the areas of positive biomarkers on the binary decision graph.

[0053] The beneficial effects of this invention are as follows:

[0054] Compared with existing methods, this invention extracts potential effective features by directly arranging a sufficient number of keypoints on each slice image, rather than relying on specific feature structures. This alleviates the problem of insufficient keypoint extraction caused by color differences, rotation, and deformation between consecutive slices. This invention designs an iterative algorithm to constrain matching landmarks in two ways: matching landmarks simultaneously satisfy high biomarker similarity and consistent spatial distribution of keypoints, increasing the number of registered landmarks while improving the reliability of the matching results. To register consistent biomarker information between multi-stained slices, this invention performs local co-localization calibration by dividing the entire slice image into blocks, overcoming the limitation of the massive computational load required for directly calibrating the entire slice image. In assisting the interpretation of tumor tissue heterogeneity, this invention supports the simultaneous registration of three entire slice images (one or two IHC images), and calculates the ratio of biomarker areas in the two IHC stained images based on the registered blocks to draw a heterogeneity heatmap. Taking breast cancer heterogeneity analysis as an example, by registering whole-slice images stained with H&E, ER, and Ki-67, objective references are provided to assist in the clinical interpretation of Ki-67 heterogeneity. Attached Figure Description

[0055] Figure 1 This is a flowchart of an image registration method for assisting in the diagnosis of heterogeneity in multi-stained pathological sections, provided in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of an image registration method for assisting in the diagnosis of heterogeneity in multi-stained pathological sections, provided in an embodiment of the present invention, which directly uses pixel coordinates as key points.

[0057] Figure 3 This is a schematic diagram of the registration algorithm framework for an iterative method of an image registration method for assisting in the diagnosis of heterogeneity in multi-stained pathological sections provided in an embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of the spatial distribution consistency algorithm for an image registration method to assist in the diagnosis of heterogeneity of multi-stained pathological sections provided in an embodiment of the present invention.

[0059] Figure 5 This is a schematic diagram of a dual-circle mapping algorithm for an image registration method to assist in the diagnosis of heterogeneity in multi-stained pathological sections, provided in an embodiment of the present invention.

[0060] Figure 6 This is a local co-localization region registration scheme for an image registration method to assist in the diagnosis of heterogeneity of multi-stained pathological sections provided in an embodiment of the present invention.

[0061] Figure 7 This is a schematic diagram of the Ki-67 heterogeneity-assisted analysis algorithm for an image registration method that assists in the diagnosis of heterogeneity in multi-stained pathological sections, provided in an embodiment of the present invention. Detailed Implementation

[0062] To clearly understand the purpose, technical solution, and advantages of this invention, the following description, in conjunction with the accompanying drawings and embodiments, further elaborates on the invention. It should be understood that the embodiments described are intended to explain the purpose of the invention and are not intended to limit the invention.

[0063] Example 1:

[0064] See Figure 1 , Figures 3 to 6 This illustrates a method for matching two whole slice images provided by an embodiment of the present invention, the steps of which are detailed below:

[0065] Step 1: Use one H&E and one IHC (P53) stained whole gastric cancer pathological slide image as the images to be registered to form the H&E-P53 slide image registration task. Then, use the FAST corner detection algorithm to mark key points on the images to be registered.

[0066] Step 2: Use the key point filtering scheme ( Figure 3 By iterating through the feature similarity and spatial distribution consistency between different key points, matching landmarks between H&E and P53 images are obtained. The specific implementation steps are as follows:

[0067] (2-1) Use the BRIEF feature similarity descriptor to encode the texture structure features and pixel energy features of biomarkers around the keypoints to obtain a 256+256=512-dimensional feature vector of the keypoints;

[0068] (2-2) Calculate the similarity of biomarkers around keypoints by representing the Euclidean distance between each pair of feature vectors, and use the most similar paired feature points as coarse matching landmarks between the images to be registered. The biomarker similarity level is calculated as follows:

[0069]

[0070] Where represents the similarity level between two feature vectors, ρ is the number of elements in the feature vector, (x ρ -y ρ () represents the difference between corresponding elements of two vectors, reflecting the basic distance unit between vectors.

[0071] (2-3) Use the spatial distribution consistency rule to delete inaccurate matching landmarks that may exist in the initial matching landmarks, and retain accurate matching landmarks as fine matching landmarks;

[0072] Calculate the Euclidean distance between all matching landmarks, and solve for the sum of the Euclidean distances from each landmark to other landmarks as a quantitative index of the feature space distribution. The quantitative index follows the formula:

[0073]

[0074]

[0075] Where D is the Euclidean distance from the matched landmarks to other landmarks in the first slice image to be matched, d represents the Euclidean distance from the matched landmarks to other landmarks in the second slice image to be matched, and i represents the number of matched landmarks. and Let j represent the number of iterations and ε represent the iteration termination threshold. Ideally, the spatial distribution of matching landmarks on the first and second images is completely consistent, in which case the iteration termination threshold is 0. For non-ideal scenarios where the entire slice image is deformed, the iteration termination threshold approaches 0. In this invention... Figure 4 The document provides an example explanation, showing how setting the number of 'i' to 5 yields matching landmarks: C1-C'1, C2-C'2, C3-C'3, C4-C'4, and C5-C'5. Taking the spatial distribution quantification indicators of C1-C'1 and C4-C'4 as examples, they satisfy the following formula:

[0076] V1 = abs(D1 - d1) + abs(D3 - d3) + abs(D2 - d2) + abs(D4 - d4)

[0077] V4 = abs(D2 - d2) + abs(D5 - d5) + abs(D6 - d6) + abs(D7 - d7)

[0078] Where, V1 and V4 are the spatial distribution quantization indices of C1 - C'1 and C4 - C'4 respectively. It is easy to understand that in the matching landmark C1 - C'1, abs(D1 - d1) + abs(D4 - d4) tends to 0 while C4 - C'4 does not have a similar feature. Therefore, V1 < V4. At this time, in the iterative operation process, the coarsely matched landmark C4 - C'4 is deleted. It can be predicted that the finally retained finely matched landmarks after multiple iterative operations can meet the iterative termination condition: Figure 3 And the finely matched landmarks maintain a high level of spatial distribution consistency among them.

[0079] (2 - 4) Based on the finely matched landmarks, use the double - circle mapping algorithm ( Figure 5 ) to guide the iterative operation process, and iterate out the registration landmarks from the unpaired key points to obtain more potential finely matched landmarks as the final matching result. The double - circle mapping algorithm includes the following steps:

[0080] Use at least three pairs of finely matched landmarks, and the coordinates of the paired landmarks are represented as: [C1(x1, y1), C'1(x'1, y'1)], [C2(x2, y2), C'2(x'2, y'2)], [C3(x3, y3), C'3(x'3, y'3)];

[0081] Calculate the Euclidean distances from the unmatched key points to the matched landmarks on one of the whole - slice images, and the distances are represented as S1, S2, S3;

[0082] Taking the distances S1, S2 as radii and the matched landmarks C'1(x'1, y'1), C'2(x'2, y'2) as the centers of the circles, establish circle equations on the other whole - slice image respectively, and solve the intersection coordinates of the two circles. The circle equations are as follows:

[0083] (x - x'1) 2 +(y - y'1) 2 = S1 2

[0084] (x - x'2) 2 +(y - y'2) 2 = S2 2

[0085] The intersection coordinates of the solved circles are represented as Oα (x α ,y α ), O β (x β ,y β );

[0086] A discriminant for the intersection point coordinates is established using distance S3, and the discriminant is defined as follows:

[0087]

[0088] Where S4 represents C'3(x'3,y'3) and O α (x α ,y α Euclidean distance, S5 represents the distance between C'3(x'3,y'3) and O. β (x β ,y β Euclidean distance.

[0089] The iterative calculation process uses fine-matched landmarks as the basis to map unmatched key points using double circles, and then performs similarity comparison to obtain coarse-matched landmarks. It also constrains the coarse-matched landmarks to simultaneously satisfy the spatial distribution consistency relationship to obtain fine-matched landmarks. This process is repeated until no new fine-matched landmarks are generated. All the fine-matched landmarks obtained at this point are taken as the final matching result.

[0090] Example 2:

[0091] See Figures 1 to 7 The present invention illustrates the steps of an image registration method for analyzing the heterogeneity of Ki-67 biomarkers in breast cancer, as provided in an embodiment of the present invention, detailed below:

[0092] Step 1: Use three consecutive whole-section breast cancer pathology slides stained with H&E, IHC (ER), and IHC (Ki-67) as the images to be registered, forming the H&E-ER, H&E-Ki-67, and ER-Ki-67 slide image registration task. Then, directly use the pixel coordinates ( Figure 2 Mark key points on the image to be registered.

[0093] Step 2: Use the key point filtering scheme ( Figure 3 By iterating the feature similarity and spatial distribution consistency between different key points, matching landmarks between H&E-ER, H&E-Ki-67, and ER-Ki-67 images are obtained respectively.

[0094] Step 3: Calculate the pixel coordinates and image rotation angles of homologous biomarkers using the matching landmarks between H&E-ER, H&E-Ki-67, and ER-Ki-67 images respectively, and locate local co-localization regions and calibrate multichromosome biomarkers in their respective tasks.

[0095] The pixel coordinates and image rotation angles of the local co-localization region are calculated using affine transformation and rigid transformation models, respectively. Local co-localization region registration follows the principle of separating localization and rotation correction, as follows: Figure 6 As shown, the coordinate positions are first located and the co-location area is extracted. Then, the rotation angle is corrected by using the center of the extracted co-location area as the rotation center, so that the biomarkers in the co-location area are in the same Cartesian coordinate system.

[0096] Furthermore, the algorithm for predicting the location of local co-localized regions in an affine transformation model includes the following steps:

[0097] (1) Predict affine transformation parameters based on matching landmarks;

[0098] (2) Based on the predicted affine transformation parameters, the coordinates of the local colocalized region on another slice image are predicted by inputting the coordinates of the breast cancer tumor tissue on one slice image.

[0099] Furthermore, the algorithm for estimating the rotation angle in the rigid transformation model includes the following steps:

[0100] (1) Predict rigid transformation parameters based on matching landmarks;

[0101] (2) Based on the predicted rigid transformation parameters, the horizontal unit vector is used as the input of the rigid transformation to map the transformed direction vector.

[0102] (3) Calculate the slope of the direction vector and convert the slope into a rotation angle. Since the direction vector is a horizontal vector before the transformation, the rotation angle at this time is the rotation angle between the two slice images in the current registration task.

[0103] Step 4: Obtain a series of calibrated colocalization image patches from the entire tumor tissue on the segmented H&E image. Calculate the ratio of the positive area of ​​different stained biomarkers in the colocalization image patches to assist in the analysis of tumor heterogeneity. In the embodiments of the present invention, such as... Figure 7 As shown, the Ki-67 heterogeneity analysis for breast cancer includes the following steps:

[0104] (1) For a full breast cancer pathological slide image magnified 40 times, a block method with a size of 500x500 micrometers (2000x2000 pixels) was used. The HE block was used as the reference to register the ER and Ki-67 blocks to obtain multiple sets of registration blocks composed of co-localized regions of three HE-ER-Ki-67 stained images.

[0105] (2) Compare the ratio of Ki-67 positive biomarkers to ER positive biomarkers within each co-localized small block region as the Ki-67 index, and simultaneously record the coordinate position of the small block corresponding to the current Ki-67 index to guide the drawing of Ki-67 heterogeneity heatmap.

[0106] (3) All calculated Ki-67 indices were used to analyze Ki-67 heterogeneity.

[0107] Furthermore, a color segmentation algorithm was used to set a color gradient threshold to segment the positive biomarker regions in the ER and Ki-67 co-localization blocks respectively. The specific formula for the color segmentation algorithm is as follows:

[0108] P=abs(RG)+abs(RB)+abs(GB)>T1 Q=abs(RG)+abs(RB)+abs(GB)>T2

[0109] In this context, P and Q are binary decision maps, representing the segmentation results of the positive regions of ER and Ki-67, respectively. R, G, and B correspond to the pixel values ​​of the three dimensions of the slice image. T1 = 180 and T2 = 170 are set to represent the color gradient thresholds of ER and Ki-67, respectively.

[0110] Furthermore, the Ki-67 index is calculated as the ratio of the area of ​​the Ki-67 positive region to the area of ​​the ER positive region. The specific formula for calculating the Ki-67 index is as follows:

[0111] K i =sum(P i ) / sum(Q i (i = 1, 2, 3, ...)

[0112] Among them, K i denoted by Ki-67 index, i represents the number of colocalized patches, which is also the number of Ki-67 indices, and sum represents the area of ​​positive biomarkers summed over the binary decision graph.

[0113] In summary, the image registration method for assisting in the diagnosis of heterogeneity in multi-stain pathological sections disclosed in this invention has the following technical advantages. This invention supports the simultaneous registration of multiple consecutive slice images with different staining, overcoming the difficulties of existing technologies in registering biomarkers related to color differences, rotation, and local deformation between slice images. This provides a technical foundation for subsequent analysis of multi-stain pathological section heterogeneity (e.g., Ki-67 in breast cancer) based on registration results. Furthermore, this invention generates a Ki-67 heterogeneity heatmap for assisting clinical diagnosis by calculating the Ki-67 index in all locally co-located patches within the tumor region, providing a valuable reference for clinical pathologists to interpret tumor heterogeneity. This invention will help pathology departments improve the clinical diagnostic level of cancer by assessing the heterogeneity of multi-stain pathological sections.

[0114] Although the present invention has been described in detail in the above description and accompanying drawings, it should be understood that the above specific embodiments and illustrations are only used to illustrate the technical solutions of the present invention and are not restrictive; those skilled in the art will understand many modified and varied embodiments by studying the disclosure of the present invention, and all such modifications and variations that do not depart from the spirit and scope of the technical solutions of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. An image registration method for assisting in the diagnosis of heterogeneity in multi-stained pathology sections, characterized in that, The method comprises the following steps: Step 1: using different color-stained whole slide images as the images to be registered, marking key points on the images to be registered; The whole slide images are obtained from tumor tissue sections stained by H&E and IHC; Step 2: using a key point screening scheme, obtaining matching landmarks between the images to be registered by iteratively calculating the feature similarity and spatial distribution consistency between different key points; Step 3: using the matching landmarks to calculate the pixel coordinate positions and image rotation angles of the homologous biomarkers between the images to be registered, which are respectively used for positioning the local co-localization regions and calibrating the multi-stained biomarkers; Step 4: dividing the whole tumor tissue into a series of calibrated co-localization image blocks, and calculating the ratio of the positive areas of different stained biomarkers in the co-localization image blocks to assist in heterogeneity analysis; In step 4, the auxiliary heterogeneity analysis comprises the following sub-steps: (4-1) dividing the whole slide image stained by H&E into 500x500 micrometer image blocks, and registering each block on the two whole slide images stained by IHC to obtain a series of corresponding co-localization regions, which are used to form a plurality of registered blocks; (4-2) calculating the ratio of the biomarker positive areas contained in each pair of co-localization block regions on the two whole slide images stained by IHC and drawing a heat map to assist in heterogeneity analysis; The method for calculating the ratio of the biomarker positive areas comprises: using a color segmentation algorithm to set a color gradient threshold to segment the positive biomarker regions in the IHC1 and IHC2 co-localization blocks, respectively, and the specific formula of the color segmentation algorithm is: wherein, , are binary decision maps representing the segmentation results of the positive regions of IHC1 and IHC2, respectively, , , corresponding to the pixel values of the three dimensions of the slice image, T1 and T2 represent the color gradient threshold values of IHC1 and IHC2, respectively; calculating the ratio of the IHC1 positive area to the IHC2 positive area, and the formula of the ratio is: wherein, the ratio of the area of the positive region, represents the number of co-localized small patches, sum represents the sum over the binary decision map representing the area of the positive biomarker.

2. The image registration method for assisting the diagnosis of heterogeneity of multi-stained pathological sections according to claim 1, characterized in that: In step 1, the following contents are included: The images to be registered are composed of whole pathological section images stained by H&E and IHC, or whole pathological section images stained by different types of IHC; The images to be registered are composed of a plurality of whole pathological section images; The marking method of the key points uses a feature point extraction algorithm, which includes a SIFT feature extraction algorithm and a FAST corner detection algorithm; Or directly using pixel coordinates as key points.

3. The image registration method for assisting the diagnosis of heterogeneity of multi-stained pathological sections according to claim 2, characterized in that: The method of first reducing the size of the images to be registered to 128x128 pixels, and then enlarging all pixel coordinates of the stained regions of the section images to the target image size range is used to take the pixel coordinates as the key points.

4. The image registration method for assisting the diagnosis of heterogeneity of multi-stained pathological sections according to claim 1, characterized in that: In step 2, the key point screening scheme comprises the following sub-steps: (2-1) using a feature similarity descriptor to encode the biomarker features around the key points to obtain a feature vector of the key points; (2-2) calculating the Euclidean distance between each pair of feature vectors to represent the similarity of the biomarkers around the key points, and taking the most similar paired feature points as the coarse matching landmarks between the images to be registered; (2-3) using a spatial distribution consistency rule to delete inaccurate matching landmarks that may exist in the initial matching landmarks, and retaining accurate matching landmarks as fine matching landmarks; (2-4) Based on the fine matching landmarks, the double circle mapping algorithm is used to guide the iterative operation process to iterate the registration landmarks from the unmatched key points to obtain more potential fine matching landmarks as the final matching results.

5. The image registration method for assisting the diagnosis of heterogeneity of multi-stained pathological sections according to claim 4, characterized in that: In step (2-1), the biomarker features include biomarker staining features, texture structure features, and pixel energy features. The feature vector of the key point is obtained by encoding the biomarker features by a key point descriptor algorithm, wherein the key point descriptor algorithm includes BRIEF and SURF descriptor algorithms.

6. The image registration method for assisting the diagnosis of heterogeneity of multi-stained pathological sections according to claim 4, characterized in that: In step (2-3), the algorithm of the spatial distribution consistency rule includes: The Euclidean distance between all matching landmarks is calculated, and the sum of the Euclidean distance of each landmark to other landmarks is solved as a quantitative indicator of the feature space distribution, which complies with the following formula: wherein, is the Euclidean distance of the matched landmark on the first slice image to other landmarks, represents the Euclidean distance of the matched landmark on the second slice image to other landmarks, represents the number of matched landmarks, and represents when the two adjacent iteration points, represents the iteration number, represents the iteration termination threshold, ideally the spatial distribution of the matched landmarks on the first and second images is completely consistent, at this time the iteration termination threshold is 0, for the non-ideal scene due to the deformation of the whole slice image, the iteration termination threshold tends to 0; The fine matching landmarks simultaneously satisfy the highest level of biomarker similarity and key point spatial distribution consistency in two dimensions.

7. The image registration method of claim 4, wherein: In step (2-4), the double circle mapping algorithm uses the position distance relationship between the matching landmarks and the unmatched feature points between the images to be registered to map the corresponding relationship of the unmatched key points, thereby guiding the matching of more potential landmarks, and the double circle mapping algorithm includes the following steps: Using at least three pairs of precisely matched landmarks, the coordinates of the paired landmarks are represented as: , , ; The Euclidean distance of the unmatched keypoint to the matched landmark is computed on one of the whole slice images, which is denoted as , , ; with the distance , as the radius, the matching landmarks , as the center of the circle, a circle equation is established on the other whole slice image respectively, and the intersection coordinates of the two circles are solved, and the circle equation is as follows: Solving the circle intersection point coordinates is expressed as , ; Using distance An intersection coordinate discriminator is established, defined as follows: wherein denotes and Euclidean distance, denotes and Euclidean distance; The iterative operation process includes: using the double circle mapping algorithm to map the unmatched key points based on the fine matching landmarks, then performing similarity comparison to obtain coarse matching landmarks and constraining the coarse matching landmarks to simultaneously satisfy the spatial distribution consistency relationship to obtain fine matching landmarks, and repeatedly iterating until no new fine matching landmark is generated, at which time all the obtained fine matching landmarks are taken as the final matching results.

8. The image registration method of claim 1, wherein: In step three, the pixel coordinate position and image rotation angle of the homologous biomarker are calculated by using affine transformation and rigid transformation models, that is, the coordinate position is first located and the co-localization region is intercepted, then the center position of the intercepted co-localization region is taken as the rotation center to correct the rotation angle, so that the biomarkers contained in the co-localization region are in the same Cartesian coordinate system, and the task of registering the biomarkers is completed.

Images