A method for classifying renal tubular segments in pathological section staining images

Abstract

This invention relates to a method for classifying renal tubular segments from stained images of pathological sections, belonging to the field of renal tubular classification technology, and solving the problem of inaccurate segment differentiation in existing technologies. The method includes: acquiring stained images and multiplex fluorescence images of pathological sections; performing multi-level matching between the stained images and a reference fluorescence channel to obtain local matching point pairs for each usable pathological strip in the stained image; for each usable pathological strip, determining the affine transformation matrix corresponding to each renal tubular contour in the usable pathological strip based on the distribution of local matching point pairs and renal tubular contours; for each renal tubular contour, obtaining the region corresponding to the renal tubular contour in the multiplex fluorescence image based on the corresponding effective affine transformation matrix, and determining the type of the renal tubular contour based on the intensity of the region corresponding to the renal tubular contour in different classification fluorescence channels. This achieves efficient and accurate classification of renal tubular segments.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of renal tubule classification, and particularly relates to a renal tubule segment classification method for pathological section staining images. BACKGROUND

[0002] Pathology plays a vital role in the medical field. After kidney tissue is punctured by a thick needle or resected, pathological sections are prepared, and doctors make pathological diagnoses in combination with images under a microscope and clinical information to guide clinical treatment. Staining can highlight specific structures, such as glomerular basement membranes or glycogen, which helps to distinguish normal tissue from abnormal tissue and is a commonly used staining technique for pathological sections.

[0003] With the continuous development of artificial intelligence and deep learning in recent years, deep learning is increasingly applied to the medical field. In order to train an image semantic segmentation intelligent model, various structures including renal tubules in kidney pathological sections need to be labeled. Renal tubules can be divided into proximal tubules (including proximal convoluted tubules, proximal straight tubules and descending thin segment of the loop of Henle), distal tubules (including ascending thin segment of the loop of Henle, ascending thick segment of the loop of Henle and distal convoluted tubule), connecting tubules and collecting ducts. Distinguishing each segment of the renal tubule is crucial for accurate disease diagnosis and the establishment of a prediction model. Although staining can distinguish normal proximal and distal tubules by displaying brush borders, the characteristics of brush borders are easily lost in the disease state, and the differences in cytoplasmic eosinophilia between distal tubules and proximal tubules are easily affected by staining batches or tissue fixation quality under conventional light microscopy. Therefore, labeling personnel are prone to confuse distal and proximal renal tubules due to color differences, and thus conventional staining cannot accurately distinguish distal tubules from proximal tubules, and it is even more difficult to finely distinguish each segment of the renal tubule. SUMMARY

[0004] In view of the above analysis, the embodiments of the present application aim to provide a renal tubule segment classification method for pathological section staining images to solve the problem that the prior art cannot accurately distinguish the types of renal segment segments.

[0005] In one aspect, the embodiments of the present application provide a renal tubule segment classification method for pathological section staining images, comprising the following steps:

[0006] obtaining a staining image and a multiple fluorescence image of a pathological section, wherein the multiple fluorescence image comprises a reference fluorescence channel and a plurality of classification fluorescence channels;

[0007] performing multi-level matching on the staining image and the reference fluorescence channel to obtain a local matching point pair of each available pathological strip in the staining image;

[0008] for each available pathological strip, determining an affine transformation matrix corresponding to each renal tubule contour in the available pathological strip based on the local matching point pair of the available pathological strip and the distribution of the renal tubule contour;

[0009] For each tubular contour, if the affine transformation matrix corresponding to the tubular contour is valid, then based on the valid affine transformation matrix corresponding to the tubular contour, a region in the multiple fluorescence image corresponding to the tubular contour is obtained, and based on the intensity of the region in the multiple fluorescence image corresponding to the tubular contour in different classification fluorescence channels, the type of the tubular contour is obtained.

[0010] Based on a further improvement of the above method, the affine transformation matrix corresponding to each tubular contour in the available pathological strip is determined based on the local matching point pairs in the available pathological strip, comprising:

[0011] Determine the image block to which each tubular contour in the available pathological strip belongs;

[0012] For each image block, the affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs in the image block;

[0013] The affine transformation matrix of each tubular contour of the image block is the affine transformation matrix corresponding to the image block.

[0014] Based on a further improvement of the above method, the image block to which each tubular contour in the available pathological strip belongs is determined in the following way:

[0015] For each tubular contour in the available pathological strip that does not belong to any image block, the circumscribed frame of the tubular contour is taken as the current region;

[0016] The top-left corner of the current region is taken as the top-left corner of the window, and a preset length is taken as the side length to obtain the current window. If there is a tubular contour in the current window that is outside the current region and does not belong to any image block, then the common circumscribed frame of the tubular contours in the current region and the tubular contour in the current window that is outside the current region and does not belong to any image block is taken as the current region. If not, the current region remains unchanged;

[0017] The top-right corner of the current region is taken as the top-right corner of the window, and a preset length is taken as the side length to obtain the current window. If there is a tubular contour in the current window that is outside the current region and does not belong to any image block, then the common circumscribed frame of the tubular contours in the current region and the tubular contour in the current window that is outside the current region and does not belong to any image block is taken as the current region. If not, the current region remains unchanged;

[0018] A current window is obtained by taking the lower right corner vertex of the current region as the lower right corner of the window and a preset length as the side length; if there is a tubule contour in the current window that is outside the current region and has not been determined to belong to an image block, a common circumscribed frame of the tubules in the current region and the tubule contour in the current window but outside the current region and not determined to belong to an image block is taken as the current region; if not, the current region remains unchanged;

[0019] A current window is obtained by taking the lower right corner vertex of the current region as the lower right corner of the window and a preset length as the side length; the region of the current window corresponds to an image block; all the tubule contours in the current window that have not been determined to belong to an image block belong to the image block.

[0020] Based on the further improvement of the above method, the affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs in the image block, comprising:

[0021] Based on the local matching point pairs in the tubule contour of the image block, affine transformation calculation is performed to obtain an affine transformation matrix M2 and the number N2 of successfully registered point pairs;

[0022] Based on the affine transformation matrix M2, the region corresponding to the image block in the reference fluorescence channel is determined;

[0023] Based on the image block and the region corresponding to the image block in the reference fluorescence channel, the affine transformation matrix corresponding to the image block is obtained.

[0024] Based on the further improvement of the above method, the affine transformation matrix corresponding to the image block is obtained based on the image block and the region corresponding to the image block in the reference fluorescence channel, comprising:

[0025] Matching is performed on the image block and the region corresponding to the image block in the reference fluorescence channel to obtain matching point pairs C2; affine transformation calculation is performed on the matching point pairs C2 to obtain an affine transformation matrix M3 and the number N3 of successfully registered point pairs;

[0026] If the number of matching point pairs located in the tubule contour in the matching point pairs C2 exceeds a third threshold value, then

[0027] Affine transformation calculation is performed on the matching point pairs located in the tubule contour in the matching point pairs C2 to obtain an affine transformation matrix M4 and the number N4 of successfully registered point pairs;

[0028] If the number N4 of successfully registered point pairs exceeds a fourth threshold value, then the affine transformation matrix M4 is taken as the effective affine transformation matrix corresponding to the image block; otherwise, if the number N3 of successfully registered point pairs exceeds the fourth threshold value, then the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the image block, otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block;

[0029] If the number of matched point pairs in the matched point pair C2 located in the renal tubular profile does not exceed the third threshold value, if the number of successfully registered point pairs N3 exceeds the fourth threshold value, the affine transformation matrix M3 is taken as the valid affine transformation matrix corresponding to the image block, otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block.

[0030] Based on the further improvement of the above method, the staining image and the reference fluorescence channel are matched in multiple stages to obtain the local matched point pairs of each pathological strip in the staining image, including:

[0031] The staining image and the reference fluorescence channel are globally matched to obtain global matched point pairs and a global affine transformation matrix;

[0032] For each available pathological strip in the staining image, the available pathological strip and the reference fluorescence channel are matched based on the global matched point pairs and the global affine transformation matrix to obtain the local matched point pairs of the available pathological strip.

[0033] Based on the further improvement of the above method, the staining image and the reference fluorescence channel are globally matched to obtain global matched point pairs and a global affine transformation matrix, including:

[0034] The reference fluorescence channel is morphologically processed;

[0035] The processed reference fluorescence channel is subjected to multi-angle rotation transformation to obtain multiple rotation images;

[0036] The staining image is matched with the processed reference fluorescence channel and the rotation images respectively, and the matched point pairs of the image pair with the highest matching degree are taken as the global matched point pairs;

[0037] An affine transformation matrix between the global matched point pairs is calculated to obtain the global affine transformation matrix.

[0038] Based on the further improvement of the above method, the available pathological strip and the reference fluorescence channel are matched based on the global matched point pairs and the global affine transformation matrix to obtain the local matched point pairs of the available pathological strip, including:

[0039] The global matched point pairs in the available pathological strip are subjected to affine transformation calculation to obtain an affine transformation matrix M1 and a number of successfully registered point pairs N1;

[0040] If the number of successfully registered point pairs N1 exceeds the second threshold value, the obtained affine transformation matrix M1 is taken as the affine transformation matrix corresponding to the available pathological strip, otherwise, the global affine transformation matrix is taken as the affine transformation matrix corresponding to the pathological strip;

[0041] The available pathological strip is slid in the slide window, and the corresponding region of the image in the reference fluorescent channel in each slide window is determined based on the affine transformation matrix corresponding to the pathological strip.

[0042] The image in each slide window and the corresponding region of the image in the reference fluorescent channel are matched to obtain the local matching point pair of the available pathological strip.

[0043] Based on the further improvement of the above method, the corresponding region of the image in the reference fluorescent channel in each slide window is determined based on the affine transformation matrix corresponding to the pathological strip in the following manner:

[0044] The vertex coordinates of the image in the slide window are subjected to affine transformation based on the affine transformation matrix corresponding to the pathological strip to obtain the corresponding points of the vertex in the reference fluorescent channel.

[0045] The region formed by the circumscribed frame of the corresponding points of the vertex in the reference fluorescent channel is the corresponding region of the image in the reference fluorescent channel in each slide window.

[0046] Based on the further improvement of the above method, after global matching of the stained image and the reference fluorescent channel to obtain the global matching point pair and the global affine transformation matrix, the stained image and the reference fluorescent channel are matched based on the global matching point pair and the global affine transformation matrix, and the method further comprises the following steps:

[0047] The distance transformation is performed on each pathological strip in the stained image to calculate the distance of each pixel point in the pathological strip to the boundary of the pathological strip, and the pixel points with a distance less than a first threshold value are removed.

[0048] Compared with the prior art, the present application firstly performs multi-level registration on the stained image and the reference fluorescent channel to obtain the corresponding local matching point pair in the pathological strip, thereby realizing fine-grained registration, and then obtains the affine transformation matrix corresponding to the renal tubular contour based on the local matching point pair of the pathological strip and the distribution of the renal tubular contour, thereby realizing more precise registration, and the corresponding region of the renal tubular contour in the multi-fluorescent image of the stained image can be obtained based on the effective affine transformation matrix corresponding to the renal tubular contour, thereby realizing the registration of the stained image and the multi-fluorescent image at the tubule level, and the type of the renal tubular segment in the renal tissue section can be obtained based on the intensity of the corresponding region of the renal tubular contour in different classification fluorescent channels in the multi-fluorescent image, thereby realizing efficient and accurate classification of the renal tubular segment in the renal tissue section, and providing technical support for clinical pathological analysis and basic medical research.

[0049] The technical solutions in the present application can be combined with each other to realize more preferred combination solutions. Other features and advantages of the present application will be described in the following description, and some advantages will become apparent from the description, or will be understood by those skilled in the art through implementation of the present application. The objects and other advantages of the present application can be realized and obtained by the content particularly indicated in the description and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of the detailed description. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application. In the drawings:

[0051] Figure 1 A flow chart of a tubular segment classification method for a pathological section staining image according to an embodiment of the present application;

[0052] Figure 2 A PAS staining image according to an embodiment of the present application;

[0053] Figure 3 A multiple fluorescence image according to an embodiment of the present application;

[0054] Figure 4 A local enlarged view of a staining image and a multiple staining image according to an embodiment of the present application;

[0055] Figure 5 A process schematic diagram of determining an image block to which a tubular contour belongs according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this application and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0057] One specific embodiment of the present application discloses a tubular segment classification method for a pathological section staining image, as shown in Figure 1 The method comprises the following steps:

[0058] S1, acquiring a staining image and a multiple fluorescence image of a pathological section, wherein the multiple fluorescence image comprises a reference fluorescence channel and a plurality of classification fluorescence channels;

[0059] S2, performing multi-level matching on the staining image and the reference fluorescence channel to obtain a local matching point pair of each available pathological bar in the staining image;

[0060] S3, for each available pathological bar, determining an affine transformation matrix corresponding to each tubular contour in the available pathological bar based on the local matching point pair of the available pathological bar and the distribution of the tubular contours;

[0061] S4, for each tubular outline, if the affine transformation matrix corresponding to the tubular outline is valid, obtaining a region corresponding to the tubular outline in the multiple fluorescence image based on the valid affine transformation matrix corresponding to the tubular outline; and obtaining the type of the tubular outline based on the intensity of the region corresponding to the tubular outline in the multiple fluorescence image under different classification fluorescence channels.

[0062] In implementation, the staining image is a PAS staining image, and the staining image is as shown in FIG. 2. Figure 2 The multiple fluorescence image is as shown in FIG. 3, which includes a reference fluorescence channel for matching and multiple classification fluorescence channels for classification, and each channel displays a different color. Figure 3 The partial enlarged view of the multiple fluorescence image and the staining image is as shown in FIG. 4. Figure 4 Figure 4 (a) is a partial enlarged view of the multiple fluorescence image, and (b) is a partial enlarged view of the staining image.

[0063] The pathological strip, that is, the colored region in the staining and multiple fluorescence images, Figure 2 The outer frame of one of the pathological strips is shown in FIG. 5.

[0064] It should be noted that the tubular outline, that is, the outline of the tubular drawn on the staining image, can be drawn by an existing tubular segmentation model. The tubular outline can be different segments, for example, the outline of the distal tubule or the outline of the thick ascending limb of the loop of Henle. Since the PAS staining can show the brush border, but the cytoplasmic eosinophilia difference between the distal tubule and the proximal tubule is easily affected by the staining batch or the tissue fixation quality under the conventional light microscope, the labeling personnel is easy to confuse the distal and proximal tubules due to the color difference, and it is difficult to distinguish the distal tubule and the proximal tubule, and it is more difficult to finely distinguish each segment.

[0065] The fluorescent staining agent can label each segment of the tubular, for example, the Opal620 fluorescent agent can label the proximal tubule, the Opal570 fluorescent agent can label the collecting duct, and the Opal690 can label the thick ascending limb of the loop of Henle. Therefore, in implementation, the multiple fluorescence of the present application can include Opal520 fluorescent staining, Opal620 fluorescent staining, Opal570 fluorescent staining, and Opal690 fluorescent staining. The staining image of each fluorescent staining is one classification fluorescence channel in the multiple staining image.

[0066] ​Compared with the prior art, the kidney tubule segment classification method of the pathological section staining image provided by the embodiment realizes fine-grained registration by first performing multi-level registration on the staining image and the reference fluorescence channel to obtain corresponding local matching point pairs in the pathological strip, realizes more fine registration by obtaining the affine transformation matrix corresponding to the kidney tubule contour based on the local matching point pairs of the pathological strip and the distribution of the kidney tubule contour, and obtains the region corresponding to the kidney tubule contour of the staining image in the multiple fluorescence images based on the effective affine transformation matrix corresponding to the kidney tubule contour, so as to realize the registration of the staining image and the multiple fluorescence images at the tubule level, and obtain the type of the kidney tubule contour based on the intensity of the region corresponding to the kidney tubule contour in different classification fluorescence channels in the multiple fluorescence images, so as to realize efficient and accurate classification of the kidney tubule segment in the kidney tissue section, realize accurate image recognition of the kidney tubule segment, and provide technical support for clinical pathological analysis and basic medical research.

[0067] In implementation, since DAPI fluorescence staining and Opal 480 fluorescence staining can display cell nuclei and epithelial cells, which are helpful for registration, the multiple fluorescence of the application further includes DAPI fluorescence and Opal 480 fluorescence, and the superimposed image of the DAPI fluorescence channel and the Opal 480 fluorescence channel is used as the reference fluorescence channel for matching.

[0068] Specifically, multi-level matching is performed on the staining image and the reference fluorescence channel to obtain local matching point pairs of each pathological strip in the staining image, including:

[0069] S21, global matching is performed on the staining image and the reference fluorescence channel to obtain global matching point pairs and a global affine transformation matrix;

[0070] S22, for each available pathological strip in the staining image, the available pathological strip is matched with the reference fluorescence channel based on the global matching point pairs and the global affine transformation matrix to obtain the local matching point pairs of the available pathological strip.

[0071] Specifically, global matching is performed on the staining image and the reference fluorescence channel to obtain global matching point pairs and a global affine transformation matrix, including the following steps:

[0072] S211, morphological processing is performed on the reference fluorescence channel;

[0073] S212, multi-angle rotation transformation is performed on the processed reference fluorescence channel to obtain multiple rotation images;

[0074] S213, the staining image is matched with the processed reference fluorescence channel and the rotation images respectively, and the matching point pairs of the image pair with the highest matching degree are taken as the global matching point pairs;

[0075] S214, calculate the affine transformation matrix between the global matching point pairs, and obtain a global affine transformation matrix.

[0076] In implementation, the reference fluorescence channel is first subjected to morphological processing to remove noise interference.

[0077] In implementation, the morphological processing includes background removal, erosion and dilation. For example, after the pixel intensity values in the reference fluorescence channel are sorted from large to small, the top 30% of the pixel intensity values are taken as valid signals, and the others are considered as background, and the corresponding pixel intensity values are set as background intensity values, for example, 0, thereby removing part of the noise interference. The reference fluorescence channel after the background removal is first eroded and then dilated; then the area of each pathological bar is reduced by ten times, and the maximum value is taken as a threshold value; the pathological bars with an area smaller than the threshold value are removed, and the significant large area is retained, thereby further excluding the noise and small fragment area interference, and the erosion and dilation can adopt the existing erosion and dilation methods in imageology.

[0078] In implementation, the processed reference fluorescence channel is subjected to multi-angle rotation transformation to obtain multiple rotation images, for example, the processed reference fluorescence channel is rotated clockwise by 90 degrees and counterclockwise by 90 degrees to obtain the corresponding rotation images.

[0079] In implementation, the existing image matching algorithm is adopted to match the staining image with the processed reference fluorescence channel and to match the staining image with each rotation image. For example, the loftr matching algorithm is adopted for image matching to find matching point pairs in the image pair, and the number of the matching point pairs represents the matching degree, and the more the number, the higher the matching degree. The matching point pairs of the image pair with the highest matching degree are selected as the global matching point pairs.

[0080] It should be noted that if the matching degree of the staining image with the image rotated clockwise by 90 degrees of the reference fluorescence channel is the highest, the coordinates of the matching points in the image rotated clockwise by 90 degrees need to be converted to the coordinates of the original image. The affine transformation matrix between the global matching point pairs is calculated to obtain the global affine transformation matrix.

[0081] The global matching is a rough matching of the global image, and the global matching point pairs and the global affine transformation matrix based on the global matching are further subjected to fine-grained matching.

[0082] In implementation, due to the edge effect, when the renal tubule is located at the edge of the complete pathological bar, the staining is uneven or impurities interfere, which will affect the color development, for example, both opal620 and opal520 channels in the multiple fluorescence image are positive. Therefore, after the global matching of the staining image and the reference fluorescence channel obtains the global matching point pairs and the global affine transformation matrix, for each available pathological bar in the staining image, before matching the available pathological bar with the reference fluorescence channel based on the global matching point pairs and the global affine transformation matrix, the following further includes:

[0083] For each pathology strip in the staining image, distance transformation is performed to calculate the distance from each pixel point in the pathology strip to the boundary of the pathology strip; pixel points with a distance less than a first threshold value are removed.

[0084] In implementation, the first threshold value can be determined according to the mpp (Microns Per Pixel, the number of microns corresponding to each pixel, used to describe the resolution of the digital pathology image, that is, the size of each pixel in the actual physical size, usually in microns) value and the scaling ratio of the image, for example, if the staining image has an mpp value, the first threshold value is 2000*(mpp / 0.46)*image scaling ratio; if the staining image has no mpp value, the first threshold value is 2000*image scaling ratio.

[0085] The present application removes the small vessels close to the edge based on distance transformation after global matching and before fine-grained matching, thereby preventing interference and improving the accuracy of classification.

[0086] Fine-grained matching is first performed in the pathology strip. Therefore, it is necessary to determine which pathology strips are available. For each pathology strip in the staining image, if the pathology strip has a global matching point, the pathology strip is an available pathology strip.

[0087] In implementation, due to the displacement and deformation of the pathology strip, the registration is offset. Therefore, the present application solves the registration offset by using the strip matching mode.

[0088] Specifically, based on the global matching point pair and the global affine transformation matrix, the available pathology strip is matched with the reference fluorescence channel to obtain the local matching point pair of the available pathology strip, including:

[0089] S221, affine transformation calculation is performed on the global matching point pair in the available pathology strip to obtain an affine transformation matrix M1 and the number N1 of successfully registered point pairs;

[0090] S222, if the number N1 of successfully registered point pairs exceeds a second threshold value, the affine transformation matrix M1 is taken as the affine transformation matrix corresponding to the available pathology strip, otherwise, the global affine transformation matrix is taken as the affine transformation matrix corresponding to the pathology strip;

[0091] S223, a sliding window is used to slide in the area of the available pathology strip, and the corresponding area of the image in the reference fluorescence channel image in each sliding window is determined based on the affine transformation matrix corresponding to the pathology strip.

[0092] S224, the image in each sliding window and the corresponding area of the image in the reference fluorescence channel in the sliding window are matched to obtain the local matching point pair of the available pathology strip.

[0093] In implementation, for an available pathology strip, affine transformation calculation is first performed on the global matching point pairs in the available pathology strip to obtain a corresponding affine transformation matrix M1 and a number N1 of successfully matched point pairs.

[0094] In implementation, the affine transformation calculation can use the estimateAffine2D function of opencv, which returns the number of successfully matched point pairs.

[0095] If the number N1 of successfully matched point pairs exceeds a second threshold, the affine transformation matrix M1 calculated by the global matching point pairs in the available pathology strip is taken as the affine transformation matrix corresponding to the available pathology strip, so that subsequent matching is more accurate. If the number N1 of successfully matched point pairs does not exceed the second threshold, the global affine transformation matrix is taken as the affine transformation matrix corresponding to the pathology strip. In implementation, the second threshold is set according to the matching accuracy requirement.

[0096] In implementation, a non-overlapping sliding window is used to slide in each available pathology strip region, and the corresponding region of the image in the reference fluorescence channel in the sliding window is determined based on the affine transformation matrix corresponding to the pathology strip.

[0097] Specifically, the corresponding region of the image in the reference fluorescence channel in each sliding window is determined in the following manner:

[0098] S2231, affine transformation is performed on the vertex coordinates of the image in the sliding window based on the affine transformation matrix corresponding to the pathology strip to obtain the corresponding points of the vertices in the reference fluorescence channel;

[0099] S2232, the region constituted by the bounding box of the corresponding points of the vertices in the reference fluorescence channel is the corresponding region of the image in the reference fluorescence channel in the sliding window.

[0100] In implementation, the coordinates of the corresponding points of the left top, right top, right bottom and left bottom four vertices of the image in the sliding window are calculated based on the affine transformation matrix corresponding to the pathology strip. The region constituted by the bounding box of these corresponding points is the corresponding region of the image in the reference fluorescence channel in the sliding window.

[0101] After obtaining the corresponding region of the image in the reference fluorescence channel in the sliding window, the image in the sliding window and its corresponding region in the reference fluorescence channel are matched (for example, the loftr matching algorithm is used for image matching) to obtain the matching point pairs corresponding to the image in the sliding window. The matching point pairs corresponding to all the images in the available pathology strip in the sliding window constitute the local matching point pairs of the available pathology strip.

[0102] Through the sliding window, fine-grained matching is performed on the available pathology strip, and the accuracy of the registration is improved.

[0103] After obtaining the local matching point pairs of each available pathological strip, an affine transformation matrix corresponding to each tubular outline in the available pathological strip is determined based on the local matching point pairs and the distribution of the tubular outlines, specifically comprising:

[0104] S31, determining the image block to which each tubular outline in the available pathological strip belongs;

[0105] S32, for each image block, obtaining an affine transformation matrix corresponding to the image block based on the local matching point pairs within the image block;

[0106] S32, the affine transformation matrix of each tubular outline of the image block is the affine transformation matrix corresponding to the image block.

[0107] Specifically, the image block to which each tubular outline in the available pathological strip belongs is determined in the following manner:

[0108] For each tubular outline in the available pathological strip that does not belong to any image block, taking the circumscribed frame of the tubular outline as the current region;

[0109] Taking the top-left corner of the current region as the top-left corner of the window and a preset length as the side length to obtain the current window; if there is a tubular outline in the current window that is outside the current region and does not belong to any image block, taking the common circumscribed frame of the tubular outlines in the current region and the tubular outlines in the current window but outside the current region and not belonging to any image block as the current region; if not, the current region remains unchanged;

[0110] Taking the top-right corner of the current region as the top-right corner of the window and a preset length as the side length to obtain the current window; if there is a tubular outline in the current window that is outside the current region and does not belong to any image block, taking the common circumscribed frame of the tubular outlines in the current region and the tubular outlines in the current window but outside the current region and not belonging to any image block as the current region; if not, the current region remains unchanged;

[0111] Taking the bottom-right corner of the current region as the bottom-right corner of the window and a preset length as the side length to obtain the current window; if there is a tubular outline in the current window that is outside the current region and does not belong to any image block, taking the common circumscribed frame of the tubular outlines in the current region and the tubular outlines in the current window but outside the current region and not belonging to any image block as the current region; if not, the current region remains unchanged;

[0112] Taking the bottom-left corner of the current region as the bottom-left corner of the window and a preset length as the side length to obtain the current window; the region of the current window is taken as an image block; all tubular outlines in the current window that do not belong to any image block belong to the image block.

[0113] Take Figure 5 as an example for illustration. Figure 5 In (a) of FIG. 1, the circle is a tubule contour which does not determine the corresponding image block, and the solid line frame is the circumscribed edge of the tubule contour. Take the circumscribed edge of one of the tubule contours which does not determine the corresponding image block as the current region, i.e., the solid line frame in the figure is the current region, and take the top-left corner of the current region as the top-left corner of the window, and take the preset length as the side length to draw the current window, i.e., the dashed line frame in (a) of FIG. 1. Figure 5 In (a) of FIG. 1, the dashed line frame in the current window exists a tubule contour which is not in the current region (solid line frame) and does not determine the corresponding image block, so the common circumscribed edge of the tubule contours of the two tubules is taken as the current region, for example, the solid line frame in (b) of FIG. 1. Figure 5 In (a) of FIG. 1, the dashed line frame in the current window does not exist a tubule contour which is not in the current region and does not determine the corresponding image block, so the current region is not changed. Take the bottom-right corner of the current region as the bottom-right corner of the window, and take the preset length as the side length to draw the current window, for example, the dashed line frame in (c) of FIG. 1.

[0114] Then take the top-right corner of the solid line frame (current region) in (b) of FIG. 1 as the top-right corner of the window, and take the preset length as the side length to draw the current window, i.e., the dashed line frame in (b) of FIG. 1. Figure 5 In (b) of FIG. 1, the dashed line frame in the current window does not exist a tubule contour which is not in the current region and does not determine the corresponding image block, so the current region is not changed. Take the bottom-right corner of the current region as the bottom-right corner of the window, and take the preset length as the side length to draw the current window, for example, the dashed line frame in (c) of FIG. 1. Figure 5 In (a) of FIG. 1, the dashed line frame in the current window still does not exist a tubule contour which is not in the current region and does not determine the corresponding image block, so the current region is continued to be taken as the left-bottom corner of the window, and the preset length is taken as the side length to draw the current window, for example, the dashed line frame in (d) of FIG. 1. Figure 5 In (a) of FIG. 1, the dashed line frame in the current window still does not exist a tubule contour which is not in the current region and does not determine the corresponding image block, so the current region is continued to be taken as the left-bottom corner of the window, and the preset length is taken as the side length to draw the current window, for example, the dashed line frame in (d) of FIG. 1. Figure 5 In (a) of FIG. 1, the dashed line frame in the current window still does not exist a tubule contour which is not in the current region and does not determine the corresponding image block, so the current region is continued to be taken as the left-bottom corner of the window, and the preset length is taken as the side length to draw the current window, for example, the dashed line frame in (d) of FIG. 1. In (a) of FIG. 1, the dashed line frame in the current window still does not exist a tubule contour which is not in the current region and does not determine the corresponding image block, so the current region is continued to be taken as the left-bottom corner of the window, and the preset length is taken as the side length to draw the current window, for example, the dashed line frame in (d) of FIG. 1.

[0115] The above process of determining the image block to which the tubule contour belongs is not to uniformly divide the pathological strip into image blocks of equal size, but to determine the image block according to the density of the tubule contour. The size of the determined image block is different, and the image block corresponding to the region where the tubule contour is dense is larger than the image block corresponding to the region where the tubule contour is sparse. The region where the tubule contour is dense is taken as an image block, which reduces the number of image blocks while not reducing the registration quality. The region without tubule contour does not need to determine the image block, so the number of determined image blocks is less than that obtained by uniform division, thereby improving the calculation efficiency. After determining the image block to which each tubule contour in the available pathological strip belongs, for each image block, an affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs in the image block, and then the region corresponding to the image block in the reference fluorescence channel is determined based on the obtained affine transformation matrix.

[0116] Specifically, the affine transformation matrix corresponding to each image block is obtained based on the local matching point pairs in the image block, including:

[0117] S321, affine transformation calculation is performed based on the local matching point pairs in the tubular outline of the image block to obtain an affine transformation matrix M2 and a number N2 of successfully registered point pairs;

[0118] S322, a corresponding region of the image block in the reference fluorescence channel is determined based on the affine transformation matrix M2;

[0119] S323, a corresponding affine transformation matrix of the image block is obtained based on the image block and the corresponding region of the image block in the reference fluorescence channel.

[0120] In implementation, for an image block, affine transformation calculation is first performed based on the local matching point pairs located in the tubular outline in the local matching point pairs in the image block, and the affine transformation calculation can use the aforementioned estimateAffine2D function to obtain an affine transformation matrix M2 and a number N2 of successfully registered point pairs.

[0121] It should be noted that if the number of the local matching point pairs located in the tubular outline of the image block is less than a preset threshold, for example, less than 50 pairs, the point pairs closest to the tubular from the local matching points located in the image block but not in the tubular outline are selected as the local matching point pairs located in the tubular outline of the image block to make up the number of the local matching point pairs located in the tubular outline of the image block, so as to facilitate more accurate matching.

[0122] Then, a corresponding region of the image block in the reference fluorescence channel is determined based on the affine transformation matrix M2.

[0123] In implementation, the affine transformation matrix M2 is used to perform affine transformation on the vertex coordinates of the image block to obtain corresponding points of the vertices in the reference fluorescence channel; and a region formed by the circumscribed frame of the corresponding points of the image block vertices in the reference fluorescence channel is the corresponding region of the image block in the reference fluorescence channel. For details, refer to steps S2231-S2232.

[0124] After that, a corresponding affine transformation matrix of the image block is obtained based on the image block and the corresponding region of the image block in the reference fluorescence channel, which specifically includes:

[0125] S3231, matching is performed on the image block and the corresponding region of the image block in the reference fluorescence channel to obtain matching point pairs C2; affine transformation calculation is performed on the matching point pairs C2 to obtain an affine transformation matrix M3 and a number N3 of successfully registered point pairs;

[0126] S3232, if the number of the matching point pairs located in the tubular outline in the matching point pairs C2 exceeds a third threshold (for example, 40 pairs), then

[0127] affine transformation matrix M4 and the number N4 of successfully registered point pairs are calculated, and then it is determined whether the number N4 of successfully registered point pairs exceeds a fourth threshold value (for example, 20 pairs). If the number N4 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M4 is taken as the effective affine transformation matrix corresponding to the image block. If the number N4 of successfully registered point pairs does not exceed the fourth threshold value, it is determined whether the number N3 of successfully registered point pairs exceeds the fourth threshold value. If the number N3 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the image block. If the number N3 of successfully registered point pairs does not exceed the fourth threshold value, the affine transformation matrix M3 is taken as the invalid affine transformation matrix corresponding to the image block.

[0128] If the number N4 of successfully registered point pairs exceeds the fourth threshold value (for example, 20 pairs), the affine transformation matrix M4 is taken as the effective affine transformation matrix corresponding to the image block. If the number N3 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the image block. If the number N3 of successfully registered point pairs does not exceed the fourth threshold value, the affine transformation matrix M3 is taken as the invalid affine transformation matrix corresponding to the image block.

[0129] If the number of matching point pairs located within the renal tubular contour in the matching point pair C2 does not exceed the third threshold value (40 pairs), it is determined whether the number N3 of successfully registered point pairs exceeds the fourth threshold value. If the number N3 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the image block. If the number N3 of successfully registered point pairs does not exceed the fourth threshold value, the affine transformation matrix M3 is taken as the invalid affine transformation matrix corresponding to the image block.

[0130] In implementation, the matching algorithm of step S3231 adopts the loftr algorithm described above to obtain the matching point pair C2, and the estimateAffine2D function described above is used to perform affine transformation calculation on the matching point pair C2 to obtain the affine transformation matrix M3 and the number N3 of successfully registered point pairs.

[0131] If the number of matching point pairs located within the renal tubular contour in the matching point pair C2 exceeds the third threshold value (for example, 40 pairs), affine transformation calculation is performed on the matching point pairs located within the renal tubular contour in the matching point pair C2 to obtain the affine transformation matrix M4 and the number N4 of successfully registered point pairs. Then, it is determined whether the number N4 of successfully registered point pairs exceeds the fourth threshold value (for example, 20 pairs). If the number N4 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M4 is taken as the affine transformation matrix corresponding to the image block, which can achieve more accurate matching. Therefore, the affine transformation matrix M4 is taken as the affine transformation matrix corresponding to the image block. If it is determined that the number N4 of successfully registered point pairs does not exceed the fourth threshold value, it is determined whether the number N3 of successfully registered point pairs exceeds the fourth threshold value. If the number N3 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M3 is taken as the affine transformation matrix corresponding to the image block. If the number N3 of successfully registered point pairs does not exceed the fourth threshold value, the affine transformation matrix M3 is taken as the invalid affine transformation matrix corresponding to the image block.

[0132] If the number of matching point pairs located within the renal tubular contour in the matching point pair C2 does not exceed the third threshold value, it is determined whether the number N3 of successfully registered point pairs exceeds the fourth threshold value. If the number N3 of successfully registered point pairs exceeds the fourth threshold value, the affine transformation matrix M3 is taken as the affine transformation matrix corresponding to the image block. If the number N3 of successfully registered point pairs does not exceed the fourth threshold value, the affine transformation matrix M3 is taken as the invalid affine transformation matrix corresponding to the image block.

[0133] When the number of successfully registered point pairs N3 is small, the calculation will have a large error. Therefore, M3 is the invalid affine transformation matrix corresponding to the image patch. Only when the image patch to which the renal tubule contour belongs has a corresponding valid affine transformation matrix will the corresponding region of the renal tubule contour be searched in the multifluorescence image to improve the accuracy of registration and classification.

[0134] During implementation, the third and fourth thresholds are set according to the matching accuracy requirements.

[0135] After obtaining the affine transformation matrix of the image patch, the affine transformation matrix of each renal tubule contour within the image patch is the affine transformation matrix corresponding to that image patch. Subsequent determinations of the corresponding regions in the fluorescence image for each renal tubule contour within the image patch are based on the effective affine transformation matrix of that image patch. By determining the image patch corresponding to the tubule contour and calculating the affine transformation matrix for that patch, it is unnecessary to calculate the affine transformation matrix for each individual renal tubule contour, thus improving computational efficiency.

[0136] After obtaining the affine transformation matrix corresponding to each renal tubule contour, if the corresponding affine transformation matrix is ​​valid, the region corresponding to that renal tubule contour in the multifluorescence image is obtained based on the corresponding valid affine transformation matrix. In practice, the coordinates of the four vertices of the outer bounding box of the renal tubule contour are affinely transformed according to its valid affine transformation matrix to obtain the points corresponding to the vertices in the multifluorescence image; the region formed by the outer bounding box of the points corresponding to the vertices in the multifluorescence image is the region corresponding to that renal tubule contour in the multifluorescence image. For details, refer to steps S2231 to S2232.

[0137] Calculate the average intensity value of the region corresponding to the renal tubule contour in each categorical fluorescence channel in the multiplex fluorescence image. The channel with the highest average intensity value corresponds to the type of the renal tubule contour. For example, if the Opal690 channel has the highest average intensity, then the renal tubule contour is the thick ascending limb of the loop of Henle.

[0138] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of classifying tubular segments of a pathology slide stained image, the method comprising: The method comprises the following steps: ​ Obtaining a staining image and a multi-fluorescence image of a pathological section, wherein the multi-fluorescence image comprises a reference fluorescence channel and a plurality of classification fluorescence channels; Performing multi-level matching on the staining image and the reference fluorescence channel to obtain a local matching point pair of each available pathological strip in the staining image; For each available pathological strip, determining an affine transformation matrix corresponding to each renal tubular contour in the available pathological strip based on the local matching point pair of the available pathological strip and the distribution of the renal tubular contour, comprising: Determining an image block to which each renal tubular contour in the available pathological strip belongs; For each image block, obtaining an affine transformation matrix corresponding to the image block based on the local matching point pair in the image block, comprising: Performing affine transformation calculation based on the local matching point pair in the renal tubular contour of the image block to obtain an affine transformation matrix M2 and a number N2 of successfully registered point pairs; Determining a corresponding region of the image block in the reference fluorescence channel based on the affine transformation matrix M2; Obtaining an affine transformation matrix corresponding to the image block based on the image block and the corresponding region of the image block in the reference fluorescence channel, comprising: Performing matching on the image block and the corresponding region of the image block in the reference fluorescence channel to obtain a matching point pair C2; performing affine transformation calculation on the matching point pair C2 to obtain an affine transformation matrix M3 and a number N3 of successfully registered point pairs; If the number of matching point pairs in the matching point pair C2 located in the renal tubular contour exceeds a third threshold value, then Performing affine transformation calculation on the matching point pair in the matching point pair C2 located in the renal tubular contour to obtain an affine transformation matrix M4 and a number N4 of successfully registered point pairs; If the number N4 of successfully registered point pairs exceeds a fourth threshold value, then the affine transformation matrix M4 is taken as the effective affine transformation matrix corresponding to the image block; otherwise, if the number N3 of successfully registered point pairs exceeds the fourth threshold value, then the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the image block, otherwise, the affine transformation matrix M3 is an invalid affine transformation matrix corresponding to the image block; If the number of matching point pairs in the matching point pair C2 located in the renal tubular contour does not exceed the third threshold value, then: if the number N3 of successfully registered point pairs exceeds the fourth threshold value, then the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the image block, otherwise, the affine transformation matrix M3 is an invalid affine transformation matrix corresponding to the image block; The affine transformation matrix of each renal tubular contour of the image block is the affine transformation matrix corresponding to the image block; For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubular contour is valid, then based on the effective affine transformation matrix corresponding to the renal tubular contour, a region corresponding to the renal tubular contour in the multi-fluorescence image is obtained, and the type of the renal tubular contour is obtained based on the intensity of the region corresponding to the renal tubular contour in different classification fluorescence channels of the multi-fluorescence image.

2. The method of tubule segment classification of a pathology slide stain image of claim 1, wherein, The image block to which each renal tubular contour in the available pathological strip belongs is determined in the following manner: For each renal tubular contour in the available pathological strip without a determined belonging image block, taking the circumscribed frame of the renal tubular contour as a current region; If there is a tubular outline in the current window which is outside the current region and has not been determined to belong to an image block, then the common circumscribed edge frame of the tubular outlines in the current region and the tubular outlines in the current window which are outside the current region and have not been determined to belong to an image block is taken as the current region; If not, the current region remains unchanged; If there is a tubular outline in the current window which is outside the current region and has not been determined to belong to an image block, then the common circumscribed edge frame of the tubular outlines in the current region and the tubular outlines in the current window which are outside the current region and have not been determined to belong to an image block is taken as the current region; If not, the current region remains unchanged; If there is a tubular outline in the current window which is outside the current region and has not been determined to belong to an image block, then the common circumscribed edge frame of the tubular outlines in the current region and the tubular outlines in the current window which are outside the current region and have not been determined to belong to an image block is taken as the current region; If not, the current region remains unchanged; If there is a tubular outline in the current window which is outside the current region and has not been determined to belong to an image block, then the common circumscribed edge frame of the tubular outlines in the current region and the tubular outlines in the current window which are outside the current region and have not been determined to belong to an image block is taken as the current region; If not, the current region remains unchanged; 3. The method of tubule segment classification of a pathology slide stain image of claim 1, wherein, If there is a tubular outline in the current window which is outside the current region and has not been determined to belong to an image block, then the common circumscribed edge frame of the tubular outlines in the current region and the tubular outlines in the current window which are outside the current region and have not been determined to belong to an image block is taken as the current region; If not, the current region remains unchanged. Multi-level matching is performed on the stained image and the reference fluorescent channel to obtain local matching point pairs of each pathological strip in the stained image, including:

4. The method of tubule segment classification of a pathology slide stain image of claim 1, wherein, Global matching is performed on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix; For each available pathological strip in the stained image, matching is performed on the available pathological strip and the reference fluorescent channel based on the global matching point pairs and the global affine transformation matrix to obtain local matching point pairs of the available pathological strip. Global matching is performed on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix, including: Morphological processing is performed on the reference fluorescent channel; Multi-angle rotation transformation is performed on the processed reference fluorescent channel to obtain multiple rotated images; 5. The method of tubule segment classification of a pathology slide stained image of claim 3, wherein, Matching is performed on the stained image and the processed reference fluorescent channel and the rotated images respectively, and the matching point pairs of the image pair with the highest matching degree are taken as the global matching point pairs; An affine transformation matrix between the global matching point pairs is calculated to obtain the global affine transformation matrix. Matching is performed on the available pathological strip and the reference fluorescent channel based on the global matching point pairs and the global affine transformation matrix to obtain local matching point pairs of the available pathological strip, including: Affine transformation calculation is performed on the global matching point pairs in the available pathological strip to obtain an affine transformation matrix M1 and a number N1 of successfully registered point pairs; If the number N1 of successfully registered point pairs exceeds a second threshold, the affine transformation matrix M1 is taken as the affine transformation matrix corresponding to the available pathological strip, otherwise, the global affine transformation matrix is taken as the affine transformation matrix corresponding to the pathological strip; The window is slid in the available pathological strip region, and a region corresponding to the image in the reference fluorescent channel in each window is determined based on an affine transformation matrix corresponding to the pathological strip; Matching the image in each window and the region corresponding to the image in the reference fluorescent channel in the window, to obtain local matching point pairs of the available pathological strip.

6. The method of tubule segment classification of a pathology slide stain image of claim 5, wherein, The region corresponding to the image in the reference fluorescent channel in each window is determined based on the affine transformation matrix corresponding to the pathological strip in the following manner: The vertex coordinates of the image in the window are subjected to affine transformation based on the affine transformation matrix corresponding to the pathological strip to obtain points corresponding to the vertex in the reference fluorescent channel; The region constituted by the circumscribed frame of the points corresponding to the vertex in the reference fluorescent channel is the region corresponding to the image in the reference fluorescent channel in the window.

7. The method of tubule segment classification of a pathology slide stained image of claim 3, wherein, After global matching of the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix, matching the available pathological strip and the reference fluorescent channel based on the global matching point pairs and the global affine transformation matrix further includes: Performing distance transformation on each pathological strip in the stained image to calculate the distance of each pixel point in the pathological strip to the boundary of the pathological strip; and eliminating pixel points with a distance less than a first threshold.

Images