Renal tubule segment classification method of pathological section staining image

Abstract

The invention relates to a renal tubule segment classification method of a pathological section staining image, belongs to the technical field of renal tubule classification, and solves the problem that small renal segments cannot be accurately distinguished in the prior art. The method comprises the following steps: acquiring a staining image and a multiple fluorescence image of a pathological section; performing multi-stage matching on the dyed image and the reference fluorescence channel to obtain a local matching point pair of each available pathological strip in the dyed image; for each available pathological strip, determining an affine transformation matrix corresponding to each renal tubule contour in the available pathological strip based on the local matching point pairs and the distribution of the renal tubule contours; and for each kidney tubule contour, obtaining an area corresponding to the kidney tubule contour in the multiple fluorescence image based on the corresponding effective affine transformation matrix, and obtaining the type of the kidney tubule contour based on the intensity of the area corresponding to the kidney tubule contour in different classification fluorescence channels. And efficient and accurate classification of the renal tubule segments is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of renal tubule classification, and in particular to a renal tubule segment classification method based on a stained pathological section image. Background Art

[0002] Pathology plays a crucial role in the medical field. After renal tissue is punctured or resected using a coarse needle, pathological sections are prepared. Doctors combine microscopic images with clinical information to make a pathological diagnosis and guide clinical treatment. Staining can highlight specific structures, such as the glomerular basement membrane or glycogen, helping to distinguish normal from abnormal tissue. It is a commonly used staining technique for pathological sections.

[0003] With the continuous development of artificial intelligence and deep learning in recent years, deep learning has been increasingly applied in the medical field. To train intelligent image semantic segmentation models, it is necessary to annotate various structures in renal pathological sections, including the renal tubules. The renal tubules can be divided into proximal tubules (including the proximal convoluted tubule, proximal straight tubule, and thin descending limb of the loop of Henle), distal tubules (including thin ascending limb of the loop of Henle, thick ascending limb of the loop of Henle, and distal convoluted tubule), connecting tubules, and collecting ducts. Distinguishing the various segments of the renal tubules is crucial for accurate disease diagnosis and the development of predictive models. Although staining can distinguish normal proximal and distal tubules by revealing the brush border, this characteristic brush border is easily lost in disease states. Furthermore, the difference in cytoplasmic eosinophilia between distal and proximal tubules under conventional light microscopy is easily affected by the staining batch or tissue fixation quality. This color difference can easily confuse the proximal and distal tubules. Consequently, conventional staining cannot accurately distinguish distal and proximal tubules, and finely distinguishing individual tubular segments is even more difficult. Summary of the Invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a method for classifying renal tubular segments in pathological section stained images, so as to solve the problem that the existing method cannot accurately distinguish the types of renal tubular segments.

[0005] In one aspect, an embodiment of the present invention provides a method for classifying renal tubular segments in a stained pathological section image, comprising the following steps:

[0006] Acquiring a stained image and a multiple fluorescence image of a pathological section, wherein the multiple fluorescence image includes a reference fluorescence channel and a plurality of classification fluorescence channels;

[0007] Perform multi-level matching on the stained image and the reference fluorescence channel to obtain local matching point pairs for each available pathological strip in the stained image;

[0008] For each available pathology strip, determining an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the local matching point pairs of the available pathology strip and the distribution of the renal tubule contours;

[0009] For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubular contour is valid, the area corresponding to the renal tubular contour in the multiple fluorescence images is obtained based on the valid affine transformation matrix corresponding to the renal tubular contour, and the type of the renal tubular contour is obtained based on the intensities of different classified fluorescence channels of the area corresponding to the renal tubular contour in the multiple fluorescence images.

[0010] Based on a further improvement of the above method, the affine transformation matrix corresponding to each renal tubule contour in the available pathology strip is determined based on the local matching point pairs of the available pathology strip, including:

[0011] determining the image block to which each renal tubule outline in the available pathology strip belongs;

[0012] For each image block, the affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs within the image block;

[0013] The affine transformation matrix of each renal tubule contour of the image block is the affine transformation matrix corresponding to the image block.

[0014] Based on a further improvement of the above method, the image block to which each renal tubule contour in the available pathology strip belongs is determined in the following manner:

[0015] For each renal tubule contour of the image block to which the available pathology strip has not been determined, the outer bounding box of the renal tubule contour is used as the current region;

[0016] The current window is obtained by taking the upper left corner vertex of the current region as the upper left corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, the current region remains unchanged;

[0017] The current window is obtained by taking the upper right corner vertex of the current region as the upper right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, the current region remains unchanged;

[0018] The current window is obtained by taking the lower right corner vertex of the current region as the lower right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is used as the current region; if there is no such border, the current region remains unchanged;

[0019] The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length; the area of ​​the current window corresponds to an image block; all renal tubule contours of the image blocks that have not been determined to belong to in the current window belong to this image block.

[0020] Based on a further improvement of the above method, the affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs in the image block, including:

[0021] Perform affine transformation calculation based on the local matching point pairs within the renal tubule contour of the image block to obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2;

[0022] Determine the area corresponding to the image block in the reference fluorescence channel based on the affine transformation matrix M2;

[0023] An affine transformation matrix corresponding to the image block is obtained based on the image block and a region corresponding to the image block in a reference fluorescent channel.

[0024] Based on a further improvement of the above method, an affine transformation matrix corresponding to the image block is obtained based on the image block and the region corresponding to the image block in the reference fluorescence channel, including:

[0025] Matching the image block with the corresponding region of the image block in the reference fluorescence channel to obtain a matching point pair C2; performing affine transformation calculation on the matching point pair C2 to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3;

[0026] If the number of matching point pairs C2 that are located within the renal tubule contour exceeds the third threshold, then

[0027] Perform affine transformation calculation on the matching point pairs C2 that are located within the renal tubule contour to obtain the affine transformation matrix M4 and the number of successfully registered point pairs N4;

[0028] If the number of successfully registered point pairs N4 exceeds the fourth threshold, the affine transformation matrix M4 is used as the valid affine transformation matrix corresponding to the image block; otherwise, if the number of successfully registered point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block;

[0029] If the number of matching point pairs in the matching point pairs C2 that are located within the renal tubule contour does not exceed the third threshold, then: if the number of successfully aligned point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block.

[0030] Based on the further improvement of the above method, multi-level matching is performed on the stained image and the reference fluorescence channel to obtain the local matching point pairs of each pathological strip in the stained image, including:

[0031] Performing global matching on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix;

[0032] For each available pathology strip in the stained image, the available pathology strip is matched with the reference fluorescence channel based on the global matching point pair and the global affine transformation matrix to obtain a local matching point pair of the available pathology strip.

[0033] Based on a further improvement of the above method, the stained image and the reference fluorescent channel are globally matched to obtain global matching point pairs and a global affine transformation matrix, including:

[0034] Perform morphological processing on the baseline fluorescence channel;

[0035] Perform multi-angle rotation transformation on the processed reference fluorescence channel to obtain multiple rotation images;

[0036] Match the stained image with the processed reference fluorescence channel and the rotation image respectively, and take the matching point pair with the highest matching degree as the global matching point pair;

[0037] Calculate the affine transformation matrix between the global matching point pairs to obtain the global affine transformation matrix.

[0038] Based on a further improvement of the above method, the available pathology strip is matched with the reference fluorescence channel based on the global matching point pair and the global affine transformation matrix to obtain the local matching point pair of the available pathology strip, including:

[0039] Perform affine transformation calculation on the global matching point pairs within the available pathology strip to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0040] If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 is obtained as the affine transformation matrix corresponding to the available pathology strip; otherwise, the global affine transformation matrix is ​​used as the affine transformation matrix corresponding to the pathology strip;

[0041] A sliding window is used to slide in the available pathology strip area, and a corresponding area of ​​the image in each sliding window in the reference fluorescence channel is determined based on an affine transformation matrix corresponding to the pathology strip;

[0042] Matching is performed between each image within the sliding window and a corresponding region of the image within the sliding window in the reference fluorescence channel to obtain a local matching point pair of the available pathology strip.

[0043] Based on the further improvement of the above method, the corresponding area of ​​the image in each sliding window in the reference fluorescence channel is determined based on the affine transformation matrix corresponding to the pathology strip in the following way:

[0044] Based on the affine transformation matrix corresponding to the pathology strip, the vertex coordinates of the image in the sliding window are affine transformed to obtain the corresponding points of the vertices in the reference fluorescence channel;

[0045] The area formed by the outer frame of the points corresponding to the vertices in the reference fluorescence channel is the area corresponding to the image in the sliding window in the reference fluorescence channel.

[0046] Based on a further improvement of the above method, after globally matching the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix, matching the available pathology strip with the reference fluorescent channel based on the global matching point pairs and the global affine transformation matrix further includes:

[0047] A distance transformation is performed on each pathology strip in the stained image to calculate the distance between each pixel in the pathology strip and the boundary of the pathology strip; and pixels whose distance is less than a first threshold are eliminated.

[0048] Compared with the existing technology, the present invention first performs multi-level registration on the stained image and the reference fluorescence channel to obtain the corresponding local matching point pairs in the pathology strip, thereby achieving fine-grained registration, and then obtains the affine transformation matrix corresponding to the renal tubular contour based on the local matching point pairs of the pathology strip and the distribution of the renal tubular contour, thereby achieving more refined registration. Based on the effective affine transformation matrix corresponding to the renal tubular contour, the corresponding area of ​​the renal tubular contour of the stained image in the multiple fluorescence images can be obtained, thereby achieving registration of the stained image and the multiple fluorescence images at the tubular level. Based on the intensity of the corresponding area of ​​the renal tubular contour in the multiple fluorescence images in different classification fluorescence channels, the type of the renal tubular contour can be obtained, thereby achieving efficient and accurate classification of renal tubular segments in renal tissue sections, providing technical support for clinical pathological analysis and basic medical research.

[0049] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0051] Figure 1 Flowchart of a method for classifying renal tubular segments in a stained pathological section image according to an embodiment of the present invention;

[0052] Figure 2 This is a PAS staining image according to an embodiment of the present invention;

[0053] Figure 3 is a multiple fluorescence image according to an embodiment of the present invention;

[0054] Figure 4 It is a partial magnified view of the stained image and the multi-stained image according to the embodiment of the present invention;

[0055] Figure 5 Schematic diagram of the process of determining the image block to which the renal tubule contour belongs according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0057] A specific embodiment of the present invention discloses a method for classifying renal tubular segments in pathological section stained images, such as Figure 1 As shown, the following steps are included:

[0058] S1. Acquire a stained image and a multiple fluorescence image of a pathological section, wherein the multiple fluorescence image includes a reference fluorescence channel and multiple classification fluorescence channels;

[0059] S2, performing multi-level matching on the stained image and the reference fluorescence channel to obtain a local matching point pair for each available pathological strip in the stained image;

[0060] S3. For each available pathology strip, determine an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the local matching point pairs of the available pathology strip and the distribution of the renal tubule contours;

[0061] S4. For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubule is valid, then the area corresponding to the renal tubular contour in the multiple fluorescence images is obtained based on the valid affine transformation matrix corresponding to the renal tubular contour; and the type of the renal tubular contour is obtained based on the intensity of the area corresponding to the renal tubular contour in the multiple fluorescence images under different classification fluorescence channels.

[0062] During implementation, the dyed image is a PAS dyed image, and the dyed image is as follows Figure 2 Multiple fluorescence images are shown in Figure 3 As shown in the figure, it includes a reference fluorescence channel for matching and multiple classification fluorescence channels for classification, each channel displays a different color. The local enlarged image of the multiple fluorescence image and the stained image is shown in the figure. Figure 4 As shown, Figure 4 (a) is a local magnification of the multiple fluorescence image, and (b) is a local magnification of the staining image.

[0063] Pathology strips are the colored areas in staining and multiple fluorescence images. Figure 2 The bounding box of one of the pathology bars is shown in .

[0064] It should be noted that the renal tubule outline, i.e., the outline of the renal tubule drawn on the stained image, can be drawn using an existing renal tubule segmentation model. The renal tubule outline may be of different segments, for example, it may be the outline of the distal tubule or the outline of the thick ascending limb of the loop of Henle. Although PAS staining can show the brush border, the difference in cytoplasmic eosinophilia between the distal and proximal tubules is easily affected by the staining batch or tissue fixation quality under conventional light microscopy. Labelers may easily confuse the distal and proximal tubules due to color differences, making it difficult to distinguish between the distal and proximal tubules, and even more difficult to refine the individual segments.

[0065] Fluorescent dyes can mark various segments of the renal tubules. For example, Opal 620 fluorescent dye can mark the proximal tubules, Opal 570 fluorescent dye can mark the collecting ducts, and Opal 690 can mark the thick ascending limb of the loop of Henle. Therefore, in practice, the multiple fluorescent dyes of the present invention can include, for example, Opal 520 fluorescent dye, Opal 620 fluorescent dye, Opal 570 fluorescent dye, and Opal 690 fluorescent dye. The dyed image of each fluorescent dye is a classified fluorescent channel in the multiple dyed image.

[0066] Compared with the existing technology, the method for classifying renal tubular segments in pathological section stained images provided in this embodiment first performs multi-level registration on the stained image and the reference fluorescence channel to obtain the corresponding local matching point pairs in the pathological strip, thereby achieving fine-grained registration, and then obtains the affine transformation matrix corresponding to the renal tubular contour based on the local matching point pairs of the pathological strip and the distribution of the renal tubular contour, thereby achieving more precise registration. Based on the effective affine transformation matrix corresponding to the renal tubular contour, the corresponding area of ​​the renal tubular contour of the stained image in the multiple fluorescence images can be obtained, thereby achieving registration of the stained image and the multiple fluorescence images at the tubular level. Based on the intensity of the corresponding area of ​​the renal tubular contour in the multiple fluorescence images in different classification fluorescence channels, the type of the renal tubular contour can be obtained, thereby achieving efficient and accurate classification of renal tubular segments in renal tissue sections, and accurate image recognition of renal tubular segments, providing technical support for clinical pathological analysis and basic medical research.

[0067] During implementation, since DAPI fluorescence staining and Opal 480 fluorescence staining can display cell nuclei and epithelial cells, which is helpful for alignment, the multiple fluorescence of the present invention also includes DAPI fluorescence and Opal480 fluorescence, and the superimposed image of the DAPI fluorescence channel and the Opal480 fluorescence channel is used as the reference fluorescence channel for matching.

[0068] Specifically, multi-level matching is performed on the stained image and the reference fluorescence channel to obtain local matching point pairs for each pathological strip in the stained image, including:

[0069] S21, performing global matching on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix;

[0070] S22 . For each available pathology strip in the stained image, match the available pathology strip with the reference fluorescence channel based on the global matching point pair and the global affine transformation matrix to obtain a local matching point pair of the available pathology strip.

[0071] Specifically, performing global matching on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix includes the following steps:

[0072] S211, performing morphological processing on the reference fluorescence channel;

[0073] S212, performing multi-angle rotation transformation on the processed reference fluorescence channel to obtain multiple rotation images;

[0074] S213, matching the stained image with the processed reference fluorescence channel and the rotation image respectively, and taking the matching point pair of the image pair with the highest matching degree as the global matching point pair;

[0075] S214. Calculate the affine transformation matrix between the global matching point pairs to obtain a global affine transformation matrix.

[0076] During implementation, morphological processing is first performed on the reference fluorescence channel to remove the interference of noise.

[0077] During implementation, morphological processing includes background removal, erosion, and dilation. For example, after sorting the pixel intensities in the baseline fluorescence channel from largest to smallest, the top 30% of pixel intensities are considered valid signals, while the remainder are considered background. The corresponding pixel intensities are set to the background intensity value, such as 0, thereby removing some noise interference. The baseline fluorescence channel, from which the background has been removed, is first eroded and then dilated. The area of ​​each pathology strip is then reduced tenfold, and the maximum value is taken as the threshold. Pathology strips with an area smaller than the threshold are removed, retaining significant large areas, thereby further eliminating interference from noise and small fragmented areas. Erosion and dilation can use existing erosion and dilation methods in image science.

[0078] During implementation, the processed reference fluorescent channel is subjected to multi-angle rotation transformation to obtain multiple rotation images. For example, the processed reference fluorescent channel is rotated 90 degrees clockwise and 90 degrees counterclockwise to obtain corresponding rotation images.

[0079] During implementation, existing image matching algorithms are used to match the stained image with the processed reference fluorescence channel and with each rotated image. For example, the loftr matching algorithm is used to find matching point pairs in the image pair. The number of matching point pairs represents the degree of matching; a greater number indicates a higher degree of matching. The matching point pair with the highest degree of matching is selected as the global matching point pair.

[0080] It should be noted that if the stained image has the highest match with the image of the reference fluorescence channel rotated 90 degrees clockwise, the coordinates of the matching points in the 90-degree clockwise rotated image need to be converted to the coordinates of the original image. The affine transformation matrix between the global matching point pairs is calculated to obtain the global affine transformation matrix.

[0081] Global matching is a rough matching of the global image, and fine-grained matching is further performed based on the global matching point pairs and global affine transformation matrix obtained by global matching.

[0082] During implementation, due to the existence of edge effects, that is, when the renal tubules are located at the edge of a complete pathology strip, uneven staining or interference from impurities will affect the color development, for example, in a multiple fluorescence image, both the opal620 and opal520 channels are positive. Therefore, after globally matching the stained image and the reference fluorescence channel to obtain global matching point pairs and a global affine transformation matrix, the present invention further includes the following steps before matching each available pathology strip in the stained image with the reference fluorescence channel based on the global matching point pairs and the global affine transformation matrix:

[0083] A distance transformation is performed on each pathology strip in the stained image to calculate the distance between each pixel in the pathology strip and the boundary of the pathology strip; and pixels whose distance is less than a first threshold are eliminated.

[0084] During implementation, the first threshold can be determined based on the mpp (Microns Per Pixel, the number of microns corresponding to each pixel, used to describe the resolution of digital pathology images, that is, the size of each pixel in actual physical size, usually in microns) value of the image and the scaling ratio. For example, if the stained image has an mpp value, the first threshold is 2000*(mpp / 0.46)*image scaling ratio; if the stained image does not have an mpp value, the first threshold is 2000*image scaling ratio.

[0085] The present invention eliminates small tubes close to the edge based on distance transformation after global matching and before fine-grained matching, thereby preventing interference and improving classification accuracy.

[0086] Fine-grained matching is first performed within the pathology strip. Therefore, it is necessary to first determine which pathology strips are available. For each pathology strip in the stained image, if there is a global matching point for that pathology strip, then that pathology strip is considered available.

[0087] During implementation, the displacement and deformation of the pathology strips may cause registration deviation. Therefore, the present invention adopts a stripe matching method to solve the registration deviation.

[0088] Specifically, matching the available pathology strip with the reference fluorescence channel based on the global matching point pair and the global affine transformation matrix to obtain the local matching point pair of the available pathology strip includes:

[0089] S221, performing affine transformation calculation on the global matching point pairs in the available pathology strip to obtain an affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0090] S222. If the number of successfully registered point pairs N1 exceeds a second threshold, the affine transformation matrix M1 is obtained as the affine transformation matrix corresponding to the available pathology strip; otherwise, the global affine transformation matrix is ​​used as the affine transformation matrix corresponding to the pathology strip;

[0091] S223, sliding a sliding window in the available pathology strip area, and determining the area corresponding to the image in each sliding window in the reference fluorescent channel image based on the affine transformation matrix corresponding to the pathology strip;

[0092] S224 : Match each image within the sliding window with the corresponding region of the image within the sliding window in the reference fluorescence channel to obtain a local matching point pair of the available pathology strip.

[0093] During implementation, for an available pathology strip, firstly, affine transformation calculation is performed on the global matching point pairs in the available pathology strip to obtain the corresponding affine transformation matrix M1 and the number of successfully registered point pairs N1.

[0094] During implementation, the affine transformation calculation can be performed using OpenCV's estimateAffine2D function, which returns the number of successfully aligned point pairs.

[0095] If the number of successfully registered point pairs N1 exceeds a second threshold, the affine transformation matrix M1 calculated from the global matching point pairs within the available pathology strip is used as the affine transformation matrix corresponding to the available pathology strip, making subsequent matching more accurate. If the number of successfully registered point pairs N1 does not exceed the second threshold, the global affine transformation matrix is ​​used as the affine transformation matrix corresponding to the pathology strip. During implementation, the second threshold is set based on the matching accuracy requirements.

[0096] During implementation, a non-overlapping sliding window is used to slide in each available pathology strip area, and the area corresponding to the image in the sliding window in the reference fluorescence channel is determined based on the affine transformation matrix corresponding to the pathology strip.

[0097] Specifically, the area corresponding to the image in each sliding window in the reference fluorescence channel is determined in the following manner:

[0098] S2231, performing affine transformation on the vertex coordinates of the image in the sliding window based on the affine transformation matrix corresponding to the pathology strip to obtain the points corresponding to the vertices in the reference fluorescence channel;

[0099] S2232: The area formed by the outer bounding box of the point corresponding to the vertex in the reference fluorescence channel is the area corresponding to the image in the sliding window in the reference fluorescence channel.

[0100] During implementation, the coordinates of the four vertices of the image in the sliding window (top left, top right, bottom right, and bottom left) are calculated based on the affine transformation matrix corresponding to the pathology line. The area formed by the bounding box of these corresponding points is the area corresponding to the image in the sliding window in the reference fluorescence channel.

[0101] The area corresponding to the image in the sliding window in the reference fluorescence channel is obtained, and the image in the sliding window and the area corresponding to the image in the reference fluorescence channel are matched (for example, using the loftr matching algorithm for image matching) to obtain matching point pairs corresponding to the image in the sliding window; the matching point pairs corresponding to all the images in the sliding windows of the available pathology strip constitute the local matching point pairs of the available pathology strip.

[0102] Fine-grained matching of available pathology strips is performed through a sliding window to improve the accuracy of registration.

[0103] After obtaining the local matching point pairs of each available pathology strip, the affine transformation matrix corresponding to each renal tubular contour in the available pathology strip is determined based on the local matching point pairs of the available pathology strip and the distribution of the renal tubular contours, specifically including:

[0104] S31, determining the image block to which each renal tubule contour in the available pathology strip belongs;

[0105] S32. For each image block, obtain an affine transformation matrix corresponding to the image block based on local matching point pairs within the image block;

[0106] S32. The affine transformation matrix of each renal tubule contour in the image block is the affine transformation matrix corresponding to the image block.

[0107] Specifically, the image block to which each renal tubule contour in the available pathology strip belongs is determined in the following manner:

[0108] For each renal tubule contour of the image block to which the available pathology strip has not been determined, the outer bounding box of the renal tubule contour is used as the current region;

[0109] The current window is obtained by taking the upper left corner vertex of the current region as the upper left corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, then the current region remains unchanged;

[0110] The current window is obtained by taking the upper right corner vertex of the current region as the upper right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, then the current region remains unchanged;

[0111] The current window is obtained by taking the lower right corner vertex of the current region as the lower right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, then the current region remains unchanged;

[0112] The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length; the area of ​​the current window is regarded as an image block; all the renal tubule contours of the image blocks that have not been determined to belong to in the current window belong to this image block.

[0113] by Figure 5 Take this as an example to illustrate. Figure 5 In (a), the circle is the outline of the renal tubule of the image block to which it is not determined, and the solid line box is the outer bounding box of the renal tubule outline. First, the outer bounding box of the renal tubule outline of one of the undetermined corresponding image blocks is used as the current area, that is, the solid line box in the figure is the current area, the upper left corner vertex of the current area is used as the upper left corner of the window, and the preset length is used as the side length to obtain the current window, that is, Figure 5 In the dotted box in (a), there are renal tubule contours in the current window that are not in the current area (solid box) and have no determined image block to which they belong. The common external frame of the renal tubule contours of these two renal tubules is used as the current area, for example Figure 5 The solid line box in (b).

[0114] Then with Figure 5 The upper right corner of the solid line frame (current area) in (b) is used as the upper right corner vertex of the window, and the preset length is the side length to draw the window to get the current window, that is, Figure 5 The dotted box in (b) of the current window. If there is no renal tubule outline that is not in the current area and whose image block is not determined, the current area remains unchanged. The current window is obtained by taking the lower right corner vertex of the current area as the lower right corner of the window and the preset length as the side length, as shown in the following example: Figure 5 The dotted box in (c) of the current window still has no renal tubule contour that is not in the current area and whose image block is not determined. The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length, as shown in the following example: Figure 5 In the dotted box of (d), the area of ​​the current window is regarded as an image block; all the renal tubule contours in the current window that have not been determined to belong to the image block belong to this image block.

[0115] The above-mentioned process for determining the image blocks to which the renal tubular contours belong does not uniformly divide the pathology strip into blocks of equal size. Instead, the blocks are determined based on the density of the renal tubules. The resulting blocks are larger in areas with dense renal tubular contours than in areas with sparser renal tubular contours. Areas with dense renal tubular contours are treated as one image block, reducing the number of image blocks without compromising registration quality. Areas without renal tubular contours do not require image blocks, resulting in fewer blocks than would be obtained by uniform division, thereby improving computational efficiency. After determining the image block to which each renal tubular contour in the available pathology strip belongs, an affine transformation matrix corresponding to each image block is derived based on the local matching point pairs within the image block. Based on this affine transformation matrix, the region corresponding to the renal tubular contour belonging to that image block in the reference fluorescence channel is then determined.

[0116] Specifically, the affine transformation matrix corresponding to each image block is obtained based on the local matching point pairs within the image block, including:

[0117] S321, performing affine transformation calculation on the local matching point pairs within the renal tubule contour of the image block to obtain an affine transformation matrix M2 and the number of successfully registered point pairs N2;

[0118] S322, determining the area corresponding to the image block in the reference fluorescence channel based on the affine transformation matrix M2;

[0119] S323 : Obtain an affine transformation matrix corresponding to the image block based on the image block and the region corresponding to the image block in the reference fluorescent channel.

[0120] During implementation, for an image block, an affine transformation is first calculated based on the local matching point pairs located in the renal tubule contour among the local matching point pairs in the image block. The affine transformation calculation can use the aforementioned estimateAffine2D function to obtain the affine transformation matrix M2 and the number of successfully aligned point pairs N2.

[0121] It should be noted that if the number of local matching point pairs located within the renal tubular contour of the image block is less than a preset threshold, for example, less than 50 pairs, then the point pairs closest to the renal tubules are screened from the local matching points located in the image block but not within the renal tubular contour as the local matching point pairs located within the renal tubular contour of the image block to supplement the number of local matching point pairs located within the renal tubular contour of the image block, thereby facilitating more accurate matching.

[0122] Then, the region corresponding to the image block in the reference fluorescence channel is determined based on the affine transformation matrix M2.

[0123] During implementation, an affine transformation is performed on the vertex coordinates of the image block based on the affine transformation matrix M2 to obtain the points corresponding to the vertices in the reference fluorescence channel. The area formed by the bounding box of the points corresponding to the vertices of the image block in the reference fluorescence channel is the area corresponding to the image block in the reference fluorescence channel. For details, refer to steps S2231 and S2232.

[0124] Then, an affine transformation matrix corresponding to the image block is obtained based on the image block and the region corresponding to the image block in the reference fluorescent channel, specifically including:

[0125] S3231, matching the image block with the corresponding region of the image block in the reference fluorescence channel to obtain a matching point pair C2; performing affine transformation calculation on the matching point pair C2 to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3;

[0126] S3232: If the number of matching point pairs C2 located within the renal tubule contour exceeds a third threshold (e.g., 40 pairs), then

[0127] Perform affine transformation calculation on the matching point pairs C2 that are located within the renal tubule contour to obtain the affine transformation matrix M4 and the number of successfully registered point pairs N4;

[0128] If the number of successfully registered point pairs N4 exceeds a fourth threshold (e.g., 20 pairs), the affine transformation matrix M4 is used as the valid affine transformation matrix corresponding to the image block; otherwise, if the number of successfully registered point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is an invalid affine transformation matrix corresponding to the image block;

[0129] If the number of matching point pairs in the matching point pairs C2 that are located within the contour of the renal tubule does not exceed the third threshold (40 pairs), then: if the number of successfully aligned point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block.

[0130] During implementation, the matching algorithm of step S3231 adopts the above-mentioned loftr algorithm to obtain the matching point pair C2, and uses the above-mentioned estimateAffine2D function to perform affine transformation calculation on the matching point pair C2 to obtain the affine transformation matrix M3 and the number of successfully aligned point pairs N3.

[0131] If the number of matching point pairs in the matching point pairs C2 that are located within the renal tubule contour exceeds a third threshold value (e.g., 40 pairs), an affine transformation calculation is performed on the matching point pairs in the matching point pairs C2 that are located within the renal tubule contour to obtain an affine transformation matrix M4 and the number of successfully registered point pairs N4. It is then determined whether the number of successfully registered point pairs N4 exceeds a fourth threshold value (e.g., 20 pairs). If it exceeds the fourth threshold value, the affine transformation matrix M4 is used as the affine transformation matrix corresponding to the image block, which can achieve more accurate matching. Therefore, the affine transformation matrix M4 is used as the affine transformation matrix corresponding to the image block. If it is successfully determined that the number of successfully registered point pairs N4 does not exceed the fourth threshold value, it is determined whether the number of successfully registered point pairs N3 exceeds the fourth threshold value. If it does, the affine transformation matrix M3 is used as the affine transformation matrix corresponding to the image block. If N3 also does not exceed the fourth threshold value, the affine transformation matrix M3 is used as the invalid affine transformation matrix corresponding to the image block.

[0132] If the number of matching point pairs in the matching point pairs C2 that are located within the contour of the renal tubule does not exceed the third threshold, then check whether the number of successfully aligned point pairs N3 exceeds the fourth threshold. If it exceeds, the affine transformation matrix M3 is used as the affine transformation matrix corresponding to the image block. If N3 also does not exceed the fourth threshold, the affine transformation matrix M3 is used as the invalid affine transformation matrix corresponding to the image block.

[0133] If the number of successfully registered point pairs N3 is small, the calculation will have a large error. Therefore, M3 is the invalid affine transformation matrix corresponding to the image block. Only when the image block to which the renal tubule contour belongs has a corresponding valid affine transformation matrix, the corresponding region of the renal tubule contour in the multi-fluorescence image is searched to improve the accuracy of registration and classification.

[0134] During implementation, the third threshold and the fourth threshold are set according to the matching accuracy requirement.

[0135] After obtaining the affine transformation matrix for the image block, the affine transformation matrix for each renal tubule contour in the image block becomes the affine transformation matrix corresponding to that image block. Subsequently, the corresponding region in the fluorescence image for each renal tubule contour in that image block is determined based on the effective affine transformation matrix for that image block. By determining the image block corresponding to the tubule contour and calculating the corresponding affine transformation matrix for that image block, the affine transformation matrix does not need to be calculated for each renal tubule contour individually, improving computational efficiency.

[0136] After obtaining the affine transformation matrix corresponding to each renal tubule contour, if the corresponding affine transformation matrix is ​​valid, the region corresponding to the renal tubule contour in the multiple fluorescence images is determined based on the corresponding valid affine transformation matrix. In implementation, the coordinates of the four vertices of the bounding box of the renal tubule contour are affine transformed according to the valid affine transformation matrix to obtain the points corresponding to the vertices in the multiple fluorescence images. The region formed by the bounding box of the points corresponding to the vertices in the multiple fluorescence images is the region corresponding to the renal tubule contour in the multiple fluorescence images. For details, refer to steps S2231-S2232.

[0137] Calculate the average intensity of the region corresponding to the renal tubule outline in the multiple fluorescence images for each classification channel. The channel with the highest average intensity value corresponds to the type of renal tubule outline. For example, if the Opal690 channel has the highest average intensity, the renal tubule outline is the thick ascending limb of the loop of Henle.

[0138] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0139] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for classifying renal tubular segments in pathological section stained images, characterized in that: The following steps are involved: Acquiring a stained image and a multiple fluorescence image of a pathological section, wherein the multiple fluorescence image includes a reference fluorescence channel and a plurality of classification fluorescence channels; Perform multi-level matching on the stained image and the reference fluorescence channel to obtain local matching point pairs for each available pathological strip in the stained image; For each available pathology strip, determining an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the local matching point pairs of the available pathology strip and the distribution of the renal tubule contours; For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubular contour is valid, the area corresponding to the renal tubular contour in the multiple fluorescence images is obtained based on the valid affine transformation matrix corresponding to the renal tubular contour, and the type of the renal tubular contour is obtained based on the intensities of different classified fluorescence channels of the area corresponding to the renal tubular contour in the multiple fluorescence images.

2. The method for classifying renal tubular segments of pathological section stained images according to claim 1, characterized in that: Determining an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the local matching point pairs of the available pathology strip includes: determining the image block to which each renal tubule outline in the available pathology strip belongs; For each image block, the affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs within the image block; The affine transformation matrix of each renal tubule contour of the image block is the affine transformation matrix corresponding to the image block.

3. The method for classifying renal tubular segments of pathological section stained images according to claim 2, characterized in that: The image block to which each renal tubule outline in the available pathology strip belongs is determined in the following manner: For each renal tubule contour of the image block to which the available pathology strip has not been determined, the outer bounding box of the renal tubule contour is used as the current region; The current window is obtained by taking the upper left corner vertex of the current region as the upper left corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed frame of the renal tubule in the current region and the renal tubule outline in the current window but outside the current region and the image block to which it belongs is used as the current region; If not, the current region remains unchanged; The current window is obtained by taking the upper right corner vertex of the current region as the upper right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed frame of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is used as the current region; If not, the current region remains unchanged; The current window is obtained by taking the lower right corner vertex of the current region as the lower right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, the common circumscribed frame of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is used as the current region; If not, the current region remains unchanged; The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length; the area of ​​the current window corresponds to an image block; All renal tubule contours in the current window whose image blocks have not been determined to belong to belong to this image block.

4. The method for classifying renal tubular segments of pathological section stained images according to claim 2, characterized in that: The affine transformation matrix corresponding to the image block is obtained based on the local matching point pairs within the image block, including: Perform affine transformation calculation based on the local matching point pairs within the renal tubule contour of the image block to obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2; Determine the area corresponding to the image block in the reference fluorescence channel based on the affine transformation matrix M2; An affine transformation matrix corresponding to the image block is obtained based on the image block and a region corresponding to the image block in a reference fluorescent channel.

5. The method for classifying renal tubular segments of pathological section stained images according to claim 4, characterized in that: Obtaining an affine transformation matrix corresponding to the image block based on the image block and a region corresponding to the image block in a reference fluorescent channel includes: Matching the image block with the corresponding region of the image block in the reference fluorescence channel to obtain a matching point pair C2; performing affine transformation calculation on the matching point pair C2 to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3; If the number of matching point pairs C2 that are located within the renal tubule contour exceeds the third threshold, then Perform affine transformation calculation on the matching point pairs C2 that are located within the renal tubule contour to obtain the affine transformation matrix M4 and the number of successfully registered point pairs N4; If the number of successfully registered point pairs N4 exceeds the fourth threshold, the affine transformation matrix M4 is used as the valid affine transformation matrix corresponding to the image block; otherwise, if the number of successfully registered point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block; If the number of matching point pairs in the matching point pairs C2 that are located within the renal tubule contour does not exceed the third threshold, then: if the number of successfully aligned point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block.

6. The method for classifying renal tubular segments of pathological section stained images according to claim 1, characterized in that: Perform multi-level matching on the stained image and the reference fluorescence channel to obtain local matching point pairs for each pathological strip in the stained image, including: Performing global matching on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix; For each available pathology strip in the stained image, the available pathology strip is matched with the reference fluorescence channel based on the global matching point pair and the global affine transformation matrix to obtain a local matching point pair of the available pathology strip.

7. The method for classifying renal tubular segments in pathological section stained images according to claim 1, characterized in that: Performing global matching on the stained image and the reference fluorescent channel to obtain global matching point pairs and a global affine transformation matrix includes: Perform morphological processing on the baseline fluorescence channel; Perform multi-angle rotation transformation on the processed reference fluorescence channel to obtain multiple rotation images; Match the stained image with the processed reference fluorescence channel and the rotation image respectively, and take the matching point pair with the highest matching degree as the global matching point pair; Calculate the affine transformation matrix between the global matching point pairs to obtain the global affine transformation matrix.

8. The method for classifying renal tubule segments in pathological section stained images according to claim 6, characterized in that: Matching the available pathology strip with the reference fluorescence channel based on the global matching point pair and the global affine transformation matrix to obtain a local matching point pair of the available pathology strip, including: Perform affine transformation calculation on the global matching point pairs within the available pathology strip to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1; If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 is obtained as the affine transformation matrix corresponding to the available pathology strip; otherwise, the global affine transformation matrix is ​​used as the affine transformation matrix corresponding to the pathology strip; A sliding window is used to slide in the available pathology strip area, and a corresponding area of ​​the image in each sliding window in the reference fluorescence channel is determined based on an affine transformation matrix corresponding to the pathology strip; Matching is performed between each image within the sliding window and a corresponding region of the image within the sliding window in the reference fluorescence channel to obtain a local matching point pair of the available pathology strip.

9. The method for classifying renal tubule segments of pathological section stained images according to claim 8, characterized in that: Based on the affine transformation matrix corresponding to the pathology strip, the corresponding area of ​​the image in each sliding window in the reference fluorescence channel is determined in the following manner: Based on the affine transformation matrix corresponding to the pathology strip, the vertex coordinates of the image in the sliding window are affine transformed to obtain the corresponding points of the vertices in the reference fluorescence channel; The area formed by the outer frame of the points corresponding to the vertices in the reference fluorescence channel is the area corresponding to the image in the sliding window in the reference fluorescence channel.

10. The method for classifying renal tubule segments in pathological section stained images according to claim 6, characterized in that: After globally matching the stained image and the reference fluorescent channel to obtain a global matching point pair and a global affine transformation matrix, matching the available pathology strip with the reference fluorescent channel based on the global matching point pair and the global affine transformation matrix further includes: A distance transformation is performed on each pathology strip in the stained image to calculate the distance between each pixel in the pathology strip and the boundary of the pathology strip; and pixels whose distance is less than a first threshold are eliminated.

Images