Training method of far-end and near-end renal tubule recognition model

Abstract

The invention relates to a training method of a far-end and near-end renal tubule recognition model, belongs to the technical field of renal tubule recognition, and solves the problem that a far-end renal tubule and a near-end renal tubule cannot be accurately recognized in the prior art. The method comprises the following steps: acquiring PAS staining images and multiple fluorescence images of pathological sections of different individuals; identifying the kidney tubule contour in the PAS dyeing image based on a semantic segmentation model; identifying whether the kidney tubule profile in the PAS staining image is a proximal kidney tubule or a distal kidney tubule based on a match of the PAS staining image and a multiple fluorescence image; constructing a training sample set based on the PAS staining images marked with the near-end renal tubules and the far-end renal tubules; and constructing a multi-type segmentation model, and training the multi-type segmentation model based on the training sample set to obtain a trained far and near renal tubule recognition model. The far-end renal tubule and the near-end renal tubule can be quickly and accurately identified.

Description

Technical Field

[0001] The present invention relates to the technical field of renal tubule recognition, and in particular to a training method for a distal and proximal renal tubule recognition model. Background Art

[0002] The renal tubule is one of the core units of kidney structure and function, consisting of the proximal tubule, distal tubule, and collecting duct (the proximal and distal parts can be further divided into the curved and straight parts). The proximal tubule is located at the beginning of the renal tubule and is directly connected to the renal capsule. Its tubular surface is covered with dense microvilli structures. It is mainly responsible for the active reabsorption of approximately 70% of water, electrolytes (such as sodium and potassium), and nutrients such as glucose and amino acids in the primary urine, and is also involved in the regulation of acid-base balance. The distal renal tubule includes the thick ascending limb of the loop of Henle and the distal convoluted tubule. The latter is located behind the thick ascending limb of the loop of Henle and connects to the collecting duct. Its epithelial cells have fewer microvilli but a high mitochondrial content. It is mainly responsible for the fine regulation of electrolytes (such as sodium ion reabsorption and potassium ion secretion) and the final regulation of acid-base balance, and is directly affected by hormones such as aldosterone.

[0003] Damage to different renal tubular segments is closely associated with specific kidney diseases. For example, early manifestations of acute kidney injury (AKI) are often characterized by proximal tubular epithelial cell necrosis or brush border loss, while distal tubular injury is significantly associated with the progression of chronic kidney disease (CKD) and electrolyte disorders (such as hypokalemia). In addition, the differential diagnosis of hereditary diseases such as Bartter syndrome (abnormal sodium and potassium metabolism in the distal tubules) and Fanconi syndrome (widespread proximal tubular dysfunction) is highly dependent on the accurate identification of renal tubular segments.

[0004] Existing deep learning neural network models are often based on the experience of pathologists to identify and label relatively normal renal tubules on PAS-stained images. This is because, while PAS staining can distinguish normal proximal and distal tubules by displaying a brush border, this characteristic is easily lost in pathological conditions, making it difficult to distinguish between distal and proximal tubules. Consequently, existing models cannot accurately identify distal and proximal tubules, especially in pathological conditions. Summary of the Invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a training method for a distal and proximal renal tubule identification model to solve the problem that the distal and proximal renal tubules cannot be accurately identified in the prior art.

[0006] In one aspect, an embodiment of the present invention provides a method for training a distal-proximal renal tubule recognition model, comprising the following steps:

[0007] Acquiring PAS-stained images and multiple fluorescence images of pathological sections of different individuals; the multiple fluorescence images include a first fluorescence channel for marking distal renal tubules and a second fluorescence channel for marking proximal renal tubules;

[0008] Identify the outline of renal tubules in PAS-stained images based on semantic segmentation model;

[0009] Identifying whether the renal tubule outline in the PAS-stained image is a proximal renal tubule or a distal renal tubule based on the matching of the PAS-stained image and the multiple fluorescence images; and constructing a training sample set based on the PAS-stained image in which the proximal renal tubule and the distal renal tubule are identified;

[0010] A multi-type segmentation model is constructed, and the multi-type segmentation model is trained based on the training sample set to obtain a trained proximal and distal renal tubule recognition model.

[0011] Based on the further improvement of the above scheme, the constructed multi-type segmentation model includes:

[0012] The encoder is used to extract semantic features of the input image layer by layer through multiple downsampling layers;

[0013] The middle layer is used to compress the semantic features extracted by the last downsampling layer of the encoder and pass the compressed semantic features to the decoder;

[0014] The decoder is used to gradually decode the semantic features through multiple upsampling layers to obtain decoded features;

[0015] Output layer, used to output segmentation results based on decoded features;

[0016] The number of downsampling layers of the encoder is the same as the number of upsampling layers of the encoder and corresponds one to one; an attention module is connected between the downsampling layer and the corresponding upsampling layer, which is used to perform attention extraction on the semantic features extracted by the downsampling layer and pass them to the corresponding upsampling layer.

[0017] Based on the further improvement of the above scheme, the following formula is used to calculate the loss of the multi-type segmentation model:

[0018]

[0019] Among them, M represents the number of pixels of the sample, C represents the number of segmentation types, and N represents the number of samples in the current training batch. Indicates whether the j-th pixel of the i-th sample belongs to the k-th type, if yes, it is 1, otherwise it is 0, It represents the probability that the j-th pixel of the i-th sample belongs to the k-th type predicted by the model.

[0020] A further improvement based on the above solution, identifying whether the renal tubule outline in the PAS-stained image is a proximal renal tubule or a distal renal tubule based on matching between the PAS-stained image and the multiple fluorescence images, includes:

[0021] Performing global matching on the PAS staining image and the multiple fluorescence images to obtain global matching point pairs and a global affine transformation matrix;

[0022] For each available pathology strip in the PAS staining image, the available pathology strip area is divided into multiple image blocks, and each image block is matched with the multiple fluorescence images based on the global matching point pairs and the global affine transformation matrix to obtain local matching point pairs corresponding to each image block;

[0023] For each renal tubule contour in the available pathology strip, obtaining an affine transformation matrix corresponding to the renal tubule contour based on a local matching point pair corresponding to the image block to which the renal tubule contour belongs;

[0024] If the affine transformation matrix corresponding to the renal tubule contour is valid, the area corresponding to the renal tubule contour in the multiple fluorescence images is obtained based on the affine transformation matrix corresponding to the renal tubule contour; based on the intensity of the area corresponding to the renal tubule contour under the first fluorescence channel and the second fluorescence channel, it is identified whether the renal tubule contour is a distal renal tubule or a proximal renal tubule.

[0025] Based on a further improvement of the above solution, identifying whether the renal tubule outline is a distal tubule or a proximal tubule based on the intensity of the area corresponding to the renal tubule outline under the first fluorescent channel and the second fluorescent channel includes:

[0026] Calculating the intensity value of the area corresponding to the renal tubule outline in the first fluorescent channel image and the second fluorescent channel of the multiple fluorescent image to obtain a first intensity value and a second intensity value;

[0027] If the first intensity value exceeds a fifth threshold value and the second intensity value does not exceed a sixth threshold value, the renal tubule contour is a distal renal tubule;

[0028] If the first intensity value does not exceed the fifth threshold value, and the second intensity value exceeds the sixth threshold value, the renal tubule contour is a proximal renal tubule;

[0029] If the first intensity value exceeds the fifth threshold and the second intensity value exceeds the sixth threshold, then: if the ratio of the first intensity value to the second intensity value exceeds the seventh threshold, the renal tubule contour is a distal renal tubule; if the ratio of the second intensity value to the first intensity value exceeds the seventh threshold, the renal tubule contour is a proximal renal tubule.

[0030] Based on a further improvement of the above solution, image block matching is performed on each image block with multiple fluorescence images based on the global matching point pairs and the global affine transformation matrix to obtain local matching points corresponding to each image block, including:

[0031] For each image block, perform affine transformation on the global matching point pairs contained in the image block to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0032] If the number of successfully registered point pairs N1 exceeds a second threshold, determining the region corresponding to the image block on the multiple fluorescence images based on the affine transformation matrix M1; otherwise, determining the region corresponding to the image block on the multiple fluorescence images based on the global affine transformation matrix;

[0033] The image block is matched with a region corresponding to the image block on the multiple fluorescence images to obtain a local matching point corresponding to the image block.

[0034] Based on a further improvement of the above solution, the region corresponding to the image block on the multiple fluorescence images is determined based on the affine transformation matrix M1, including:

[0035] Performing affine transformation on the vertex coordinates of the image block based on the affine transformation matrix M1 to obtain the points corresponding to the vertices on the multiple fluorescence images;

[0036] The area formed by the outer bounding box of the points corresponding to the vertices on the multiple fluorescence images is the area corresponding to the image block on the multiple fluorescence images.

[0037] Based on a further improvement of the above solution, the affine transformation matrix corresponding to the renal tubule contour is obtained based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, including:

[0038] Perform affine transformation calculation on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, and obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2;

[0039] Performing affine transformation calculation on the local matching point pairs located within the renal tubule contour among the local matching points corresponding to the image block to which the renal tubule contour belongs, to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3;

[0040] If the number of successfully registered point pairs N3 exceeds the third threshold, the affine transformation matrix M3 is used as the effective affine transformation matrix corresponding to the renal tubule contour; otherwise:

[0041] If the number of successfully registered point pairs N2 exceeds the third threshold, the affine transformation matrix M2 is used as the valid affine transformation matrix corresponding to the renal tubule contour; otherwise, the affine transformation matrix M2 is used as the invalid affine transformation matrix corresponding to the renal tubule contour.

[0042] Based on the further improvement of the above scheme, the multiple fluorescence also includes DAPI fluorescence image and Opal480 fluorescence;

[0043] Performing global matching on the PAS staining image and the multiple fluorescence images to obtain global matching point pairs includes:

[0044] Morphological processing was performed on the superimposed images of DAPI fluorescence images and Opal480 fluorescence images;

[0045] Perform multi-angle rotation transformation on the processed superimposed image to obtain multiple rotation images;

[0046] Match the PAS-stained image with the processed overlay image and each rotation image;

[0047] The matching point pair of the image pair with the highest matching degree is taken as the global matching point pair;

[0048] Perform affine transformation calculation on the global matching point pairs to obtain the global affine transformation matrix.

[0049] Based on a further improvement of the above solution, after obtaining the global matching point pairs and the global affine transformation matrix by global matching, for each available pathology strip in the PAS-stained image, before segmenting the available pathology strip area into multiple image blocks, the following is further included:

[0050] A distance transformation is performed on each pathological strip in the PAS staining image, and the distance from each pixel in each pathological strip to the border of the pathological strip is calculated, and pixels with a distance less than a first threshold are eliminated.

[0051] Compared with the existing technology, the present invention obtains PAS-stained images and multiple fluorescence images of pathological sections of different individuals, identifies the contours of renal tubules in the PAS-stained images based on a semantic segmentation model, and identifies the proximal and distal renal tubules (including normal and diseased states) based on the matching of the PAS-stained images and the multiple fluorescence images, thereby quickly constructing a training sample set. Based on the constructed training sample set, a distal and proximal renal tubule recognition model is trained. The PAS chromosome to be identified is input into the obtained distal and proximal renal tubule recognition model, so that the distal and proximal renal tubules in normal and diseased states can be quickly and accurately identified, thereby filling a technical gap and assisting doctors in disease diagnosis.

[0052] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0054] Figure 1 Flowchart of a training method for a distal and proximal renal tubule identification model according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of a PAS staining image according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the process of determining the image block to which the renal tubule contour belongs according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0058] A specific embodiment of the present invention discloses a training method for a distal and proximal renal tubule recognition model, such as Figure 1 The following steps are shown:

[0059] S1. Acquire PAS-stained images and multiple fluorescence images of pathological sections of different individuals; the multiple fluorescence images include a first fluorescence channel for marking distal renal tubules and a second fluorescence channel for marking proximal renal tubules;

[0060] S2, identifying the outline of renal tubules in PAS-stained images based on the semantic segmentation model;

[0061] S3. Identifying whether the renal tubule outline in the PAS-stained image is a proximal renal tubule or a distal renal tubule based on the matching of the PAS-stained image and the multiple fluorescence images; and constructing a training sample set based on the PAS-stained image in which the proximal renal tubule and the distal renal tubule are identified;

[0062] S3. Construct a multi-type segmentation model, and train the multi-type segmentation model based on the training sample set to obtain a trained proximal and distal renal tubule recognition model.

[0063] Since PAS staining is shorter in both staining time and cost than fluorescent staining, PAS staining images are typically used to train neural network models. Although PAS staining can reveal the brush border, the difference in cytoplasmic eosinophilia between the distal and proximal tubules is easily affected by the staining batch or tissue fixation quality under conventional light microscopy, and labelers are easily confused by color differences. In addition, the brush border features are easily lost in pathological conditions, making it difficult to distinguish between the distal and proximal tubules. Therefore, the present invention identifies the proximal and distal tubules in PAS staining images based on matching of PAS staining images and multiple fluorescence images, thereby not only improving labeling efficiency but also reducing labeling errors.

[0064] During implementation, the first fluorescent channel of the present invention may be Opal520 fluorescence, and the second fluorescent channel may be Opal620 fluorescence.

[0065] Compared with the existing technology, the training method for the proximal and distal renal tubules recognition model provided in this embodiment obtains PAS-stained images and multiple fluorescence images of pathological sections of different individuals, identifies the renal tubule contours in the PAS-stained images based on a semantic segmentation model, and identifies the proximal and distal renal tubules (including normal and diseased states) based on the matching of the PAS-stained images and the multiple fluorescence images, thereby quickly constructing a training sample set. The proximal and distal renal tubules recognition model is trained based on the constructed training sample set. The PAS chromosome to be identified is input into the obtained proximal and distal renal tubules recognition model, which can quickly and accurately identify the distal and proximal renal tubules in normal and diseased states, thereby filling a technical gap and assisting doctors in disease diagnosis.

[0066] The thumbnail of PAS staining is as follows Figure 2 As shown in the figure, the black box marks one of the stained areas. Pathological bars are the stained areas in the PAS staining image.

[0067] During implementation, an existing renal tubule semantic segmentation model can be used to perform image semantic segmentation to segment the renal tubule contours in the PAS-stained image.

[0068] Specifically, identifying whether the renal tubule outline in the PAS-stained image is a proximal renal tubule or a distal renal tubule based on the matching of the PAS-stained image and the multiple fluorescence images includes:

[0069] S21, performing global matching on the PAS staining image and the multiple fluorescence images to obtain global matching point pairs and a global affine transformation matrix;

[0070] S22. For each available pathology strip in the PAS-stained image, calculate an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the global matching point pairs and the global affine transformation matrix corresponding to the available pathology strip;

[0071] S23. For each renal tubular contour in the available pathology strip, if the affine transformation matrix corresponding to the renal tubular contour is valid, then based on the affine transformation matrix corresponding to the renal tubular contour, obtain the area corresponding to the renal tubular contour in the multiple fluorescence images; based on the intensity of the area corresponding to the renal tubular contour under the first fluorescence channel and the second fluorescence channel, identify whether the renal tubular contour is a distal renal tubule or a proximal renal tubule.

[0072] By first performing coarse-grained matching at the global level and then fine-grained matching at the tubular level, the area of ​​the tubular contour in the multiple fluorescence images is determined based on the affine transformation matrix corresponding to the fine-grained matched tubular contour. Then, based on the intensity of the area, it is determined whether the tubular contour is the distal tubule or the proximal tubule, thereby achieving efficient and accurate identification of tubular segments.

[0073] During implementation, since DAPI fluorescence staining and Opal 480 fluorescence staining can display cell nuclei and epithelial cells, which is helpful for registration, the multiple fluorescence of the present invention also includes DAPI fluorescence and Opal 480 fluorescence.

[0074] Specifically, the PAS staining image and the multiple fluorescence images are globally matched to obtain global matching point pairs, including:

[0075] S211, performing morphological processing on the superimposed image of the DAPI fluorescence image and the Opal480 fluorescence image;

[0076] S212, performing multi-angle rotation transformation on the processed superimposed image to obtain multiple rotation images;

[0077] S213, matching the PAS staining image with the processed superimposed image and each rotation image;

[0078] S214, taking the matching point pair of the image pair with the highest matching degree as the global matching point pair;

[0079] S215 , performing affine transformation calculation on the global matching point pairs to obtain a global affine transformation matrix.

[0080] During implementation, morphological processing is first performed on the superimposed image to remove the interference of noise.

[0081] During implementation, morphological processing includes background removal, erosion, and dilation. For example, after sorting the pixel intensities in the stacked image from largest to smallest, the top 30% of pixel intensities are considered valid signals, while the remainder are considered background. The corresponding pixel intensities are set to the background intensity value, such as 0, thereby removing some noise interference. The stacked image with the background removed is first eroded and then dilated. The area of ​​each pathology strip is then reduced tenfold, and the maximum value is taken as the threshold. Pathology strips with an area smaller than the threshold are removed, retaining significant large areas, thereby further eliminating interference from noise and small debris areas. Erosion and dilation can use existing erosion and dilation methods in image science.

[0082] In the following process of the present invention, the matching with multiple fluorescence images is performed with the processed superimposed image.

[0083] During implementation, the processed superimposed image is subjected to multi-angle rotation transformation to obtain multiple rotation images. For example, the processed superimposed image is rotated 90 degrees clockwise and 90 degrees counterclockwise to obtain corresponding rotation images.

[0084] During implementation, existing image matching algorithms are used to match the PAS-stained image with the processed overlay image, and also with each rotated image. For example, the loftr matching algorithm is used for image matching. Matching point pairs are found in the image pairs. The number of matching point pairs represents the degree of match; a greater number indicates a higher degree of match. The matching point pair with the highest degree of match is selected as the global matching point pair.

[0085] It should be noted that if the PAS staining image has the highest matching degree with the image obtained by rotating the superimposed image 90 degrees clockwise, the coordinates of the matching points in the image obtained by rotating the superimposed image 90 degrees clockwise need to be converted into the coordinates of the original image.

[0086] Perform affine transformation on the globally matched point pairs to obtain the global affine transformation matrix. The affine transformation can be calculated using the OpenCV estimateAffine2D function, which returns the number of successfully aligned point pairs.

[0087] Global matching is a rough matching of the global image, and fine-grained matching is further performed based on the global matching point pairs and global affine transformation matrix obtained by global matching.

[0088] During implementation, due to the existence of edge effects, that is, when the renal tubules are located at the edge of a complete pathology strip, uneven staining or interference from impurities will affect the color development, for example, both the first and second fluorescence channels in a multiple fluorescence image are positive. Therefore, after the global matching point pairs and the global affine transformation matrix are obtained by global matching, the present invention further includes the following steps before calculating the affine transformation matrix corresponding to each renal tubule outline in the available pathology strip based on the global matching point pairs and the global affine transformation matrix corresponding to the available pathology strip:

[0089] A distance transformation is performed on each pathological strip in the PAS-stained image to calculate the distance from each pixel in the pathological strip to the border of the pathological strip; and pixels whose distance is less than a first threshold are eliminated.

[0090] During implementation, the first threshold can be determined based on the mpp (Microns Per Pixel, the number of microns corresponding to each pixel, used to describe the resolution of digital pathology images, that is, the size of each pixel in actual physical size, usually in microns) value of the image and the scaling ratio. For example, if the stained image has an mpp value, the first threshold is 2000*(mpp / 0.46)*image scaling ratio; if the stained image does not have an mpp value, the first threshold is 2000*image scaling ratio.

[0091] The present invention eliminates small tubes close to the edge based on distance transformation after global matching and before fine-grained matching, thereby preventing interference and improving the accuracy of classification and recognition.

[0092] Fine-grained matching is first performed within the pathology strip. Therefore, it is necessary to first determine which pathology strips are available. For each pathology strip in the PAS-stained image, if there is a global matching point for that pathology strip, then that pathology strip is considered available.

[0093] During implementation, the displacement and deformation of the pathology strips may cause registration deviation. Therefore, the present invention adopts a stripe matching method to solve the registration deviation.

[0094] Specifically, for each available pathology strip in the PAS staining image, the affine transformation matrix corresponding to each renal tubule contour in the available pathology strip is calculated based on the global matching point pair corresponding to the available pathology strip and the global affine transformation matrix, including:

[0095] S221, dividing the available pathology strip area into a plurality of image blocks, and determining the image block to which each renal tubule contour belongs;

[0096] S222, matching each image block with the multiple fluorescence images based on the global matching point pairs and the global affine transformation matrix to obtain local matching point pairs corresponding to each image block;

[0097] S223 . For each renal tubule contour in the available pathology strip, obtain an affine transformation matrix corresponding to the renal tubule contour based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs.

[0098] During implementation, a non-overlapping sliding window can be used to slide across each available pathology strip in the PAS-stained image to divide the available pathology strip into multiple image blocks. For each renal tubular contour in the available pathology strip, the image block to which the renal tubular contour belongs is determined based on the intersection-and-union ratio (IOR) between the renal tubular contour and the image block. For a renal tubular contour, the image block with the largest IOR is selected as the image block representing the renal tubular contour.

[0099] During implementation, the available pathological strip area may be divided into multiple image blocks in the following manner to determine the image block to which the renal tubule contour belongs:

[0100] For each renal tubule contour of the image block to which the available pathology strip has not been determined, the outer bounding box of the renal tubule contour is used as the current region;

[0101] The current window is obtained by taking the upper left corner vertex of the current region as the upper left corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, then the current region remains unchanged;

[0102] The current window is obtained by taking the upper right corner vertex of the current region as the upper right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, then the current region remains unchanged;

[0103] The current window is obtained by taking the lower right corner vertex of the current region as the lower right corner of the window and the preset length as the side length; if there is a renal tubule outline outside the current region and the image block to which it belongs is not determined in the current window, then the common circumscribed border of the renal tubule in the current region and the renal tubule outline within the current window but outside the current region and the image block to which it belongs is taken as the current region; if there is no such border, then the current region remains unchanged;

[0104] The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length; the area of ​​the current window is regarded as an image block; all the renal tubule contours of the image blocks that have not been determined to belong to in the current window belong to this image block.

[0105] by Figure 3 Take this as an example to illustrate. Figure 3 In (a), the circle is the outline of the renal tubule of the image block to which it is not determined, and the solid line box is the outer bounding box of the renal tubule outline. First, the outer bounding box of the renal tubule outline of one of the undetermined corresponding image blocks is used as the current area, that is, the solid line box in the figure is the current area, the upper left corner vertex of the current area is used as the upper left corner of the window, and the preset length is used as the side length to obtain the current window, that is, Figure 3 In the dotted box in (a), there are renal tubule contours in the current window that are not in the current area (solid box) and have no determined image block to which they belong. The common external frame of the renal tubule contours of these two renal tubules is used as the current area, for example Figure 3 The solid line box in (b).

[0106] Then with Figure 3 The upper right corner of the solid line frame (current area) in (b) is used as the upper right corner vertex of the window, and the preset length is the side length to draw the window to get the current window, that is, Figure 3 The dotted box in (b) of the current window. If there is no renal tubule outline that is not in the current area and whose image block is not determined, the current area remains unchanged. The current window is obtained by taking the lower right corner vertex of the current area as the lower right corner of the window and the preset length as the side length, as shown in the following example: Figure 3 The dotted box in (c) of the current window still has no renal tubule contour that is not in the current area and whose image block is not determined. The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length, as shown in the following example: Figure 3 In the dotted box of (d), the area of ​​the current window is regarded as an image block; all the renal tubule contours in the current window that have not been determined to belong to the image block belong to this image block.

[0107] The above process for determining the image blocks to which the renal tubular outlines belong does not uniformly divide the pathological strip into equal-sized blocks. Instead, the blocks are determined based on the density of the renal tubules. The blocks corresponding to areas with dense tubular outlines are larger than those corresponding to areas with sparse tubular outlines. Areas with dense tubular outlines are treated as one block, reducing the number of blocks without compromising registration quality. Areas without tubular outlines do not require blocks to be determined, resulting in fewer blocks than those obtained by uniform division, thus improving computational efficiency.

[0108] Specifically, each image block is matched with the multiple fluorescence images based on the global matching point pair and the global affine transformation matrix to obtain a local matching point pair corresponding to each image block, including:

[0109] S2221. For each image block, perform affine transformation calculation on the global matching point pairs contained in the image block to obtain an affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0110] S2222. If the number N1 of successfully registered point pairs exceeds a second threshold, determining the region corresponding to the image block on the multiple fluorescence images based on the affine transformation matrix M1; otherwise, determining the region corresponding to the image block on the multiple fluorescence images based on the global affine transformation matrix;

[0111] S2223 : Match the image block with the region corresponding to the image block on the multiple fluorescence images to obtain a local matching point corresponding to the image block.

[0112] During implementation, the affine transformation calculation may use the aforementioned estimateAffine2D function of opencv.

[0113] If the number of successfully registered point pairs N1 exceeds a second threshold, the corresponding region of the image block on the multiple fluorescence images is determined based on the affine transformation matrix M1. If the number of successfully registered point pairs N1 does not exceed the second threshold, the corresponding region of the image block on the multiple fluorescence images is determined based on the global affine transformation matrix. This results in more accurate matching. During implementation, the second threshold is set based on the required matching accuracy.

[0114] Specifically, determining the region corresponding to the image block on the multiple fluorescence images based on the affine transformation matrix M1 includes:

[0115] S22221, performing affine transformation on the vertex coordinates of the image block based on the affine transformation matrix M1 to obtain points corresponding to the vertices on the multiple fluorescence images;

[0116] S22222: The area formed by the bounding box of the points corresponding to the vertices on the multiple fluorescence images is the area corresponding to the image block on the multiple fluorescence images.

[0117] During implementation, the coordinates of the four vertices of the image block (top left, top right, bottom right, and bottom left) are calculated based on the affine transformation matrix M1 to correspond to the coordinates of the points in the multiple fluorescence images. The area defined by the bounding box of these corresponding points is the area corresponding to the image block in the multiple fluorescence images. The process for determining the area corresponding to the image block in the multiple fluorescence images based on the global affine transformation matrix is ​​similar.

[0118] Obtain the region corresponding to the image block in the multiple fluorescence images, and match the image block with the region corresponding to the image block in the multiple fluorescence images (e.g., using the loftr matching algorithm for image matching), to obtain a pair of local matching points corresponding to the image block. It should be noted that matching the image block with the region corresponding to the multiple fluorescence images refers to matching the image block with the region corresponding to the superimposed image after the aforementioned morphological processing.

[0119] The registration accuracy is improved by performing fine-grained matching on image blocks.

[0120] For each renal tubule contour in the available pathology strip, an affine transformation matrix corresponding to the renal tubule contour is obtained based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, specifically including:

[0121] S231, performing affine transformation calculation on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, to obtain an affine transformation matrix M2 and the number of successfully registered point pairs N2;

[0122] S232, performing affine transformation calculation on the local matching point pairs located within the renal tubule contour among the local matching points corresponding to the image block to which the renal tubule contour belongs, to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3;

[0123] S233. If the number of successfully registered point pairs N3 exceeds a third threshold, the affine transformation matrix M3 is used as the effective affine transformation matrix corresponding to the renal tubule contour; otherwise:

[0124] If the number of successfully registered point pairs N2 exceeds the third threshold, the affine transformation matrix M2 is used as the valid affine transformation matrix corresponding to the renal tubule contour; otherwise, the affine transformation matrix M2 is used as the invalid affine transformation matrix corresponding to the renal tubule contour.

[0125] During implementation, an affine transformation is first performed on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, resulting in an affine transformation matrix M2 and the number of successfully registered point pairs N2. Furthermore, an affine transformation is performed on the local matching point pairs within the renal tubule contour among the local matching points corresponding to the image block to which they belong, resulting in an affine transformation matrix M3 and the number of successfully registered point pairs N3.

[0126] If the number of successfully aligned point pairs N3 exceeds the third threshold, it means that the area corresponding to the renal tubular contour on the multiple fluorescence images can be accurately determined using only the local matching points within the renal tubular contour. Therefore, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the renal tubular contour. If the number of successfully aligned point pairs N3 does not exceed the third threshold, the number of local matching points within the renal tubular contour is insufficient, which may affect the determination of subsequent areas. Therefore, further judgment is made. If the number of successfully aligned point pairs N2 exceeds the third threshold, the affine transformation matrix M2 is used as the valid affine transformation matrix corresponding to the renal tubular contour. Otherwise, the affine transformation matrix M2 is used as the invalid affine transformation matrix corresponding to the renal tubular contour.

[0127] During implementation, the third threshold is set according to the matching accuracy requirement, for example, 20 pairs.

[0128] If the affine transformation matrix corresponding to the renal tubule contour is valid, then the region corresponding to the renal tubule contour in the multiple fluorescence images is obtained based on the affine transformation matrix corresponding to the renal tubule contour.

[0129] During implementation, the coordinates of the four vertices of the upper left, upper right, lower right, and lower left of the bounding box of the renal tubule outline are calculated based on the affine transformation matrix corresponding to the renal tubule outline to obtain the coordinates of the corresponding points in the multiple fluorescence images. The area formed by the bounding box of these corresponding points is the area corresponding to the renal tubule outline in the multiple fluorescence images.

[0130] Based on the intensity of the area corresponding to the renal tubule outline under the first fluorescence channel and the second fluorescence channel, the renal tubule outline is identified as the distal renal tubule or the proximal renal tubule. Specifically, the method includes:

[0131] Calculating the intensity value of the area corresponding to the renal tubule outline in the first fluorescence channel and the second fluorescence channel of the multiple fluorescence image to obtain a first intensity value and a second intensity value;

[0132] If the first intensity value exceeds a fifth threshold value and the second intensity value does not exceed a sixth threshold value, the renal tubule contour is a distal renal tubule;

[0133] If the first intensity value does not exceed the fifth threshold value, and the second intensity value exceeds the sixth threshold value, the renal tubule contour is a proximal renal tubule;

[0134] If the first intensity value exceeds the fifth threshold and the second intensity value exceeds the sixth threshold, then: if the ratio of the first intensity value to the second intensity value exceeds the seventh threshold, the renal tubule contour is a distal renal tubule; if the ratio of the second intensity value to the first intensity value exceeds the seventh threshold, the renal tubule contour is a proximal renal tubule.

[0135] During implementation, the intensity value of the area corresponding to the renal tubule outline in the first fluorescent channel may be calculated in the following manner:

[0136] If the number of valid pixels in the area corresponding to the renal tubular outline in the first fluorescent channel exceeds a ninth threshold value (e.g., 100), the pixel intensity values ​​of the area corresponding to the renal tubular outline in the first fluorescent channel are sorted from largest to smallest, and the mean of the first 50% of the pixel intensity values ​​in the sequence is taken as the intensity value of the area corresponding to the renal tubular outline in the first fluorescent channel, i.e., the first intensity value. If the number of valid pixels does not exceed the ninth threshold value, the intensity value of the area corresponding to the renal tubular outline (the first intensity value) is set to 0, thereby avoiding small sample error. The ninth threshold value can be determined based on the pixel intensity distribution of the image.

[0137] The second intensity value is obtained in an in-phase manner.

[0138] The distal tubule and the proximal tubule are then determined based on the first intensity value and the second intensity value.

[0139] It should be noted that if the first intensity value exceeds the fifth threshold, and the second intensity value exceeds the sixth threshold, and the ratio of the first intensity value to the second intensity value and the ratio of the second intensity value to the first intensity value do not exceed the seventh threshold, then no label is assigned to the renal tubular contour, and the type of the renal tubular contour needs to be manually confirmed later.

[0140] If the first intensity value does not exceed the fifth threshold value and the second intensity value does not exceed the sixth threshold value, no label is assigned to the renal tubule contour. The type of the renal tubule contour requires subsequent manual confirmation.

[0141] During implementation, the seventh threshold may be set to 1.2, for example.

[0142] The fifth threshold value can be calculated using the Otsu algorithm based on the first intensity values ​​of all renal tubular contours. The sixth threshold value can be calculated using the Otsu algorithm based on the second intensity values ​​corresponding to all renal tubular contours.

[0143] By matching PAS-stained images with multiple fluorescence images, the distal and proximal renal tubules in the PAS-stained images can be quickly identified, significantly reducing labor costs and improving efficiency. Furthermore, by using multiple fluorescence images to identify not only normal but also diseased renal tubular segments in PAS images, the trained model can also identify both normal and diseased distal and proximal tubules.

[0144] A training sample set is constructed based on PAS-stained images that identify the proximal and distal renal tubules. Specifically, a PAS-stained image of an individual case section is used as input data for a sample, and the masks of the distal and proximal renal tubules in the PAS-stained image are used as labels to construct a sample, thereby constructing a training sample set.

[0145] A multi-type segmentation model is constructed, and the multi-type segmentation model is trained based on the constructed training sample set to obtain a trained distal and proximal renal tubule recognition model.

[0146] Specifically, the constructed multi-type segmentation models include:

[0147] The encoder is used to extract semantic features of the input image layer by layer through multiple downsampling layers;

[0148] The middle layer is used to compress the semantic features extracted by the last downsampling layer of the encoder and pass the compressed semantic features to the decoder;

[0149] The decoder is used to gradually decode the semantic features through multiple upsampling layers to obtain decoded features;

[0150] Output layer, used to output segmentation results based on decoded features;

[0151] The number of downsampling layers of the encoder is the same as the number of upsampling layers of the encoder and corresponds one to one; an attention module is connected between the downsampling layer and the corresponding upsampling layer, which is used to perform attention extraction on the semantic features extracted by the downsampling layer and pass them to the corresponding upsampling layer.

[0152] In implementation, the multi-type segmentation model uses a symmetrical encoder-decoder structure to capture spatial information for image semantic segmentation, improving segmentation accuracy. An attention module is also incorporated into the skip link structure. This module weights feature maps, increasing the weight of important features while simultaneously reducing the influence of irrelevant and minor features. This improves the ability to capture important details, allowing the model to focus more on the area to be segmented, and improving segmentation accuracy.

[0153] In implementation, the first encoder extracts semantic features. The encoder consists of four sequentially connected downsampling units, which gradually reduce the size of the feature map while increasing the number of channels. This allows the capture of image features at different scales and improves the accuracy of the segmentation results.

[0154] In practice, a downsampling unit consists of a series of convolutional layers, activation functions, and max pooling layers. The convolutional layers are used for feature extraction, and each downsampling unit typically consists of two consecutive convolution operations. Each convolutional layer is followed by an activation function to increase the model's nonlinear expressiveness. Finally, a max pooling layer performs downsampling, reducing the spatial dimensions of the image by half, thereby reducing computational complexity and obtaining more abstract features.

[0155] The last downsampling unit of the encoder is connected to the middle layer, which compresses and refines the semantic features output by the downsampling unit while preserving the semantic information of the features. In implementation, the middle layer is a convolutional layer.

[0156] During implementation, the output module is a convolutional layer that generates the final segmentation result. The number of channels in the output module corresponds to the segmentation type. This application is used to distinguish the proximal and distal renal tubules in the PAS-stained image, so the segmentation types include three categories: proximal renal tubules, distal renal tubules, and other.

[0157] In implementation, the loss of the multi-class segmentation model is calculated using the following formula:

[0158]

[0159] Among them, M represents the number of pixels of the sample, C represents the number of segmentation types, and N represents the number of samples in the current training batch. Indicates whether the j-th pixel of the i-th sample belongs to the k-th type, if yes, it is 1, otherwise it is 0, It represents the probability that the j-th pixel of the i-th sample belongs to the k-th type predicted by the model.

[0160] During implementation, the gradient descent method is used to update the parameters of the multi-type segmentation model based on the training loss until the model converges to obtain a trained multi-type segmentation model.

[0161] For the PAS staining image to be identified, inputting it into the trained multi-type segmentation model can quickly and accurately identify the distal renal tubules and proximal renal tubules.

[0162] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0163] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A training method for a distal and proximal renal tubule identification model, characterized in that: The following steps are involved: Acquiring PAS-stained images and multiple fluorescence images of pathological sections of different individuals; the multiple fluorescence images include a first fluorescence channel for marking distal renal tubules and a second fluorescence channel for marking proximal renal tubules; Identify the outline of renal tubules in PAS-stained images based on semantic segmentation model; Identifying whether the renal tubule outline in the PAS-stained image is a proximal renal tubule or a distal renal tubule based on the matching of the PAS-stained image and the multiple fluorescence images; and constructing a training sample set based on the PAS-stained image in which the proximal renal tubule and the distal renal tubule are identified; A multi-type segmentation model is constructed, and the multi-type segmentation model is trained based on the training sample set to obtain a trained proximal and distal renal tubule recognition model.

2. The training method for the proximal and distal renal tubule identification model according to claim 1, characterized in that: The multi-type segmentation models constructed include: The encoder is used to extract semantic features of the input image layer by layer through multiple downsampling layers; The middle layer is used to compress the semantic features extracted by the last downsampling layer of the encoder and pass the compressed semantic features to the decoder; The decoder is used to gradually decode the semantic features through multiple upsampling layers to obtain decoded features; Output layer, used to output segmentation results based on decoded features; The number of downsampling layers of the encoder is the same as the number of upsampling layers of the encoder and corresponds one to one; an attention module is connected between the downsampling layer and the corresponding upsampling layer, which is used to perform attention extraction on the semantic features extracted by the downsampling layer and pass them to the corresponding upsampling layer.

3. The training method for the proximal and distal renal tubule identification model according to claim 2, characterized in that: The loss of the multi-class segmentation model is calculated using the following formula: Among them, M represents the number of pixels of the sample, C represents the number of segmentation types, and N represents the number of samples in the current training batch. Indicates whether the j-th pixel of the i-th sample belongs to the k-th type, if yes, it is 1, otherwise it is 0, It represents the probability that the j-th pixel of the i-th sample belongs to the k-th type predicted by the model.

4. The training method for the proximal and distal renal tubule identification model according to claim 1, characterized in that: Identifying whether the renal tubule outline in the PAS-stained image is a proximal renal tubule or a distal renal tubule based on matching between the PAS-stained image and the multiple fluorescence images includes: Performing global matching on the PAS staining image and the multiple fluorescence images to obtain global matching point pairs and a global affine transformation matrix; For each available pathology strip in the PAS staining image, calculating an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the global matching point pairs and the global affine transformation matrix corresponding to the available pathology strip; For each renal tubular contour in the available pathology strip, if the affine transformation matrix corresponding to the renal tubular contour is valid, the area corresponding to the renal tubular contour in the multiple fluorescence images is obtained based on the affine transformation matrix corresponding to the renal tubular contour; based on the intensity of the area corresponding to the renal tubular contour under the first fluorescence channel and the second fluorescence channel, it is identified whether the renal tubular contour is a distal renal tubule or a proximal renal tubule.

5. The training method for the proximal and distal renal tubule identification model according to claim 4, characterized in that: Identifying whether the renal tubule outline is a distal tubule or a proximal tubule based on the intensity of the area corresponding to the renal tubule outline under the first fluorescent channel and the second fluorescent channel includes: Calculating the intensity value of the area corresponding to the renal tubule outline in the first fluorescence channel and the second fluorescence channel of the multiple fluorescence image to obtain a first intensity value and a second intensity value; If the first intensity value exceeds a fifth threshold value and the second intensity value does not exceed a sixth threshold value, the renal tubule contour is a distal renal tubule; If the first intensity value does not exceed the fifth threshold value, and the second intensity value exceeds the sixth threshold value, the renal tubule contour is a proximal renal tubule; If the first intensity value exceeds the fifth threshold and the second intensity value exceeds the sixth threshold, then: if the ratio of the first intensity value to the second intensity value exceeds the seventh threshold, the renal tubule contour is a distal renal tubule; if the ratio of the second intensity value to the first intensity value exceeds the seventh threshold, the renal tubule contour is a proximal renal tubule.

6. The training method for the proximal and distal renal tubule identification model according to claim 4, characterized in that: Calculating an affine transformation matrix corresponding to each renal tubule contour in the available pathology strip based on the global matching point pair and the global affine transformation matrix corresponding to the available pathology strip includes: Dividing the available pathological strip area into a plurality of image blocks, and determining the image block to which each renal tubule contour belongs; Matching each image block with the multiple fluorescence images based on the global matching point pair and the global affine transformation matrix to obtain a local matching point pair corresponding to each image block; For each renal tubule contour in the available pathology strip, an affine transformation matrix corresponding to the renal tubule contour is obtained based on a local matching point pair corresponding to the image block to which the renal tubule contour belongs.

7. The training method for the proximal and distal renal tubule identification model according to claim 6, characterized in that: Obtaining an affine transformation matrix corresponding to the renal tubule contour based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs includes: Perform affine transformation calculation on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, and obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2; Performing affine transformation calculation on the local matching point pairs located within the renal tubule contour among the local matching points corresponding to the image block to which the renal tubule contour belongs, to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3; If the number of successfully registered point pairs N3 exceeds the third threshold, the affine transformation matrix M3 is used as the effective affine transformation matrix corresponding to the renal tubule contour; otherwise: If the number of successfully registered point pairs N2 exceeds the third threshold, the affine transformation matrix M2 is used as the valid affine transformation matrix corresponding to the renal tubule contour; otherwise, the affine transformation matrix M2 is used as the invalid affine transformation matrix corresponding to the renal tubule contour.

8. The method for training the proximal and distal renal tubule identification model according to claim 6, wherein: Performing image block matching on each image block and the multiple fluorescence images based on the global matching point pairs and the global affine transformation matrix to obtain local matching points corresponding to each image block includes: For each image block, perform affine transformation on the global matching point pairs contained in the image block to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1; If the number of successfully registered point pairs N1 exceeds a second threshold, determining the region corresponding to the image block on the multiple fluorescence images based on the affine transformation matrix M1; otherwise, determining the region corresponding to the image block on the multiple fluorescence images based on the global affine transformation matrix; The image block is matched with a region corresponding to the image block on the multiple fluorescence images to obtain a local matching point corresponding to the image block.

9. The method for training the proximal and distal renal tubule identification model according to claim 8, characterized in that: Determining a region corresponding to the image block on the multiple fluorescence images based on the affine transformation matrix M1 includes: Performing affine transformation on the vertex coordinates of the image block based on the affine transformation matrix M1 to obtain the points corresponding to the vertices on the multiple fluorescence images; The area formed by the outer bounding box of the points corresponding to the vertices on the multiple fluorescence images is the area corresponding to the image block on the multiple fluorescence images.

10. The method for training the proximal and distal renal tubule identification model according to claim 4, characterized in that: The multiple fluorescence also includes DAPI fluorescence image and Opal480 fluorescence; Performing global matching on the PAS staining image and the multiple fluorescence images to obtain global matching point pairs includes: Morphological processing was performed on the superimposed images of DAPI fluorescence images and Opal480 fluorescence images; Perform multi-angle rotation transformation on the processed superimposed image to obtain multiple rotation images; Match the PAS-stained image with the processed overlay image and each rotation image; The matching point pair of the image pair with the highest matching degree is taken as the global matching point pair; Perform affine transformation calculation on the global matching point pairs to obtain the global affine transformation matrix.

Images