A training method of a proximal-distal renal tubule identification model

Abstract

The present application relates to a kind of training methods of far and near end renal tubule identification model, belong to renal tubule identification technical field, solve the problem that far end renal tubule and near end renal tubule cannot be accurately identified in prior art.Method includes: obtaining the PAS staining image and multiple fluorescence image of the pathological section of different individuals;Based on semantic segmentation model, identify the renal tubule contour in PAS staining image;Based on the matching of the PAS staining image and multiple fluorescence image, identify whether the renal tubule contour in the PAS staining image is near end renal tubule or far end renal tubule;Based on the PAS staining image that near end renal tubule and far end renal tubule are identified, construct training sample set;Construct multiple type segmentation model, based on the training sample set, train the multiple type segmentation model to obtain the trained far and near end renal tubule identification model.Accurately identify far end renal tubule and near end renal tubule quickly is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of renal tubule recognition, and particularly relates to a training method of a proximal and distal renal tubule recognition model. BACKGROUND

[0002] The renal tubule is one of the core units of kidney structure and function, which is composed of proximal renal tubule, distal renal tubule and collecting duct (the proximal and distal renal tubules can be further divided into curved and straight parts). Among them, the proximal renal tubule is located at the initial segment of the renal tubule and is directly connected with the renal capsule, and the luminal surface of the proximal renal tubule is covered with dense microvilli structures, which is mainly responsible for the active reabsorption of about 70% of water, electrolytes (such as sodium and potassium), glucose, amino acids and other nutrients in the primary urine, and participates in the regulation of acid-base balance. The distal renal tubule includes the thick ascending limb of the loop of Henle and the distal convoluted tubule, which is located after the thick ascending limb of the loop of Henle and is connected with the collecting duct. The epithelial cells of the distal convoluted tubule have fewer microvilli but higher content of mitochondria, and mainly undertake the fine adjustment of electrolytes (such as sodium ion reabsorption and potassium ion secretion) and the final regulation of acid-base balance, which is directly affected by hormones such as aldosterone.

[0003] The damage of different segments of the renal tubule is closely related to specific kidney diseases. For example, the early stage of acute kidney injury (AKI) often shows necrosis of proximal renal tubular epithelial cells or shedding of brush border, while the damage of distal renal tubule is significantly associated with the progression of chronic kidney disease (CKD) and electrolyte disorders (such as hypokalemia). In addition, the differential diagnosis of genetic diseases such as Bartter syndrome (sodium and potassium metabolism abnormality in distal renal tubule) and Fanconi syndrome (extensive dysfunction of proximal renal tubule) highly depends on the accurate identification of the segment of the renal tubule.

[0004] The existing deep learning neural network model is based on the experience of pathologists to identify and label the relatively normal renal tubule on the PAS staining image. Although the PAS staining can distinguish the normal proximal and distal tubules by showing the brush border, the characteristics of the brush border are easily lost in the pathological state, so it is difficult to distinguish whether it is a distal renal tubule or a proximal renal tubule. Therefore, the existing model cannot accurately identify the distal renal tubule and the proximal renal tubule, especially the distal renal tubule and the proximal renal tubule in the pathological state. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a training method of a proximal and distal renal tubule recognition model to solve the problem that the existing model cannot accurately identify the distal renal tubule and the proximal renal tubule.

[0006] In one aspect, the embodiments of the present application provide a training method of a proximal and distal renal tubule recognition model, which comprises the following steps:

[0007] Obtaining a PAS staining image and a multi-fluorescence image of pathological sections of different individuals; the multi-fluorescence image includes a first fluorescence channel for marking distal renal tubules and a second fluorescence channel for marking proximal renal tubules;

[0008] Identifying the renal tubule contour in the PAS staining image based on a semantic segmentation model;

[0009] Identifying whether the renal tubule contour in the PAS staining image is a proximal renal tubule or a distal renal tubule based on matching of the PAS staining image and the multi-fluorescence image; constructing a training sample set based on the PAS staining images in which the proximal renal tubule and the distal renal tubule are identified;

[0010] Constructing a multi-type segmentation model, and training the multi-type segmentation model based on the training sample set to obtain a trained proximal and distal renal tubule identification model.

[0011] Based on the further improvement of the above scheme, the constructed multi-type segmentation model includes:

[0012] An encoder for performing layer-by-layer semantic feature extraction on the input image through multiple down-sampling layers;

[0013] An intermediate layer for compressing the semantic features extracted by the last down-sampling layer of the encoder and transmitting the compressed semantic features to a decoder;

[0014] A decoder for decoding the semantic features step by step through multiple up-sampling layers to obtain decoding features;

[0015] An output layer for outputting a segmentation result based on the decoding features;

[0016] The number of down-sampling layers of the encoder and the number of up-sampling layers of the encoder are the same and one-to-one corresponding; the attention module is connected between the down-sampling layer and the corresponding up-sampling layer for attention extraction of the semantic features extracted by the down-sampling layer and transmission to the corresponding up-sampling layer.

[0017] Based on the further improvement of the above scheme, the loss of the multi-type segmentation model is calculated using the following formula:

[0018]

[0019] Wherein, M represents the number of pixels of the sample, C represents the number of segmentation types, N represents the number of samples in the current training batch, represents whether the jth pixel of the ith sample belongs to the kth type, and is 1 if it belongs to the kth type, otherwise it is 0, represents the probability of the jth pixel of the ith sample predicted by the model belonging to the kth type.

[0020] Based on the further improvement of the above scheme, based on the matching of the PAS staining image and the multiple fluorescence image, identifying whether the renal tubule contour in the PAS staining image is a proximal renal tubule or a distal renal tubule, comprising:

[0021] Performing global matching on the PAS staining image and the multiple fluorescence image to obtain global matching point pairs and a global affine transformation matrix;

[0022] For each available pathological strip in the PAS staining image, the available pathological strip region is divided into multiple image blocks, and each image block is matched with the multiple fluorescence image based on the global matching point pairs and the global affine transformation matrix to obtain the local matching point pairs corresponding to each image block;

[0023] For each renal tubule contour in the available pathological strip, the affine transformation matrix corresponding to the renal tubule contour is obtained based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs;

[0024] If the affine transformation matrix corresponding to the renal tubule contour is valid, the region corresponding to the renal tubule contour in the multiple fluorescence image is obtained based on the affine transformation matrix corresponding to the renal tubule contour; and whether the renal tubule contour is a distal renal tubule or a proximal renal tubule is identified based on the intensity of the region corresponding to the renal tubule contour under the first fluorescence channel and the second fluorescence channel.

[0025] Based on the further improvement of the above scheme, based on the intensity of the region corresponding to the renal tubule contour under the first fluorescence channel and the second fluorescence channel, identifying whether the renal tubule contour is a distal tubule or a proximal tubule, comprising:

[0026] The intensity values of the region corresponding to the renal tubule contour are calculated in the first fluorescence channel image and the second fluorescence channel of the multiple fluorescence image to obtain a first intensity value and a second intensity value;

[0027] If the first intensity value exceeds a fifth threshold value, and the second intensity value does not exceed a sixth threshold value, the renal tubule contour is a distal renal tubule;

[0028] If the first intensity value does not exceed the fifth threshold value, and the second intensity value exceeds the sixth threshold value, the renal tubule contour is a proximal renal tubule;

[0029] If the first intensity value exceeds the fifth threshold value, and the second intensity value exceeds the sixth threshold value, then: if the ratio of the first intensity value to the second intensity value exceeds a seventh threshold value, the renal tubule contour is a distal renal tubule, and if the ratio of the second intensity value to the first intensity value exceeds the seventh threshold value, the renal tubule contour is a proximal renal tubule.

[0030] Based on the further improvement of the above scheme, the image block matching is performed on each image block and the multi-fluorescence image based on the global matching point pair and the global affine transformation matrix to obtain the local matching point corresponding to each image block, including:

[0031] For each image block, affine transformation is performed on the global matching point pair contained in the image block to obtain the affine transformation matrix M1 and the number N1 of successfully registered point pairs;

[0032] If the number N1 of successfully registered point pairs exceeds the second threshold value, the corresponding region of the image block on the multi-fluorescence image is determined based on the affine transformation matrix M1; otherwise, the corresponding region of the image block on the multi-fluorescence image is determined based on the global affine transformation matrix;

[0033] The image block and the corresponding region of the image block on the multi-fluorescence image are matched to obtain the local matching point corresponding to the image block.

[0034] Based on the further improvement of the above scheme, the corresponding region of the image block on the multi-fluorescence image is determined based on the affine transformation matrix M1, including:

[0035] The vertex coordinates of the image block are subjected to affine transformation based on the affine transformation matrix M1 to obtain the corresponding points of the vertices on the multi-fluorescence image;

[0036] The region formed by the circumscribed frame of the corresponding points of the vertices on the multi-fluorescence image is the corresponding region of the image block on the multi-fluorescence image.

[0037] Based on the further improvement of the above scheme, the affine transformation matrix corresponding to the renal tubular contour is obtained based on the local matching point pair corresponding to the image block to which the renal tubular contour belongs, including:

[0038] Affine transformation is performed on the local matching point pair corresponding to the image block to which the renal tubular contour belongs to obtain the affine transformation matrix M2 and the number N2 of successfully registered point pairs;

[0039] Affine transformation is performed on the local matching point pair located in the renal tubular contour among the local matching points corresponding to the image block to which the renal tubular contour belongs to obtain the affine transformation matrix M3 and the number N3 of successfully registered point pairs;

[0040] If the number N3 of successfully registered point pairs exceeds the third threshold value, the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the renal tubular contour; otherwise:

[0041] If the number N2 of successfully registered point pairs exceeds the third threshold value, the affine transformation matrix M2 is taken as the effective affine transformation matrix corresponding to the renal tubular contour; otherwise, the affine transformation matrix M2 is taken as the invalid affine transformation matrix corresponding to the renal tubular contour.

[0042] Based on the further improvement of the above scheme, the multiple fluorescence further includes a DAPI fluorescence image and an Opal480 fluorescence;

[0043] The global matching point pair is obtained by globally matching the PAS staining image and the multiple fluorescence image, including:

[0044] The superimposed image of the DAPI fluorescence image and the Opal480 fluorescence image is morphologically processed;

[0045] The processed superimposed image is subjected to multi-angle rotation transformation to obtain multiple rotation images;

[0046] The PAS staining image and the processed superimposed image and each rotation image are matched;

[0047] The matching point pair of the image pair with the highest matching degree is taken as the global matching point pair;

[0048] The global affine transformation matrix is calculated by affine transformation of the global matching point pair.

[0049] Based on the further improvement of the above scheme, after the global matching obtains the global matching point pair and the global affine transformation matrix, before the available pathological strip region in the PAS staining image is segmented into multiple image blocks, it further includes:

[0050] The distance transformation is performed on each pathological strip in the PAS staining image, the distance of each pixel point in each pathological strip to the boundary of the pathological strip is calculated, and the pixel points with a distance less than a first threshold value are removed.

[0051] Compared with the prior art, the PAS staining image and the multiple fluorescence image of the pathological section of different individuals are obtained, the renal tubule contour in the PAS staining image is identified based on a semantic segmentation model, the proximal renal tubule and the distal renal tubule (including normal state and lesion state) are identified based on the matching of the PAS staining image and the multiple fluorescence image, and then the training sample set is quickly constructed, the proximal and distal renal tubule identification model is trained based on the constructed training sample set, and the PAS staining body to be identified is input into the obtained proximal and distal renal tubule identification model, so that the distal renal tubule and the proximal renal tubule in the normal state and the lesion state can be quickly and accurately identified, thereby filling the technical gap and assisting doctors in disease judgment.

[0052] In the present application, each of the above technical solutions can be combined with each other to realize more preferred combination schemes. Other features and advantages of the present application will be set forth in the subsequent description, and some advantages will become apparent from the description, or will be understood by those skilled in the art through implementation of the present application. The purpose and other advantages of the present application can be realized and obtained from the contents specifically pointed out in the specification and the drawings. Attached Figure Description

[0053] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0054] Figure 1 This is a flowchart of the training method for the proximal and distal renal tubule recognition model according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of PAS staining images according to an embodiment of the present invention;

[0056] Figure 3 A schematic diagram illustrating the process of determining the image block to which the renal tubule contour belongs in an embodiment of the present invention. Detailed Implementation

[0057] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0058] A specific embodiment of the present invention discloses a training method for a proximal and distal renal tubule recognition model, such as... Figure 1 The steps shown are as follows:

[0059] S1. Obtain PAS-stained images and multiplex fluorescence images of pathological sections from different individuals; the multiplex fluorescence images include a first fluorescence channel for labeling distal renal tubules and a second fluorescence channel for labeling proximal renal tubules;

[0060] S2. Identify renal tubular contours in PAS-stained images based on semantic segmentation models;

[0061] S3. Based on the matching of the PAS staining image and the multiple fluorescence image, identify whether the renal tubule outline in the PAS staining image is a proximal renal tubule or a distal renal tubule; construct a training sample set based on the PAS staining images that have identified the proximal and distal renal tubules;

[0062] S3. Construct a multi-type segmentation model, and train the multi-type segmentation model based on the training sample set to obtain a trained proximal and distal renal tubule recognition model.

[0063] Since the PAS staining is less time-consuming and less costly than the fluorescence staining, the PAS staining image is usually used to train the neural network model. Although the PAS staining can show the brush border, the cytoplasmic acidophilic difference between the distal tubule and the proximal tubule is easily affected by the staining batch or the tissue fixation quality under the conventional light microscope, and the color difference is easy to confuse the labeling personnel, and the characteristics of the brush border in the pathological state are easy to lose, so it is difficult to distinguish the distal tubule and the proximal tubule. Therefore, the present application identifies the proximal tubule and the distal tubule in the PAS staining image based on the matching of the PAS staining image and the multiple fluorescence image, thereby not only improving the identification efficiency, but also reducing the labeling errors.

[0064] In implementation, the first fluorescence channel can be Opal520 fluorescence, and the second fluorescence channel can be Opal620 fluorescence.

[0065] Compared with the prior art, the training method of the proximal and distal tubule recognition model provided in the embodiment can obtain the PAS staining image and the multiple fluorescence image of the pathological section of different individuals, recognize the tubule contour in the PAS staining image based on a semantic segmentation model, identify the proximal tubule and the distal tubule (including the normal state and the pathological state) in the PAS staining image and the multiple fluorescence image based on the matching, and then quickly construct a training sample set, train the proximal and distal tubule recognition model based on the constructed training sample set, and input the obtained proximal and distal tubule recognition model to the PAS staining image to be recognized, so as to quickly and accurately identify the distal tubule and the proximal tubule in the normal state and the pathological state, thereby filling the technical gap and assisting doctors in disease judgment.

[0066] The thumbnail of the PAS staining is as shown in Figure 2 The black box marks one of the color developing regions. The pathology strip is a block of color developing region in the PAS staining image.

[0067] In implementation, the existing renal tubule semantic segmentation model can be used for image semantic segmentation to segment the tubule contour in the PAS staining image.

[0068] Specifically, identifying whether the tubule contour in the PAS staining image is a proximal tubule or a distal tubule based on the matching of the PAS staining image and the multiple fluorescence image comprises:

[0069] S21, globally matching the PAS staining image and the multiple fluorescence image to obtain a global matching point pair and a global affine transformation matrix;

[0070] S22, for each available pathology strip in the PAS staining image, calculating the affine transformation matrix corresponding to each tubule contour in the available pathology strip based on the global matching point pair and the global affine transformation matrix corresponding to the available pathology strip;

[0071] S23, for each tubular profile in the available pathological strip, if the affine transformation matrix corresponding to the tubular profile is valid, obtaining the region corresponding to the tubular profile in the multiple fluorescence image based on the affine transformation matrix corresponding to the tubular profile; identifying whether the tubular profile is a distal tubular profile or a proximal tubular profile based on the intensity of the region corresponding to the tubular profile under the first fluorescence channel and the second fluorescence channel.

[0072] By first performing coarse-grained matching at the global level, then performing fine-grained matching at the tubular level, and finally determining the region of the tubular profile in the multiple fluorescence image based on the affine transformation matrix of the tubular profile matched in the fine-grained manner, and then determining whether the tubular profile is a distal tubular profile or a proximal tubular profile based on the intensity of the region, the efficient and accurate identification of the tubular segment is realized.

[0073] In implementation, since DAPI fluorescence staining and Opal 480 fluorescence staining can display cell nuclei and epithelial cells, which helps to perform registration, the multiple fluorescence of the present application further includes DAPI fluorescence and Opal 480 fluorescence.

[0074] Specifically, the global matching of the PAS-stained image and the multiple fluorescence image obtains global matching point pairs, including:

[0075] S211, performing morphological processing on the superimposed image of the DAPI fluorescence image and the Opal 480 fluorescence image;

[0076] S212, performing multi-angle rotation transformation on the processed superimposed image to obtain multiple rotated images;

[0077] S213, matching the PAS-stained image with the processed superimposed image and each rotated image;

[0078] S214, taking the matching point pair of the image pair with the highest matching degree as the global matching point pair;

[0079] S215, performing affine transformation calculation on the global matching point pair to obtain a global affine transformation matrix.

[0080] In implementation, the superimposed image is first morphologically processed to remove the interference of noise.

[0081] In implementation, the morphological processing includes background removal, erosion and dilation. For example, after sorting the pixel intensity values in the superimposed image from large to small, the top 30% of the pixel intensity values are taken as the effective signal, and the others are considered as background, and the corresponding pixel intensity values are set as the background intensity value, for example, 0, thereby removing part of the noise interference. The superimposed image after the background removal is first eroded and then dilated; then the area of each pathological bar is reduced by ten times, and the maximum value is taken as the threshold value; the pathological bars with an area smaller than the threshold value are removed, and the significant large area is retained, thereby further excluding the noise and small fragment area interference, and the erosion and dilation can use the existing erosion and dilation method in imageology.

[0082] In the following process of the present application, the matching with the multiple fluorescence images is all matching with the processed superimposed image.

[0083] In implementation, the processed superimposed image is subjected to multi-angle rotation transformation to obtain multiple rotated images, for example, the processed superimposed image is rotated clockwise by 90 degrees and counterclockwise by 90 degrees to obtain the corresponding rotated images.

[0084] In implementation, the existing image matching algorithm is used to match the PAS staining image with the processed superimposed image and to match the PAS staining image with each rotated image. For example, the loftr matching algorithm is used for image matching to find the matching point pairs in the image pair, and the number of matching point pairs represents the matching degree, and the more the number is, the higher the matching degree is. The matching point pairs of the image pair with the highest matching degree are selected as the global matching point pairs.

[0085] It should be noted that if the PAS staining image has the highest matching degree with the image rotated clockwise by 90 degrees, the coordinates of the matching points in the image rotated clockwise by 90 degrees need to be converted to the coordinates of the original image.

[0086] The global matching point pairs are subjected to affine transformation calculation to obtain the global affine transformation matrix. In implementation, the affine transformation calculation can use the estimateAffine2D function of opencv, which will return the number of successfully registered point pairs.

[0087] The global matching is a rough matching of the global image, and the global matching point pairs and the global affine transformation matrix obtained based on the global matching are further subjected to fine-grained matching.

[0088] In implementation, due to edge effect, that is, when the renal tubule is located at the edge of the complete pathological strip, the color development is affected due to uneven staining or impurity interference, for example, the first fluorescence channel and the second fluorescence channel in the multiple fluorescence image are both positive. Therefore, after the global matching point pair and the global affine transformation matrix are obtained by global matching, for each available pathological strip in the PAS staining image, the affine transformation matrix corresponding to each renal tubule contour in the available pathological strip is calculated based on the global matching point pair and the global affine transformation matrix corresponding to the available pathological strip, and before the affine transformation matrix is calculated, the method further comprises the following steps of:

[0089] For each pathological strip in the PAS staining image, the distance transformation is performed to calculate the distance from each pixel point in the pathological strip to the boundary of the pathological strip; and the pixel point with a distance less than a first threshold value is removed.

[0090] In implementation, the first threshold value can be determined according to the mpp (Microns Per Pixel, that is, the number of microns corresponding to each pixel, which is used to describe the resolution of the digital pathology image, that is, the size of each pixel in the actual physical size, which is usually measured in microns) value and the scaling ratio of the image, for example, if the staining image has an mpp value, the first threshold value is 2000*(mpp / 0.46)*image scaling ratio; if the staining image does not have an mpp value, the first threshold value is 2000*image scaling ratio.

[0091] According to the method, after the global matching, the tubules close to the edge are removed based on the distance transformation before the fine-grained matching, so that the interference is prevented and the accuracy of classification and identification is improved.

[0092] The fine-grained matching is first performed in the pathological strip. Therefore, it is necessary to determine which pathological strips are available. For each pathological strip in the PAS staining image, if the pathological strip has a global matching point, the pathological strip is an available pathological strip.

[0093] In implementation, due to the displacement and deformation of the pathological strip, the registration is deviated. Therefore, the method adopts the strip matching mode to solve the registration deviation.

[0094] Specifically, for each available pathological strip in the PAS staining image, the affine transformation matrix corresponding to each renal tubule contour in the available pathological strip is calculated based on the global matching point pair and the global affine transformation matrix corresponding to the available pathological strip, and the method comprises the following steps of:

[0095] S221, the available pathological strip region is divided into a plurality of image blocks, and the image block to which each renal tubule contour belongs is determined;

[0096] S222, each image block is matched with the multiple fluorescence image based on the global matching point pair and the global affine transformation matrix, and a local matching point pair corresponding to each image block is obtained.

[0097] S223、for each tubule contour in the available pathology strip, obtaining an affine transformation matrix corresponding to the tubule contour based on the local matching point pairs corresponding to the image block to which the tubule contour belongs.

[0098] In implementation, for each available pathology strip in the PAS-stained image, a non-overlapping sliding window can be used to slide over each available pathology strip to divide the available pathology strip into a plurality of image blocks. For each tubule contour in the available pathology strip, the image block to which the tubule contour belongs is determined according to the intersection-over-union ratio of the tubule contour and the image block. For a tubule contour, the image block with the largest intersection-over-union ratio is taken as the image block to which the tubule contour belongs.

[0099] In implementation, the available pathology strip region can also be divided into a plurality of image blocks in the following manner to determine the image block to which the tubule contour belongs:

[0100] for each tubule contour in the available pathology strip that does not have a determined image block to which it belongs, taking the circumscribed frame of the tubule contour as a current region;

[0101] taking the top-left corner vertex of the current region as the top-left corner of a window and a preset length as the side length to obtain a current window; if there is a tubule contour in the current window that is located outside the current region and does not have a determined image block to which it belongs, taking the common circumscribed frame of the tubules in the current region and the tubule contour in the current window but located outside the current region and not having a determined image block to which it belongs as the current region; if not, keeping the current region unchanged;

[0102] taking the top-right corner vertex of the current region as the top-right corner of a window and a preset length as the side length to obtain a current window; if there is a tubule contour in the current window that is located outside the current region and does not have a determined image block to which it belongs, taking the common circumscribed frame of the tubules in the current region and the tubule contour in the current window but located outside the current region and not having a determined image block to which it belongs as the current region; if not, keeping the current region unchanged;

[0103] taking the top-right corner vertex of the current region as the top-right corner of a window and a preset length as the side length to obtain a current window; if there is a tubule contour in the current window that is located outside the current region and does not have a determined image block to which it belongs, taking the common circumscribed frame of the tubules in the current region and the tubule contour in the current window but located outside the current region and not having a determined image block to which it belongs as the current region; if not, keeping the current region unchanged;

[0104] taking the top-right corner vertex of the current region as the top-right corner of a window and a preset length as the side length to obtain a current window; if there is a tubule contour in the current window that is located outside the current region and does not have a determined image block to which it belongs, taking the common circumscribed frame of the tubules in the current region and the tubule contour in the current window but located outside the current region and not having a determined image block to which it belongs as the current region; if not, keeping the current region unchanged;

[0105] taking the top-right corner vertex of the current region as the top-right corner of a window and a preset length as the side length to obtain a current window; if there is a tubule contour in the current window that is located outside the current region and does not have a determined image block to which it belongs, taking the common circumscribed frame of the tubules in the current region and the tubule contour in the current window but located outside the current region and not having a determined image block to which it belongs as the current region; if not, keeping the current region unchanged;Figure 3 For example. Figure 3 In (a) of FIG. 1, the circle is a tubule contour which does not determine the corresponding image block, and the solid line frame is the circumscribed frame of the tubule contour. One of the circumscribed frames of the tubule contours which do not determine the corresponding image block is taken as the current region, i.e., the solid line frame in the figure is the current region. The top-left corner of the current region is taken as the top-left corner of the window, and the preset length is the side length to draw the current window, i.e. Figure 3 The dashed line frame in (a) of FIG. 1, there are tubule contours in the current window which are not in the current region (solid line frame) and do not determine the corresponding image block. The common circumscribed frame of the tubule contours of the two tubules is taken as the current region, for example Figure 3 The solid line frame in (b) of FIG. 1.

[0106] Then the top-right corner of the solid line frame (current region) in (b) of FIG. 1 is taken as the top-right corner of the window, and the preset length is the side length to draw the current window, i.e. Figure 3 The dashed line frame in (b) of FIG. 1. There are no tubule contours in the current window which are not in the current region and do not determine the corresponding image block, so the current region is unchanged. The bottom-right corner of the current region is taken as the bottom-right corner of the window, and the preset length is the side length to draw the current window, i.e. Figure 3 The dashed line frame in (c) of FIG. 1. There are still no tubule contours in the current window which are not in the current region and do not determine the corresponding image block, so the bottom-left corner of the current region is taken as the bottom-left corner of the window, and the preset length is the side length to draw the current window, i.e. Figure 3 The dashed line frame in (d) of FIG. 1. There are still no tubule contours in the current window which are not in the current region and do not determine the corresponding image block, so the bottom-left corner of the current region is taken as the bottom-left corner of the window, and the preset length is the side length to draw the current window, i.e. Figure 3 The dashed line frame in (d) of FIG. 1. The region of the current window is taken as an image block; all the tubule contours in the current window which do not determine the corresponding image block belong to the image block.

[0107] The above process of determining the image block to which the tubule contour belongs is not to uniformly divide the pathological strip into image blocks of equal size, but to determine the image block according to the density of the tubule contour. The size of the determined image block is different, and the image block corresponding to the region where the tubule contour is dense is larger than the image block corresponding to the region where the tubule contour is sparse. The region where the tubule contour is dense is taken as an image block, which reduces the number of image blocks while not reducing the registration quality. The region without tubule contour does not need to determine the image block, and the determined image block is less than the image block obtained by uniform division, thereby improving the calculation efficiency.

[0108] Specifically, based on the global matching point pair and the global affine transformation matrix, each image block is matched with the multi-fluorescence image to obtain a local matching point pair corresponding to each image block, including:

[0109] S2221, for each image block, affine transformation is performed on the global matching point pair contained in the image block to obtain an affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0110] S2222, if the number N1 of successfully matched point pairs exceeds the second threshold value, determining the corresponding region of the image block on the multi-fluorescence image based on the affine transformation matrix M1; otherwise, determining the corresponding region of the image block on the multi-fluorescence image based on the global affine transformation matrix;

[0111] S2223, matching the image block and the corresponding region of the image block on the multi-fluorescence image to obtain the local matching point pairs corresponding to the image block.

[0112] In implementation, the affine transformation calculation can use the estimateAffine2D function of the aforementioned opencv.

[0113] If the number N1 of successfully matched point pairs exceeds the second threshold value, the corresponding region of the image block on the multi-fluorescence image is determined according to the affine transformation matrix M1. If the number N1 of successfully matched point pairs does not exceed the second threshold value, the corresponding region of the image block on the multi-fluorescence image is determined based on the global affine transformation matrix. Thus, the matching is more accurate. In implementation, the second threshold value is set according to the matching accuracy requirement.

[0114] Specifically, determining the corresponding region of the image block on the multi-fluorescence image based on the affine transformation matrix M1 includes:

[0115] S22221, performing affine transformation on the coordinates of the vertices of the image block based on the affine transformation matrix M1 to obtain the corresponding points of the vertices on the multi-fluorescence image;

[0116] S22222, the region constituted by the circumscribed frame of the corresponding points of the vertices on the multi-fluorescence image is the corresponding region of the image block on the multi-fluorescence image.

[0117] In implementation, the coordinates of the top-left, top-right, bottom-right, and bottom-left vertices of the image block are calculated according to the affine transformation matrix M1 to obtain the coordinates of the corresponding points in the multi-fluorescence image. The region constituted by the circumscribed frame of these corresponding points is the corresponding region of the image block in the multi-fluorescence image. The process of determining the corresponding region of the image block on the multi-fluorescence image based on the global affine transformation matrix is also the same.

[0118] After obtaining the corresponding region of the image block in the multi-fluorescence image, the image block and the corresponding region thereof in the multi-fluorescence image are matched (for example, the loftr matching algorithm is used for image matching) to obtain the local matching point pairs corresponding to the image block. It should be noted that the matching of the image block and the corresponding region thereof in the multi-fluorescence image is the matching of the image block and the corresponding region thereof in the aforementioned morphologically processed superimposed image.

[0119] Through fine-grained matching of the image block, the accuracy of registration is improved.

[0120] For each tubule contour in the available pathological strip, an affine transformation matrix corresponding to the tubule contour is obtained based on the local matching point pairs corresponding to the image block to which the tubule contour belongs, specifically comprising:

[0121] S231, affine transformation calculation is performed on the local matching point pairs corresponding to the image block to which the tubule contour belongs, to obtain an affine transformation matrix M2 and the number N2 of successfully registered point pairs;

[0122] S232, affine transformation calculation is performed on the local matching point pairs located in the tubule contour among the local matching points corresponding to the image block to which the tubule contour belongs, to obtain an affine transformation matrix M3 and the number N3 of successfully registered point pairs;

[0123] S233, if the number N3 of successfully registered point pairs exceeds a third threshold, the affine transformation matrix M3 is taken as the valid affine transformation matrix corresponding to the tubule contour; otherwise:

[0124] If the number N2 of successfully registered point pairs exceeds the third threshold, the affine transformation matrix M2 is taken as the valid affine transformation matrix corresponding to the tubule contour; otherwise, the affine transformation matrix M2 is taken as the invalid affine transformation matrix corresponding to the tubule contour.

[0125] In implementation, affine transformation calculation is first performed on the local matching point pairs corresponding to the image block to which the tubule contour belongs, to obtain an affine transformation matrix M2 and the number N2 of successfully registered point pairs. Further affine transformation calculation is performed on the local matching point pairs located in the tubule contour among the local matching points corresponding to the image block to which the tubule contour belongs, to obtain an affine transformation matrix M3 and the number N3 of successfully registered point pairs.

[0126] If the number N3 of successfully registered point pairs exceeds the third threshold, it indicates that only the local matching points in the tubule contour can be used to accurately determine the region corresponding to the tubule contour on the multiple fluorescence image, and therefore the affine transformation matrix M3 is taken as the valid affine transformation matrix corresponding to the tubule contour. If the number N3 of successfully registered point pairs does not exceed the third threshold, the number of local matching points in the tubule contour is insufficient, which may affect the determination of the subsequent region. Therefore, further judgment is performed. If the number N2 of successfully registered point pairs exceeds the third threshold, the affine transformation matrix M2 is taken as the valid affine transformation matrix corresponding to the tubule contour; otherwise, the affine transformation matrix M2 is taken as the invalid affine transformation matrix corresponding to the tubule contour.

[0127] In implementation, the third threshold is set according to the matching accuracy requirement, for example, 20 pairs.

[0128] If the affine transformation matrix corresponding to the renal tubule contour is valid, a region corresponding to the renal tubule contour in the multiple fluorescence image is obtained based on the affine transformation matrix corresponding to the renal tubule contour.

[0129] In implementation, coordinates of the four vertices of the bounding box of the renal tubule contour, i.e., the upper-left vertex, the upper-right vertex, the lower-right vertex and the lower-left vertex, are calculated according to the corresponding points of the renal tubule contour in the multiple fluorescence image based on the affine transformation matrix corresponding to the renal tubule contour. The region formed by the bounding box of the corresponding points is the region corresponding to the renal tubule contour in the multiple fluorescence image.

[0130] The intensity of the region corresponding to the renal tubule contour under the first fluorescence channel and the second fluorescence channel is used to identify whether the renal tubule contour is a distal renal tubule or a proximal renal tubule. Specifically, the method comprises the following steps.

[0131] Intensity values of the region corresponding to the renal tubule contour in the first fluorescence channel and the second fluorescence channel of the multiple fluorescence image are calculated to obtain a first intensity value and a second intensity value.

[0132] If the first intensity value exceeds a fifth threshold value and the second intensity value does not exceed a sixth threshold value, the renal tubule contour is a distal renal tubule.

[0133] If the first intensity value does not exceed the fifth threshold value and the second intensity value exceeds the sixth threshold value, the renal tubule contour is a proximal renal tubule.

[0134] If the first intensity value exceeds the fifth threshold value and the second intensity value exceeds the sixth threshold value, if the ratio of the first intensity value to the second intensity value exceeds a seventh threshold value, the renal tubule contour is a distal renal tubule, and if the ratio of the second intensity value to the first intensity value exceeds the seventh threshold value, the renal tubule contour is a proximal renal tubule.

[0135] In implementation, the intensity value of the region corresponding to the renal tubule contour in the first fluorescence channel can be calculated in the following manner:

[0136] If the number of valid pixels of the region corresponding to the renal tubule contour in the first fluorescence channel exceeds a ninth threshold value (for example, 100), the pixel intensity values of the region corresponding to the renal tubule contour in the first fluorescence channel are sorted in descending order, and the mean value of the first 50% of the pixel intensity values in the sequence is taken as the intensity value of the region corresponding to the renal tubule contour in the first fluorescence channel, i.e., the first intensity value. If the number of valid pixels does not exceed the ninth threshold value, the intensity value of the region corresponding to the renal tubule contour (the first intensity value) is set to 0, thereby avoiding small sample errors. The ninth threshold value can be determined according to the pixel intensity distribution of the image.

[0137] The second intensity value is obtained in the same manner.

[0138] Then, the distal tubule and the proximal tubule are determined based on the first intensity value and the second intensity value.

[0139] It should be noted that if the first intensity value exceeds the fifth threshold value, and the second intensity value exceeds the sixth threshold value, and the ratio of the first intensity value to the second intensity value, the ratio of the second intensity value to the first intensity value does not exceed the seventh threshold value, the kidney tubule profile is not assigned a label, and the type of the kidney tubule profile needs to be confirmed by a person later.

[0140] If the first intensity value does not exceed the fifth threshold value, and the second intensity value does not exceed the sixth threshold value, the kidney tubule profile is not assigned a label. The type of the kidney tubule profile needs to be confirmed by a person later.

[0141] In implementation, the seventh threshold value may be set to 1.2, for example.

[0142] The fifth threshold value may be calculated using the Otsu algorithm according to the first intensity values of all kidney tubule profiles. The sixth threshold value may be calculated using the Otsu algorithm according to the second intensity values corresponding to all kidney tubule profiles.

[0143] By matching the PAS staining image and the multiple fluorescence image, the distal tubule and the proximal tubule in the PAS staining image can be quickly identified, greatly saving the labor cost and improving the efficiency. Moreover, by using the multiple fluorescence, not only the normal tubule segments in the PAS image can be identified, but also the diseased tubule segments can be identified, so the trained model can identify not only the normal distal tubule and proximal tubule, but also the diseased distal tubule and proximal tubule.

[0144] A training sample set is constructed based on the PAS staining image in which the proximal tubule and the distal tubule are identified. That is, the PAS staining image of the case slice of an individual is taken as the input data of a sample, and the mask of the distal tubule and the proximal tubule in the PAS staining image is taken as the label to construct a sample, thereby constructing the training sample set.

[0145] A multi-type segmentation model is constructed, and the multi-type segmentation model is trained based on the constructed training sample set to obtain a trained distal-proximal tubule identification model.

[0146] Specifically, the constructed multi-type segmentation model includes:

[0147] An encoder is configured to perform layer-by-layer semantic feature extraction on the input image through multiple down-sampling layers;

[0148] An intermediate layer is configured to compress the semantic features extracted by the last down-sampling layer of the encoder, and transmit the compressed semantic features to a decoder;

[0149] A decoder is configured to decode the semantic features step by step through multiple up-sampling layers to obtain decoding features;

[0150] an output layer configured to output a segmentation result based on the decoded features;

[0151] The number of down-sampling layers of the encoder and the number of up-sampling layers of the encoder are the same and one-to-one corresponding; the attention module is connected between the down-sampling layer and the corresponding up-sampling layer, for attention extraction of the semantic features extracted by the down-sampling layer and transmission to the corresponding up-sampling layer.

[0152] In implementation, the multi-type segmentation model adopts an encoder-decoder symmetric structure, can obtain spatial information for image semantic segmentation, improves segmentation precision, and also adds an attention module in the skip link structure. The attention module can weight the feature map, enhance the weight of important features, and at the same time reduce the influence of irrelevant and secondary features, improve the ability to capture important details, and make the model pay more attention to the region to be segmented, improve the precision of segmentation.

[0153] In implementation, the first encoder extracts semantic features. The encoder includes four down-sampling units connected in sequence, gradually reduces the size of the feature map, and at the same time increases the number of channels of the feature map, so as to capture image features of different scales and improve the precision of the segmentation result.

[0154] In implementation, the down-sampling unit includes a convolution layer, an activation function and a max-pooling layer connected in sequence. The convolution layer is used for feature extraction, and each down-sampling unit usually includes two consecutive convolution operations; an activation function is connected after each convolution layer to increase the nonlinear expression ability of the model; finally, the image spatial dimension is reduced by half through the max-pooling layer for down-sampling, thereby reducing the calculation complexity and obtaining more abstract features.

[0155] The last down-sampling unit of the encoder is connected to the intermediate layer, which is used for compression and extraction of the semantic features output by the down-sampling unit, while maintaining the semantic information of the features. In implementation, the intermediate layer is a convolution layer.

[0156] In implementation, the output module is a convolution layer, which is used to generate the final segmentation result, and the number of channels of the output module corresponds to the segmentation types one-to-one. The present application is used to distinguish the proximal tubule and the distal tubule in the PAS staining image, so the segmentation types include three types: proximal tubule, distal tubule and others.

[0157] In implementation, the loss of the multi-type segmentation model is calculated by the following formula:

[0158]

[0159] wherein M represents the number of pixels of the sample, C represents the number of segmentation types, and N represents the number of samples in the current training batch, represents whether the jthpixel of the ithsample belongs to the kthtype, and is 1 if yes, otherwise 0, represents the probability of the jthpixel of the ithsample predicted by the model belonging to the kthtype.

[0160] In implementation, the parameters of the multi-type segmentation model are updated based on the training loss by using the gradient descent method until the model converges, and a trained multi-type segmentation model is obtained.

[0161] For a PAS-stained image to be identified, it can be input into the trained multi-type segmentation model to quickly and accurately identify the distal tubules and proximal tubules therein.

[0162] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium is a disk, an optical disk, a read-only memory, a random access memory, etc.

[0163] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A training method for a renal tubule identification model, characterized in that, Includes the following steps: Acquire PAS-stained images and multiplex fluorescence images of pathological sections from different individuals; the multiplex fluorescence images include a first fluorescence channel for labeling distal renal tubules and a second fluorescence channel for labeling proximal renal tubules; Identifying renal tubular contours in PAS-stained images based on a semantic segmentation model; The PAS-stained images and multiple fluorescence images are matched to identify whether the renal tubule contours in the PAS-stained images are proximal or distal renal tubules; a training sample set is constructed based on the PAS-stained images that identify proximal and distal renal tubules. Construct a multi-type segmentation model, and train the multi-type segmentation model based on the training sample set to obtain a trained proximal and distal renal tubule recognition model; The constructed multi-type segmentation models include: An encoder is used to extract semantic features from an input image layer by layer through multiple downsampling layers; The intermediate layer is used to compress the semantic features extracted by the last downsampling layer of the encoder and pass the compressed semantic features to the decoder. The decoder is used to decode semantic features step by step through multiple upsampling layers to obtain decoded features; The output layer is used to output segmentation results based on the decoded features; The number of downsampling layers and the number of upsampling layers in the encoder are the same and correspond one-to-one; an attention module is connected between the downsampling layer and the corresponding upsampling layer, which is used to extract the semantic features extracted by the downsampling layer and pass them to the corresponding upsampling layer. The matching of the PAS-stained image and the multiple fluorescence image identifies whether the renal tubular contour in the PAS-stained image is a proximal or distal renal tubule, including: Global matching of the PAS-stained image and the multiple fluorescence image is performed to obtain global matching point pairs and a global affine transformation matrix; For each available pathological strip in the PAS-stained image, the affine transformation matrix corresponding to each renal tubular contour within the available pathological strip is calculated based on the global matching point pair and the global affine transformation matrix corresponding to the available pathological strip. For each renal tubule contour in the available pathological strip, if the affine transformation matrix corresponding to the renal tubule contour is valid, the region corresponding to the renal tubule contour in the multiple fluorescence image is obtained based on the affine transformation matrix corresponding to the renal tubule contour; the intensity of the region corresponding to the renal tubule contour in the first fluorescence channel and the second fluorescence channel indicates whether the renal tubule contour is a distal renal tubule or a proximal renal tubule.

2. The training method for the proximal and distal renal tubule recognition model according to claim 1, characterized in that, The loss of the multi-type segmentation model is calculated using the following formula: Where M represents the number of pixels in the sample, C represents the number of segmentation types, and N represents the number of samples in the current training batch. This indicates whether the j-th pixel of the i-th sample belongs to the k-th type; a value of 1 indicates it belongs, and a value of 0 indicates it doesn't. This represents the probability predicted by the model that the j-th pixel of the i-th sample belongs to the k-th type.

3. The training method for the proximal and distal renal tubule recognition model according to claim 1, characterized in that, The intensity of the region corresponding to the renal tubule contour under the first and second fluorescence channels indicates whether the renal tubule contour is a distal or proximal tubule, including: The intensity values ​​of the region corresponding to the renal tubule contour are calculated in the first and second fluorescence channels of the multiple fluorescence image to obtain the first intensity value and the second intensity value, respectively. If the first intensity value exceeds the fifth threshold and the second intensity value does not exceed the sixth threshold, then the renal tubule profile is a distal renal tubule. If the first intensity value does not exceed the fifth threshold and the second intensity value exceeds the sixth threshold, then the renal tubule profile is a proximal renal tubule. If the first intensity value exceeds the fifth threshold and the second intensity value exceeds the sixth threshold, then: if the ratio of the first intensity value to the second intensity value exceeds the seventh threshold, then the renal tubule profile is a distal renal tubule; if the ratio of the second intensity value to the first intensity value exceeds the seventh threshold, then the renal tubule profile is a proximal renal tubule.

4. The training method for the proximal and distal renal tubule recognition model according to claim 1, characterized in that, Based on the globally matched point pairs and the global affine transformation matrix corresponding to the available pathological strip, the affine transformation matrix corresponding to each renal tubular contour within the available pathological strip is calculated, including: The available pathological strip area is divided into multiple image blocks, and the image block to which each renal tubule contour belongs is determined. Based on the global matching point pairs and the global affine transformation matrix, each image block is matched with multiple fluorescence images to obtain the local matching point pairs corresponding to each image block; For each renal tubule contour in the available pathological strip, the affine transformation matrix corresponding to the renal tubule contour is obtained based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs.

5. The training method for the proximal and distal renal tubule recognition model according to claim 4, characterized in that, Based on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, the affine transformation matrix corresponding to the renal tubule contour is obtained, including: Affine transformation is performed on the local matching point pairs corresponding to the image block to which the renal tubule contour belongs, to obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2. Based on the local matching points within the renal tubule contour of the image block to which the renal tubule contour belongs, affine transformation calculation is performed to obtain the affine transformation matrix M3 and the number of successfully registered point pairs N3. If the number of successfully registered point pairs N3 exceeds the third threshold, then the affine transformation matrix M3 is taken as the effective affine transformation matrix corresponding to the renal tubule contour; otherwise: If the number of successfully registered point pairs N2 exceeds the third threshold, then the affine transformation matrix M2 is taken as the valid affine transformation matrix corresponding to the renal tubule contour; otherwise, the affine transformation matrix M2 is taken as the invalid affine transformation matrix corresponding to the renal tubule contour.

6. The training method for the proximal and distal renal tubule recognition model according to claim 4, characterized in that, Based on the global matching point pairs and the global affine transformation matrix, image block matching is performed between each image block and the multiple fluorescence image to obtain the local matching point pairs corresponding to each image block, including: For each image patch, perform an affine transformation on the globally matched point pairs contained in the image patch to calculate the affine transformation matrix M1 and the number of successfully registered point pairs N1; If the number of successfully registered point pairs N1 exceeds the second threshold, the region corresponding to the image patch in the multifluorescence image is determined based on the affine transformation matrix M1; otherwise, the region corresponding to the image patch in the multifluorescence image is determined based on the global affine transformation matrix. Match the image patch with the corresponding region of the image patch in the multiple fluorescence image to obtain the local matching point corresponding to the image patch.

7. The training method for the proximal and distal renal tubule recognition model according to claim 6, characterized in that, The region corresponding to the image patch in the multiple fluorescence image is determined based on the affine transformation matrix M1, including: Based on the affine transformation matrix M1, the vertex coordinates of the image patch are transformed affinely to obtain the points corresponding to the vertices in the multiple fluorescence image. The region formed by the bounding box of the point corresponding to the vertex in the multifluorescence image is the region corresponding to the image patch in the multifluorescence image.

8. The training method for the proximal and distal renal tubule recognition model according to claim 1, characterized in that, The multiple fluorescence also includes DAPI fluorescence images and Opal480 fluorescence; Global matching of the PAS-stained image and the multiple fluorescence image yields globally matched point pairs, including: Morphological processing was performed on the superimposed images of DAPI fluorescence images and Opal480 fluorescence images; The processed superimposed image is subjected to multi-angle rotation transformation to obtain multiple rotated images; Match the PAS-stained image with the processed overlay image and each rotated image; The matching point pair of the image pair with the highest matching degree is selected as the global matching point pair; Perform affine transformation calculations on the globally matched point pairs to obtain the global affine transformation matrix.

Images