An adtkd-umod auxiliary diagnosis system based on deep learning

Abstract

The present application relates to a kind of ADTKD-UMOD based on deep learning auxiliary diagnosis system, belong to auxiliary diagnosis technical field, solve the problem that prior art lacks ADTKD-UMOD auxiliary diagnosis.System includes: data acquisition module, collects the staining image of renal tubular pathological section and multi-channel fluorescence staining image;Contour drawing module, based on the multi-channel fluorescence staining image of each individual renal tubular pathological section obtains the medullary loop ascending branch thick segment renal tubular contour in staining image;Training sample construction module, based on the staining graph that has outlined contour, constructs training sample set;Model training module, constructs deep learning network model, based on training sample set, carries out semi-supervised training to deep learning network model and obtains prediction model;Prediction module, the renal tubular contour map in the staining image of individual to be predicted is inputed into prediction model and obtains prediction result.Quick ADTKD-UMOD prediction is realized.

Description

Technical Field

[0001] This invention relates to the field of auxiliary diagnostic technology for ADTKD-UMOD, and more particularly to an auxiliary diagnostic system for ADTKD-UMOD based on deep learning. Background Technology

[0002] Autosomal dominant tubulointerstitial nephropathy (ADTKD) is one of the most common monogenic inherited kidney diseases. Patients typically experience kidney function impairment during adolescence, gradually progressing to end-stage renal disease. Timely diagnosis of this disease can help patients avoid excessive testing and treatment, facilitating future life planning for patients and their families. ADTKD-UMOD caused by mutations in the uromodulin (UMOD) gene is the most common subtype of ADTKD and currently the only type with characteristic pathological features. Specifically, the abnormal UMOD protein caused by the mutation accumulates in the cytoplasm of the epithelial cells of the thick ascending limb of the loop of Henle, forming diagnostically significant positive inclusion bodies, such as… Figure 2 As shown. Identification of inclusion bodies requires careful observation and identification of pathological slides by experienced pathologists. However, this pathological change is extremely subtle and its distribution is limited. Unless this disease is considered in advance, pathologists will not perform a full-slide search for characteristic lesions in every patient. Therefore, the rate of missed diagnosis and misdiagnosis is quite high, especially for patients without a family history.

[0003] Genetic testing is the gold standard for diagnosing ADTKD, but it is expensive and far from widespread. Using pathological slides for screening and diagnosis would not only compensate for the limited coverage of genetic testing and improve the diagnostic accuracy, but also allow for retrospective analysis of previous slides.

[0004] In recent years, with the development of artificial intelligence, deep learning algorithms have demonstrated powerful advantages in medical image recognition and classification. Automatic segmentation algorithms have been developed for the automatic identification and segmentation of normal kidney structures (such as glomeruli, renal tubules, and arterioles), as well as for the automatic classification and scoring of common diseases (such as IgA nephropathy and lupus nephritis). However, these studies focus on the automatic segmentation of common kidney structures (such as glomeruli, renal tubules, and arterioles), lacking research on the diagnosis of ADTKD-UMOD. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a deep learning-based auxiliary diagnostic system for ADTKD-UMOD, in order to solve the problem of the lack of existing systems that use intelligent algorithms to assist in the diagnosis of ADTKD-UMOD.

[0006] On one hand, embodiments of the present invention provide an ADTKD-UMOD-based assisted diagnostic system, comprising:

[0007] The data acquisition module is used to collect stained images and multi-channel fluorescence stained images of renal tubular pathological sections from multiple individuals;

[0008] The contouring module is used to obtain the contour of the thick ascending limb of the loop of Henle in the stained image of each individual based on the multichannel fluorescent staining image of the renal tubular pathological section.

[0009] Training sample construction module: used to construct a training sample set based on a staining map that outlines the renal tubules of the thick ascending limb of the loop of Henle; the training sample set includes labeled samples and unlabeled samples;

[0010] The model training module is used to construct a deep learning network model; based on the training sample set, the deep learning network model is semi-supervised to obtain the ADTKD-UMOD prediction model;

[0011] The prediction module is used to input the renal tubule contour map from the stained image of the individual to be predicted into the ADTKD-UMOD prediction model to obtain the ADTKD-UMOD prediction result of the individual to be predicted.

[0012] Based on further improvements to the above scheme, the deep learning network model includes:

[0013] The multi-mode enhancement module is used to perform first-mode enhancement and second-mode enhancement on unlabeled samples to obtain first-enhanced images and second-enhanced images, and input both the first-enhanced images and second-enhanced images into the first branch and the second branch; and to perform second-mode enhancement on labeled samples to obtain second-enhanced images, and input the second-enhanced images into the first branch and the second branch.

[0014] The first branch is used to extract features from the enhanced image using a Transformer network to obtain the first feature of the enhanced image, and to perform ADTKD-UMOD prediction based on the first feature;

[0015] The second branch is used to extract features from the enhanced image using a convolutional network to obtain the second feature of the enhanced image, and to perform ADTKD-UMOD prediction based on the second feature.

[0016] Based on the further improvements to the above scheme, the training loss of the deep learning network model is calculated using the following formula:

[0017] Loss = Loss1 + λLoss2

[0018] Where Loss1 represents the loss of labeled samples, Loss2 represents the loss of unlabeled samples, and λ represents the weight parameter.

[0019] Based on the further improvements to the above scheme, the loss of labeled samples is calculated using the following formula:

[0020]

[0021] Where, x i Let y represent the input image of the i-th labeled sample. i N represents the label of the i-th labeled sample. l This indicates the number of labeled samples in the current training batch. Let g represent the loss function. + (·) indicates a second mode enhancement, f T (·) represents the Transformer network, f C (·) represents a convolutional network.

[0022] Based on the further improvements to the above scheme, the loss of unlabeled samples is calculated using the following formula:

[0023]

[0024] Where, x j Let N represent the input image of the j-th unlabeled sample. u This indicates the number of unlabeled samples in the current training batch. Represents the loss function. Let g represent the category predicted by the second branch for the first augmented image of the j-th unlabeled sample. + (·) indicates a second mode enhancement, f T (·) represents the Transformer network, f C (·) denotes a convolutional network, I[·] denotes an indicator function, and p j Let represent the probability that the first augmented image of the j-th unlabeled sample belongs to each class predicted by the second branch, and <·,·> represent the similarity. This represents the filtering threshold.

[0025] Based on further improvements to the above scheme, the Transformer network of the first branch includes:

[0026] The image block coding module is used to segment the enhanced image and obtain the embedding representation corresponding to each image block;

[0027] Multiple Transformer blocks connected in sequence are used to perform multi-layer feature extraction on the embedded representation of image patches;

[0028] The spatial feature extraction module is used to extract spatial features from the enhanced image based on the residual convolutional network.

[0029] The feature interaction module is used to inject spatial features into the first Transformer block and extract multi-scale features after the Transformer block; for other Transformer blocks, it extracts and injects the multi-scale features extracted after the previous Transformer block into the Transformer block and extracts multi-scale features after the Transformer block; the multi-scale features extracted after the last Transformer block are the first features of the obtained enhanced image.

[0030] The first prediction module is used to perform ADTKD-UMOD prediction based on the first feature.

[0031] Based on further improvements to the above scheme, the multi-channel fluorescence staining image includes a DAPI fluorescence staining channel, an Opal 480 fluorescence staining channel, and an Opal 690 fluorescence staining channel;

[0032] Based on multichannel fluorescence staining images of renal tubular pathological sections for each individual, the outline of the thick ascending limb of the loop of Henle renal tubules in the stained images of each individual was obtained, including:

[0033] Image segmentation is performed on the stained image to obtain the renal tubule contours within the stained image;

[0034] Images from the DAPI fluorescence staining channel and the Opal 480 fluorescence staining channel were overlaid; the stained image and the overlaid image were matched to obtain matching point pairs for each effective colorimetric region in the stained image;

[0035] For each effective color region, the affine transformation matrix corresponding to each renal tubular contour in the effective color region is determined based on the matching point pairs of the effective color region.

[0036] For each renal tubule contour, if the affine transformation matrix corresponding to the renal tubule contour is valid, then based on the affine transformation matrix corresponding to the renal tubule contour, the region corresponding to the renal tubule contour in the multi-channel fluorescent staining image is obtained. Based on the intensity of the region corresponding to the renal tubule contour in the Opal 690 fluorescent staining channel in the multi-channel fluorescent staining image, it is determined whether the renal tubule contour is the contour of the thick ascending limb of the loop of Henle.

[0037] Based on further improvements to the above scheme, the stained image and the overlay image are matched to obtain matching point pairs for each effective color-developing region in the stained image, including:

[0038] Initial matching of the stained image and the overlay image is performed to obtain initial matching point pairs and an initial affine transformation matrix;

[0039] For each effective color-developing region in the stained image, the effective color-developing region is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair for each effective color-developing region.

[0040] Based on a further improvement to the above scheme, the effective color rendering region is matched with the overlay image based on the initial matching point pair and the initial affine transformation matrix to obtain a matching point pair for each effective color rendering region, including:

[0041] Perform affine transformation calculations on the initial matching point pairs of the effective color rendering area to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0042] If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 will be obtained as the affine transformation matrix corresponding to the effective color region; otherwise, the initial affine transformation matrix will be used as the affine transformation matrix corresponding to the effective color region.

[0043] A sliding window is used to slide within the effective color rendering area, and the region corresponding to the image in each sliding window in the superimposed image is determined based on the affine transformation matrix corresponding to the effective color rendering area.

[0044] Match the image within each sliding window with the corresponding region of that image in the overlay image to obtain the matching point pair of the effective color display area.

[0045] Based on the further improvement of the above scheme, the affine transformation matrix corresponding to each renal tubular contour in the effective colorimetric region is determined based on the matching point pairs of the effective colorimetric region, including:

[0046] Determine the image block to which each renal tubule contour belongs within the effective colorimetric region;

[0047] For each image patch, the affine transformation matrix corresponding to the image patch is obtained based on the matching point pairs within the image patch;

[0048] The affine transformation matrix corresponding to this image patch is the affine transformation matrix corresponding to the renal tubule contour belonging to this image patch.

[0049] Compared with existing technologies, this invention collects stained images and multi-channel fluorescent stained images of renal tubular pathological sections from multiple individuals through a data acquisition module. A contour delineation module determines the contour of the thick ascending limb of the loop of Henle tubule in each individual's stained image, thereby efficiently constructing a training sample set. A model training module then performs semi-supervised training on a constructed deep learning network model based on this training sample set to obtain the ADTKD-UMOD prediction model. This eliminates the need to label all samples, enabling accurate ADTKD-UMOD prediction even with limited labeled data. For the individual to be predicted, inputting the renal tubular contour from the stained image into the ADTKD-UMOD prediction model quickly yields the ADTKD-UMOD prediction result for that individual. Kidney pathology diagnosis is a challenging area in specialized pathology diagnosis, and rare genetic diseases, such as ADTKD-UMOD, have a high rate of missed diagnosis. This invention fills the gap in intelligent identification systems for easily missed kidney pathology lesions. It can be used not only for screening and diagnosis in primary hospitals but also in kidney pathology diagnostic centers to assist pathologists in their daily work, compensate for the insufficient coverage of gene testing, and improve diagnostic efficiency and accuracy.

[0050] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0051] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0052] Figure 1 This is a block diagram of the ADTKD-UMOD auxiliary diagnostic system based on deep learning, according to an embodiment of the present invention.

[0053] Figure 2 A schematic diagram of an inclusion body;

[0054] Figure 3 The staining image is an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram illustrating the process of determining the image block to which the renal tubule contour belongs, according to an embodiment of the present invention. Detailed Implementation

[0056] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0057] One specific embodiment of the present invention discloses an ADTKD-UMOD auxiliary diagnostic system based on deep learning, such as... Figure 1 As shown, it includes:

[0058] The data acquisition module is used to collect stained images and multi-channel fluorescence stained images of renal tubular pathological sections from multiple individuals;

[0059] The contouring module is used to obtain the contour of the thick ascending limb of the loop of Henle in the stained image of each individual based on the multichannel fluorescent staining image of the renal tubular pathological section.

[0060] Training sample construction module: used to construct a training sample set based on a staining map that outlines the renal tubules of the thick ascending limb of the loop of Henle; the training sample set includes labeled samples and unlabeled samples;

[0061] The model training module is used to construct a deep learning network model; based on the training sample set, the deep learning network model is semi-supervised to obtain the ADTKD-UMOD prediction model;

[0062] The prediction module is used to input the renal tubule contour map from the stained image of the individual to be predicted into the ADTKD-UMOD prediction model to obtain the ADTKD-UMOD prediction result of the individual to be predicted.

[0063] Directly annotating UMOD inclusion bodies in stained images pixel-by-pixel under high-power objectives is extremely time-consuming. Furthermore, while staining can reveal the brush border, the difference in cytoplasmic eosinophilicity between the distal and proximal tubules is easily affected by staining batches or tissue fixation quality under conventional light microscopy, leading to confusion among annotators due to color differences and making it difficult to distinguish segments, such as the thick ascending limb of the loop of Henle. Therefore, this invention uses a contour delineation module to first determine the contour of the thick ascending limb of the loop of Henle tubule in the stained image, and then constructs a training sample set, improving the efficiency and accuracy of sample construction.

[0064] During implementation, multiple individuals include those with ADTKD-UMOD and those without ADTKD-UMOD.

[0065] In practice, the stained images are PAS stained images. Since PAS staining is less time-consuming and less costly than fluorescent staining, stained images are typically used to train neural network models. A thumbnail of the stained image is shown below. Figure 3 As shown, the black box marks one of the colored areas.

[0066] Compared with existing technologies, the deep learning-based ADTKD-UMOD auxiliary diagnostic system provided in this embodiment collects stained images and multi-channel fluorescent stained images of renal tubular pathological sections from multiple individuals through a data acquisition module. A contour delineation module determines the contour of the thick ascending limb of the loop of Henle tubule in each individual's stained image, thereby efficiently constructing a training sample set. A model training module then performs semi-supervised training on the constructed deep learning network model based on this training sample set to obtain the ADTKD-UMOD prediction model. This eliminates the need to label all samples, enabling accurate ADTKD-UMOD prediction even with limited labeled data. For the individual to be predicted, the renal tubular contour map from the stained image is input into the ADTKD-UMOD prediction model to quickly obtain the ADTKD-UMOD prediction result for that individual. Kidney pathology diagnosis is a challenging area in specialized pathology diagnosis, and rare genetic diseases, such as ADTKD-UMOD, have a high rate of missed diagnosis. This invention fills the gap in intelligent identification systems for easily missed kidney pathology lesions. It can be used not only for screening and diagnosis in primary hospitals but also in kidney pathology diagnostic centers to assist pathologists in their daily work, compensate for the insufficient coverage of gene testing, and improve diagnostic efficiency and accuracy.

[0067] Since DAPI fluorescence staining and Opal 480 fluorescence staining can visualize cell nuclei and epithelial cells, which is helpful for registration, and Opal 690 can mark the thick ascending limb of the loop of Henle, the multichannel fluorescence staining images include the DAPI fluorescence staining channel, the Opal 480 fluorescence staining channel, and the Opal 690 fluorescence staining channel.

[0068] Specifically, the contouring module obtains the contour of the thick ascending limb of the loop of Henle in the stained image of each individual's renal tubular pathological section using the following method based on the multichannel fluorescence staining image:

[0069] S1. Perform image segmentation on the stained image to obtain the renal tubule contour in the stained image;

[0070] S2. Overlay the images from the DAPI fluorescence staining channel and the Opal 480 fluorescence staining channel; match the stained image and the overlaid image to obtain matching point pairs for each effective color development area in the stained image;

[0071] S3. For each effective coloring region, determine the affine transformation matrix corresponding to each renal tubular contour in the effective coloring region based on the matching point pairs of the effective coloring region.

[0072] S4. For each renal tubule contour, if the affine transformation matrix corresponding to the renal tubule contour is valid, then based on the affine transformation matrix corresponding to the renal tubule contour, the region corresponding to the renal tubule contour in the multi-channel fluorescent staining image is obtained. Based on the intensity of the region corresponding to the renal tubule contour in the Opal 690 fluorescent staining channel in the multi-channel fluorescent staining image, it is determined whether the renal tubule contour is the contour of the thick ascending limb of the loop of Henle.

[0073] In practice, existing kidney organ image segmentation methods can be used to segment the stained image to obtain the renal tubule contours in the stained image.

[0074] Images from the DAPI fluorescence staining channel (Opal 480 fluorescence staining channel) are overlaid to obtain an overlaid image. Then, the stained image and the overlaid image are matched to obtain matching point pairs for each effective colorimetric region in the stained image, specifically including:

[0075] S21. Perform initial matching on the stained image and the superimposed image to obtain initial matching point pairs and initial affine transformation matrix;

[0076] S22. For each effective color-developing region in the stained image, the effective color-developing region is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair for each effective color-developing region.

[0077] Specifically, the initial matching of the stained image and the overlay image to obtain the initial matching point pairs and the initial affine transformation matrix includes the following steps:

[0078] S211. Perform morphological processing on the superimposed images;

[0079] S212. Perform multi-angle rotation transformation on the processed superimposed image to obtain multiple rotated images;

[0080] S213. Match the stained image with the processed superimposed image and the rotated image respectively, and take the matching point pair of the image pair with the highest matching degree as the initial matching point pair.

[0081] S214. Calculate the affine transformation matrix between the initial matching point pairs to obtain the initial affine transformation matrix.

[0082] In practice, the superimposed images are first subjected to morphological processing to remove noise interference.

[0083] In practice, morphological processing includes background removal, erosion, and dilation. For example, after sorting the pixel intensity values ​​in the overlay image from largest to smallest, the top 30% of pixel intensity values ​​are taken as valid signals, while the rest are considered background, and their corresponding pixel intensity values ​​are set to background intensity values, such as 0, thereby removing some noise interference. The overlay image with the background removed is first eroded and then dilated; then, the area of ​​each colored region is reduced by a factor of ten, and the maximum value is taken as a threshold; colored regions with an area smaller than the threshold are removed, retaining significant large areas, thereby further eliminating interference from noise and small fragmented areas. Erosion and dilation can adopt existing erosion and dilation methods in image processing.

[0084] During implementation, the processed superimposed image is rotated at multiple angles to obtain multiple rotated images. For example, the processed superimposed image is rotated 90 degrees clockwise and 90 degrees counterclockwise to obtain the corresponding rotated images.

[0085] During implementation, existing image matching algorithms are used to match the stained image with the processed overlay image, and to match the stained image with each rotated image. For example, the Loftr matching algorithm is used for image matching, finding matching point pairs in the image pairs. The number of matching point pairs represents the matching degree; the more pairs, the higher the matching degree. The matching point pairs of the image pairs with the highest matching degree are selected as the initial matching point pairs.

[0086] It should be noted that if the colorized image and the overlaid image are rotated 90 degrees clockwise to achieve the highest matching degree, the coordinates of the matching points in the rotated image need to be converted to the coordinates of the original image. The affine transformation matrix between the initial matching point pairs is then calculated to obtain the initial affine transformation matrix.

[0087] The initial matching is a coarse matching of the global image. Based on the initial matching point pairs and the initial affine transformation matrix obtained from the initial matching, further fine-grained matching is performed.

[0088] Fine-grained matching is first performed within the stained areas. Therefore, it is necessary to first determine which stained areas are valid. For each stained area in the stained image, if the stained area has an initial matching point, then the stained area is a valid stained area.

[0089] Specifically, for each effective color-developing region in the stained image, the effective color-developing region is matched with the overlaid image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair for each effective color-developing region, including:

[0090] S221. Perform affine transformation calculation on the initial matching point pairs of the effective color display area to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1.

[0091] S222. If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 will be obtained as the affine transformation matrix corresponding to the effective color region; otherwise, the initial affine transformation matrix will be used as the affine transformation matrix corresponding to the effective color region.

[0092] S223. A sliding window is used to slide within the effective color rendering area, and the region corresponding to the image in each sliding window in the superimposed image is determined based on the affine transformation matrix corresponding to the effective color rendering area.

[0093] S224. Match the image within each sliding window with the corresponding region of that image in the overlay image to obtain the effective color display region matching point pair.

[0094] During implementation, for a valid color rendering area, the initial matching point pairs in the valid color rendering area are first subjected to affine transformation calculation to obtain the corresponding affine transformation matrix M1 and the number of successfully registered point pairs N1.

[0095] In practice, the affine transformation calculation can be performed using OpenCV's estimateAffine2D function, which returns the number of successfully registered point pairs.

[0096] If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 calculated from the initial matched point pairs within the effective color rendering region is used as the affine transformation matrix corresponding to that effective color rendering region, making subsequent matching more accurate. If the number of successfully registered point pairs N1 does not exceed the second threshold, the initial affine transformation matrix is ​​used as the affine transformation matrix corresponding to that effective color rendering region. In implementation, the second threshold is set according to the matching accuracy requirements.

[0097] Due to the displacement and deformation of the pathological strips, registration shift occurs. Therefore, this invention employs a strip-based matching method to resolve registration shift. In implementation, a non-overlapping sliding window is used, sliding within each effective color rendering area. For each image within the sliding window, the corresponding region in the overlay image is determined based on the affine transformation matrix corresponding to that effective color rendering area.

[0098] Specifically, the region corresponding to each image within a sliding window in the overlay image is determined using the following method:

[0099] S2231. Based on the affine transformation matrix corresponding to the effective color rendering area, perform an affine transformation on the vertex coordinates of the image within the sliding window to obtain the point corresponding to the vertex in the superimposed image.

[0100] S2232. The region formed by the bounding box of the point corresponding to the vertex in the overlay image is the region corresponding to the image in the overlay image within the sliding window.

[0101] During implementation, the coordinates of the four vertices (top left, top right, bottom right, and bottom left) of the image within the sliding window are used to calculate the coordinates of their corresponding points in the overlay image based on the affine transformation matrix corresponding to the effective color display area. The region formed by the bounding boxes of these corresponding points is the region of the image within the sliding window that corresponds to the image in the overlay image.

[0102] The region corresponding to the image within the sliding window in the overlay image is obtained. The image within the sliding window and its corresponding region in the overlay image are matched (for example, image matching is performed using the Loftr matching algorithm) to obtain the matching point pairs corresponding to the images within the sliding window. All the matching point pairs corresponding to the images within the sliding windows in the effective color display area constitute the matching point pairs of the effective color display area.

[0103] By using a sliding window to perform fine-grained matching of the effective color rendering area, the registration accuracy is improved.

[0104] After obtaining the matching point pairs for each effective colorimetric region, the affine transformation matrix corresponding to each renal tubular contour in that effective colorimetric region is determined based on the matching point pairs, specifically including:

[0105] S31. Determine the image block to which each renal tubule outline belongs in the effective color rendering area;

[0106] S32. For each image block, obtain the affine transformation matrix corresponding to the image block based on the matching point pairs within the image block;

[0107] S32. The affine transformation matrix corresponding to the image block is the affine transformation matrix corresponding to the renal tubule contour belonging to the image block.

[0108] Specifically, the image patch to which each renal tubule contour belongs within the effective colorimetric region is determined using the following method:

[0109] S311. For each renal tubule contour in the effective color display area that does not belong to a specific image block; use the outer border of the renal tubule contour as the current area;

[0110] S312. The current window is obtained with the top left corner of the current region as the top left corner of the window and the preset length as the side length. If there is a renal tubule outline outside the current region and whose image block to which it belongs in the current window, the common outer border of the renal tubules in the current region and the renal tubule outlines in the current window but outside the current region and whose image block to which it belongs in the current window is defined as the current region. If not, the current region remains unchanged.

[0111] S313. The current window is obtained with the upper right corner of the current region as the upper right corner of the window and the preset length as the side length. If there is a renal tubule outline outside the current region in the current window and the image block to which it belongs is not determined, then the common outer border of the renal tubule in the current region and the renal tubule outline in the current window but outside the current region and the image block to which it belongs is not determined is taken as the current region. If not, the current region is kept unchanged.

[0112] S314. The current window is obtained with the lower right corner of the current region as the lower right corner of the window and the preset length as the side length. If there is a renal tubule outline outside the current region and not belonging to a defined image block in the current window, the common outer border of the renal tubules in the current region and the renal tubule outlines in the current window but outside the current region and not belonging to a defined image block is taken as the current region. If not, the current region is kept unchanged.

[0113] S315. The current window is obtained by taking the lower left corner of the current region as the lower left corner of the window and the preset length as the side length; the region of the current window is taken as an image block; all image blocks in the current window that do not have a defined affiliation, such as the renal tubule outline, belong to this image block.

[0114] by Figure 4 Let's take an example to illustrate. Figure 4 In (a), the circle represents the renal tubule outline of an image block whose corresponding image block is not yet determined, and the solid frame represents the outer border of the renal tubule outline. First, the outer border of one of the renal tubule outlines of an image block whose corresponding image block is not yet determined is taken as the current region, i.e., the solid frame in the figure is the current region. The upper left corner of the current region is taken as the upper left corner of the window, and the preset length is used as the side length to obtain the current window. Figure 4 In (a), if there is a renal tubule outline within the current window that is not in the current region (solid box) and does not belong to a defined image block, then the common outer border of the renal tubule outlines of these two renal tubules is taken as the current region. For example... Figure 4 The solid-line box in (b).

[0115] Then with Figure 4 The top right corner of the solid-line frame (current region) in (b) is used as the top right corner vertex of the window. The window is drawn with a preset length equal to the side length to obtain the current window. Figure 4 The dashed box in (b). If there are no renal tubular outlines outside the current region and whose belonging image block is not determined within the current window, then the current region remains unchanged. The current window is obtained by taking the lower right corner of the current region as the lower right corner of the window and the preset length as the side length, as shown below. Figure 4 The dashed box in (c). There are still no renal tubular outlines outside the current region or belonging to undetermined image blocks within the current window. Continue to use the lower left corner vertex of the current region as the lower left corner of the window and a preset length as the side length to obtain the current window, as shown below. Figure 4The dashed box of (d) represents the area of ​​the current window as an image block; all image blocks within the current window that do not have a defined affiliation, including the renal tubule outline, belong to this image block.

[0116] The process of determining the image patch to which the renal tubule contour belongs does not involve uniformly dividing the pathological strip into image patches of equal size. Instead, it determines the size based on the density of the renal tubules. The determined image patch sizes differ, with the image patch corresponding to a region with dense renal tubule contours being larger than that corresponding to a region with sparse renal tubule contours. Treating a region with dense renal tubule contours as a single image patch reduces the number of image patches without compromising registration quality. Regions without renal tubule shapes do not require determining an image patch. The number of determined image patches is less than that obtained through uniform division, thereby improving computational efficiency.

[0117] After determining the image block to which each renal tubule contour belongs in the effective color rendering area, for each image block, the affine transformation matrix corresponding to the image block is obtained based on the matching point pairs within the image block, and then the region corresponding to the renal tubule contour belonging to the image block in the superimposed image is determined based on the obtained affine transformation matrix.

[0118] Specifically, the affine transformation matrix corresponding to each image patch is obtained based on the matching point pairs within each image patch, including:

[0119] S321. Perform affine transformation calculation on the matching point pairs within the renal tubule contour of the image block to obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2.

[0120] S322. Determine the region corresponding to the image patch in the superimposed image based on the affine transformation matrix M2;

[0121] S323. Based on the image block and the corresponding region of the image block in the superimposed image, obtain the affine transformation matrix corresponding to the image block.

[0122] In practice, for an image patch, affine transformation calculation is first performed on the matching point pairs located in the renal tubule contour within the matching point pairs in the image patch. The affine transformation calculation can use the aforementioned estimateAffine2D function to obtain the affine transformation matrix M2 and the number of successfully registered point pairs N2.

[0123] It should be noted that if the number of matching point pairs located within the renal tubule contour of the image patch is less than a preset threshold, such as less than 50 pairs, then the point pairs closest to the renal tubule contour are selected from the matching points located in the image patch but not within the renal tubule contour as matching point pairs located within the renal tubule contour of the image patch, in order to make up the number of matching point pairs located within the renal tubule contour of the image patch, so as to facilitate more accurate matching.

[0124] Then, the region corresponding to the image patch in the overlay image is determined based on the affine transformation matrix M2.

[0125] In implementation, an affine transformation is performed on the vertex coordinates of the image patch based on the affine transformation matrix M2 to obtain the corresponding point in the overlay image; the region formed by the bounding box of the corresponding point of the image patch vertex in the overlay image is the region corresponding to the image patch in the overlay image. For details, please refer to steps S2231 to S2232.

[0126] Then, based on the image patch and its corresponding region in the overlay image, the affine transformation matrix corresponding to the image patch is obtained, specifically including:

[0127] S3231. Match the image block with the corresponding region in the overlay image to obtain a matching point pair C2; perform affine transformation calculation on the matching point pair C2 to obtain the affine transformation matrix M3 and the number of successfully registered point pairs N3.

[0128] S3232. If the number of matching point pairs within the renal tubular contour in matching point pair C2 exceeds the third threshold (e.g., 40 pairs), then

[0129] Affine transformation calculation is performed on the matching point pairs located within the renal tubule contour in the matching point pair C2 to obtain the affine transformation matrix M4 and the number of successfully registered point pairs N4.

[0130] If the number of successfully registered point pairs N4 exceeds the fourth threshold (e.g., 20 pairs), then the affine transformation matrix M4 is used as the valid affine transformation matrix corresponding to the image patch; otherwise, if the number of successfully registered point pairs N3 exceeds the fourth threshold, then the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image patch; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image patch.

[0131] If the number of matching point pairs within the renal tubule contour in matching point pair C2 does not exceed the third threshold (40 pairs), then: if the number of successfully registered point pairs N3 exceeds the fourth threshold, then the affine transformation matrix M3 is taken as the valid affine transformation matrix corresponding to the image patch; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image patch.

[0132] In implementation, the matching algorithm in step S3231 adopts the above-mentioned loftr algorithm to obtain the matching point pair C2. The aforementioned estimateAffine2D function is used to perform affine transformation calculation on the matching point pair C2 to obtain the affine transformation matrix M3 and the number of successfully registered point pairs N3.

[0133] If the number of matching point pairs within the renal tubule contour in matching point pair C2 exceeds the third threshold (e.g., 40 pairs), an affine transformation is performed on these matching point pairs to obtain the affine transformation matrix M4 and the number of successfully registered point pairs N4. Then, it is determined whether the number of successfully registered point pairs N4 exceeds the fourth threshold (e.g., 20 pairs). If it does, the affine transformation matrix M4 is used as the affine transformation matrix corresponding to the image patch, achieving more accurate matching. Therefore, the affine transformation matrix M4 is used as the affine transformation matrix corresponding to the image patch. If the number of successfully registered point pairs N4 does not exceed the fourth threshold, it is checked whether the number of successfully registered point pairs N3 exceeds the fourth threshold. If it does, the affine transformation matrix M3 is used as the affine transformation matrix corresponding to the image patch. If N3 also does not exceed the fourth threshold, the affine transformation matrix M3 is used as the invalid affine transformation matrix corresponding to the image patch.

[0134] If the number of matching point pairs within the renal tubule contour in matching point pair C2 does not exceed the third threshold, then check if the number of successfully registered point pairs N3 exceeds the fourth threshold. If it does, then use the affine transformation matrix M3 as the affine transformation matrix corresponding to the image patch. If N3 also does not exceed the fourth threshold, then use the affine transformation matrix M3 as the invalid affine transformation matrix corresponding to the image patch.

[0135] When the number of successfully registered point pairs N3 is small, the calculation will have a large error. Therefore, M3 is the invalid affine transformation matrix corresponding to the image patch.

[0136] During implementation, the third and fourth thresholds are set according to the matching accuracy requirements.

[0137] The affine transformation matrix corresponding to this image patch is the affine transformation matrix corresponding to the contours of all renal tubules belonging to this image patch.

[0138] By determining the image patch corresponding to the renal tubule contour, the affine transformation matrix corresponding to the image patch is calculated, thus eliminating the need to calculate the affine transformation matrix for each renal tubule contour individually, thereby improving computational efficiency.

[0139] For each renal tubule contour, if the affine transformation matrix corresponding to the contour is valid, the region corresponding to that contour in the multi-channel fluorescence staining image is obtained based on the affine transformation matrix. In practice, the coordinates of the four vertices of the outer bounding box of the renal tubule contour are affinely transformed according to the affine transformation matrix to obtain the points corresponding to those vertices in the multi-channel fluorescence staining image; the region formed by the bounding box of the points corresponding to those vertices in the multi-channel fluorescence staining image is the region corresponding to that renal tubule contour in the multi-channel fluorescence staining image. For details, refer to steps S2231 to S2232.

[0140] If the intensity of the Opal 690 fluorescence staining channel exceeds a preset threshold within the corresponding region of the renal tubule outline in the multichannel fluorescence staining image, then it represents the outline of the thick segment of the ascending limb of the loop of Henle.

[0141] After obtaining the outline of the thick ascending limb of the loop of Henle renal tubule in the stained image, a training sample set was constructed based on the stained image that outlined the thick ascending limb of the loop of Henle renal tubule.

[0142] During implementation, a professional physician identifies and judges the presence of inclusion bodies (i.e., whether ADTKD-UMOD is present) on the outline of the thick ascending limb of the loop of Henle tubule. Since fully supervised annotation in pathological slide images with millions of cells is extremely costly, partial annotation of stained images is performed, specifically indicating the presence of inclusion bodies in the thick ascending limb of the loop of Henle. Images of the thick ascending limb of the loop of Henle tubule and other tubule outlines not in the thick ascending limb of the loop of Henle are extracted from the stained images and used as input images to construct a training sample set. The sample set includes labeled and unlabeled samples. Labeled samples are those with the label indicating the presence or absence of inclusion bodies in the tubule outline. Unlabeled samples are those without labels. A deep learning network model is constructed; based on the training sample set, the deep learning network model is semi-supervised trained to obtain the ADTKD-UMOD prediction model.

[0143] Specifically, the deep learning network models constructed include:

[0144] The multi-mode enhancement module is used to perform first-mode enhancement and second-mode enhancement on unlabeled samples to obtain first-enhanced images and second-enhanced images, and input both the first-enhanced images and second-enhanced images into the first branch and the second branch; and to perform second-mode enhancement on labeled samples to obtain second-enhanced images, and input the second-enhanced images into the first branch and the second branch.

[0145] The first branch is used to extract features from the enhanced image using a Transformer network to obtain the first feature of the enhanced image, and to perform ADTKD-UMOD prediction based on the first feature;

[0146] The second branch is used to extract features from the enhanced image using a convolutional network, obtaining the second feature of the enhanced image, and then performing ADTKD-UMOD prediction based on the second feature.

[0147] In implementation, the first enhancement mode is a weak enhancement, such as image flipping and translation. The second enhancement mode is a strong enhancement mode, and the enhancement strategy can be CTAugment or RandAugment enhancement strategies.

[0148] For unlabeled samples, first-mode enhancement and second-mode enhancement are performed respectively to obtain the corresponding first-enhanced image and second-enhanced image. For labeled samples, only the sample image is enhanced using the second-mode enhancement to obtain the corresponding second-enhanced image.

[0149] The resulting enhanced images are input into the first branch and the second branch, respectively.

[0150] Specifically, the first branch uses a Transformer network to extract features from the enhanced image, obtaining the first feature of the enhanced image. The Transformer network of the first branch includes:

[0151] The image block coding module is used to segment the enhanced image and obtain the embedding representation corresponding to each image block;

[0152] Multiple Transformer blocks connected in sequence are used to perform multi-layer feature extraction on the embedded representation of image patches;

[0153] The spatial feature extraction module is used to extract spatial features from the enhanced image based on the residual convolutional network.

[0154] The feature interaction module is used to inject spatial features into the first Transformer block and extract multi-scale features after the Transformer block; for other Transformer blocks, it extracts and injects the multi-scale features extracted after the previous Transformer block into the Transformer block and extracts multi-scale features after the Transformer block; the multi-scale features extracted after the last Transformer block are the first features of the obtained enhanced image.

[0155] The first prediction module is used to perform ADTKD-UMOD prediction based on the first feature.

[0156] In implementation, the image block coding module first segments the enhanced image into multiple image blocks, for example, dividing a 256×256 image into multiple 16×16 image blocks. Each image block is flattened and projected onto the D dimension, and the resulting D vector is added to the positional code of the image block to obtain the embedding representation of each image block.

[0157] The embedding representation of each image patch is obtained by feature extraction through multiple sequentially connected Transformer blocks.

[0158] To supplement more local image information, spatial features are extracted through a spatial feature extraction module and injected into the Transformer block through a feature interaction module to improve model performance.

[0159] Specifically, the spatial feature extraction module uses a residual convolutional network to extract multi-scale features from the enhanced image to obtain spatial features. For example, ResNet can be used as the residual convolutional network. In implementation, multiple feature maps of different resolutions are extracted using the residual convolutional network, and these feature maps are flattened and concatenated to obtain the spatial features, represented as follows:

[0160] Specifically, the feature interaction module injects spatial features into the i-th Transformer block using the following formula:

[0161]

[0162] Where, γ i Let represent the learnable parameters corresponding to the i-th Transformer block, Attention(·) represent the attention mechanism, and LN(·) represent layer normalization. When i>1, This represents the multi-scale features extracted after the (i-1)th Transformer block, where i=1. Represents spatial characteristics; if i = 1, This indicates an embedding representation; if i > 1, This represents the output feature of the (i-1)th Transformer block; This represents the input features of the i-th Transformer block.

[0163] During implementation, spatial features are injected into the Transformer block through an attention mechanism.

[0164] Specifically, the feature interaction module extracts multi-scale features after the i-th Transformer block using the following formula:

[0165]

[0166] in, Let represent the multi-scale features extracted after the (i-1)th Transformer block, Attention(·) denotes the attention mechanism, and LN(·) denotes layer normalization. This represents the output feature of the i-th Transformer block. FFN(·) represents intermediate parameters, and FFN(·) represents a feedforward neural network module. This represents the multi-scale features extracted after the i-th Transformer block.

[0167] During implementation, multi-scale features are extracted after the i-th Transformer block. After that, if there is an (i+1)th Transformer block, then... Inject it as part of the input features of the (i+1)th Transformer block; otherwise, As the first feature for enhancing the image.

[0168] The first prediction module performs ADTKD-UMOD prediction based on the first feature. In implementation, the first prediction module can use an existing classification head.

[0169] The second branch uses a convolutional network to extract features from the enhanced image, obtaining a second feature of the enhanced image, and then performs ADTKD-UMOD prediction based on the second feature. The second branch can use existing convolutional network structures.

[0170] In practice, in order to integrate the inductive bias of the convolutional network into the Transformer network and enhance the spatial information of the convolutional network for knowledge sharing, the deep learning network model also includes an information interaction module to exchange information on the features extracted by the convolutional network and the Transformer network.

[0171] Specifically, the information interaction module uses the following methods to exchange information between the two branches:

[0172]

[0173] in, M represents the feature of the j-th image patch output by the i-th Transformer block in the Transformer network. C,j Let LN(·) represent the features of the region corresponding to the j-th image patch in the feature map of the convolutional network. LN(·) represents layer normalization, pooling(·) represents downsampling, align(·) represents mapping features to the same dimensional space, BN(·) represents batch normalization, and upsample(·) represents upsampling.

[0174] It should be noted that the number of information interactions corresponds to the number of Transformer blocks. That is, the i-th information interaction fuses the features output by the i-th Transformer block with the features extracted by the convolutional network to become the features output by the i-th Transformer block, and adds the features extracted by the convolutional network to the features extracted by the i-th Transformer block to become the features extracted by the convolutional network after the i-th information interaction.

[0175] After the first feature extraction branch and the second feature branch extract the first feature and the second feature, the prediction module predicts whether the sample has ADTKD-UMOD based on the first feature and the second feature, respectively.

[0176] The loss is calculated based on the prediction results of the prediction module, and the parameters of the deep learning network model are updated.

[0177] Specifically, the training loss of the deep learning network model is calculated using the following formula:

[0178] Loss = Loss1 + λLoss2

[0179] Where Loss1 represents the loss of labeled samples, Loss2 represents the loss of unlabeled samples, and λ represents the weight parameter.

[0180] For labeled samples, the loss is calculated based on the prediction results based on the first feature, the prediction results based on the second feature, and the label information.

[0181] Specifically, the loss of labeled samples is calculated using the following formula:

[0182]

[0183] Where, x i Let y represent the input image of the i-th labeled sample. i N represents the label of the i-th labeled sample. l This indicates the number of labeled samples in the current training batch. Let g represent the loss function. + (·) indicates a second mode enhancement, f T (·) represents the Transformer network, f C (·) represents a convolutional network.

[0184] The labeled samples are only enhanced in the second mode. Therefore, the loss is calculated by combining the prediction results of the second enhanced image obtained through two branches with the loss calculated from the sample label. Existing loss calculation functions can be used, such as the cross-entropy loss function.

[0185] For unlabeled samples, since there is no label information, unsupervised loss needs to be calculated. The input image of the unlabeled sample is enhanced in two ways. Since the weak enhancement retains more information of the original image, the prediction result of the second branch on the first enhanced image is used as the pseudo-label, and the pseudo-label is used for training.

[0186] Specifically, the loss for unlabeled samples is calculated using the following formula:

[0187]

[0188] Where, x j Let N represent the input image of the j-th unlabeled sample. uThis indicates the number of unlabeled samples in the current training batch. Represents the loss function. Let g represent the category predicted by the second branch for the first augmented image of the j-th unlabeled sample. + (·) indicates a second mode enhancement, f T (·) represents the Transformer network, f C (·) denotes a convolutional network, I[·] denotes an indicator function, and p j Let represent the probability that the first augmented image of the j-th unlabeled sample belongs to each class after the second branch prediction, where <·,·> represents the similarity and t represents the filtering threshold.

[0189] It should be noted that the sample labels are one-hot encoded vectors, and the category predicted by the second branch for the first augmented image of the i-th unlabeled sample. It is the one-hot encoded vector of the class corresponding to the highest predicted probability. The probability p of the first augmented image of the i-th unlabeled sample belonging to each class predicted by the second branch. j A vector consisting of the predicted probabilities of belonging to each category. Filter threshold. This is to enhance the reliability of pseudo-tags; only those with a similarity greater than [a certain value] are considered pseudo-tags. Only pseudo-labels are used for loss calculation, thereby filtering out uncertain pseudo-labels.

[0190] The parameters of the deep learning network model are trained based on the loss. When the change in loss reaches the preset accuracy or the number of iterations reaches the preset number, the training stops, and the ADTKD-UMOD prediction model is obtained.

[0191] For the individual to be predicted, inputting the outline of each renal tubule in the stained image into the obtained ADTKD-UMOD prediction model can quickly predict whether the renal tubule outline is ADTKD-UMOD.

[0192] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0193] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep learning-based ADTKD-UMOD auxiliary diagnostic system, characterized in that, include: The data acquisition module is used to collect PAS-stained images and multi-channel fluorescence-stained images of renal tubular pathological sections from multiple individuals. The contouring module is used to obtain the contour of the thick ascending limb of the loop of Henle in the PAS-stained image of each individual based on the multichannel fluorescent staining image of the renal tubular pathological section. The training sample construction module is used to construct a training sample set based on a staining map that outlines the renal tubules of the thick ascending limb of the loop of Henle; the training sample set includes labeled samples and unlabeled samples; The model training module is used to construct a deep learning network model; based on the training sample set, the deep learning network model is semi-supervised to obtain the ADTKD-UMOD prediction model; The prediction module is used to input the renal tubular contour map in the PAS staining image of the individual to be predicted into the ADTKD-UMOD prediction model to obtain the ADTKD-UMOD prediction result of the individual to be predicted. The multi-channel fluorescence staining image includes the DAPI fluorescence staining channel, the Opal 480 fluorescence staining channel, and the Opal 690 fluorescence staining channel; Based on multichannel fluorescence staining images of renal tubular pathological sections for each individual, the outline of the thick ascending limb of the loop of Henle renal tubules in the PAS staining images of each individual was obtained, including: Image segmentation was performed on the PAS-stained image to obtain the renal tubular contours in the PAS-stained image; The images of the DAPI fluorescence staining channel and the Opal 480 fluorescence staining channel were overlaid; the PAS staining image and the overlaid image were matched to obtain the matching point pairs of each effective color development area in the PAS staining image; For each effective color region, the affine transformation matrix corresponding to each renal tubular contour in the effective color region is determined based on the matching point pairs of the effective color region. For each renal tubule contour, if the affine transformation matrix corresponding to the renal tubule contour is valid, then based on the affine transformation matrix corresponding to the renal tubule contour, the region corresponding to the renal tubule contour in the multi-channel fluorescent staining image is obtained. Based on the intensity of the region corresponding to the renal tubule contour in the Opal 690 fluorescent staining channel in the multi-channel fluorescent staining image, it is determined whether the renal tubule contour is the contour of the thick ascending limb of the loop of Henle.

2. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 1, characterized in that, The deep learning network model includes: The multi-mode enhancement module is used to perform first-mode enhancement and second-mode enhancement on unlabeled samples to obtain first-enhanced images and second-enhanced images, and input both the first-enhanced images and second-enhanced images into the first branch and the second branch; and to perform second-mode enhancement on labeled samples to obtain second-enhanced images, and input the second-enhanced images into the first branch and the second branch. The first branch is used to extract features from the enhanced image using a Transformer network to obtain the first feature of the enhanced image, and to perform ADTKD-UMOD prediction based on the first feature; The second branch is used to extract features from the enhanced image using a convolutional network to obtain the second feature of the enhanced image, and to perform ADTKD-UMOD prediction based on the second feature.

3. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 2, characterized in that, The training loss of the deep learning network model is calculated using the following formula: in, This represents the loss of the labeled samples. This represents the loss of unlabeled samples. This represents the weighting parameter.

4. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 3, characterized in that, The loss of labeled samples is calculated using the following formula: in, This represents the input image of the i-th labeled sample. This represents the label of the i-th labeled sample. This indicates the number of labeled samples in the current training batch. Represents the loss function. This indicates an enhancement to the second mode. Represents the Transformer network. This represents a convolutional network.

5. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 3, characterized in that, The loss for unlabeled samples is calculated using the following formula: in, This represents the input image of the j-th unlabeled sample. This indicates the number of unlabeled samples in the current training batch. Represents the loss function. This represents the category predicted by the second branch for the first augmented image of the j-th unlabeled sample. This indicates an enhancement to the second mode. Represents the Transformer network. Represents a convolutional network. Represents the characteristic function, This represents the probability that the first augmented image of the j-th unlabeled sample belongs to any class after the second branch prediction. Indicates similarity. This represents the filtering threshold.

6. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 2, characterized in that, The Transformer network of the first branch includes: The image block coding module is used to segment the enhanced image and obtain the embedding representation corresponding to each image block; Multiple Transformer blocks connected in sequence are used to perform multi-layer feature extraction on the embedded representation of image patches; The spatial feature extraction module is used to extract spatial features from the enhanced image based on the residual convolutional network. The feature interaction module is used to inject spatial features into the first Transformer block and extract multi-scale features after the Transformer block; for other Transformer blocks, it extracts and injects the multi-scale features extracted after the previous Transformer block into the Transformer block and extracts multi-scale features after the Transformer block; the multi-scale features extracted after the last Transformer block are the first features of the obtained enhanced image. The first prediction module is used to perform ADTKD-UMOD prediction based on the first feature.

7. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 1, characterized in that, Matching the PAS-stained image with the overlay image yields matching point pairs for each effective colorimetric region in the PAS-stained image, including: Initial matching of the PAS-stained image and the superimposed image is performed to obtain initial matching point pairs and an initial affine transformation matrix; For each effective color region in the PAS staining image, the effective color region is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair for each effective color region.

8. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 7, characterized in that, Based on the initial matching point pairs and the initial affine transformation matrix, the effective color rendering region is matched with the overlaid image to obtain matching point pairs for each effective color rendering region, including: Perform affine transformation calculations on the initial matching point pairs of the effective color rendering area to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1; If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 will be obtained as the affine transformation matrix corresponding to the effective color region; otherwise, the initial affine transformation matrix will be used as the affine transformation matrix corresponding to the effective color region. A sliding window is used to slide within the effective color rendering area, and the region corresponding to the image in each sliding window in the superimposed image is determined based on the affine transformation matrix corresponding to the effective color rendering area. Match the image within each sliding window with the corresponding region of that image in the overlay image to obtain the matching point pair of the effective color display area.

9. The deep learning-based ADTKD-UMOD assisted diagnostic system according to claim 1, characterized in that, Based on the matching point pairs within the effective colorimetric region, the affine transformation matrix corresponding to each renal tubular contour within the effective colorimetric region is determined, including: Determine the image block to which each renal tubule contour belongs within the effective colorimetric region; For each image patch, the affine transformation matrix corresponding to the image patch is obtained based on the matching point pairs within the image patch; The affine transformation matrix corresponding to this image patch is the affine transformation matrix corresponding to the renal tubule contour belonging to this image patch.

Images