ADTKD-UMOD auxiliary diagnosis system based on deep learning

Abstract

The invention relates to an ADTKD-UMOD auxiliary diagnosis system based on deep learning, belongs to the technical field of auxiliary diagnosis, and solves the problem of lack of ADTKD-UMOD auxiliary diagnosis in the prior art. The system comprises a data acquisition module for collecting a staining image and a multi-channel fluorescent staining image of a renal tubule pathological section; the contour sketching module is used for obtaining a marrow loop rising branch thick section renal tubule contour in a staining image based on the multi-channel fluorescent staining image of the renal tubule pathological section of each individual; the training sample construction module is used for constructing a training sample set on the basis of the dyed image delineated with the outline; the model training module is used for constructing a deep learning network model and carrying out semi-supervised training on the deep learning network model based on the training sample set to obtain a prediction model; and the prediction module inputs the renal tubule profile diagram in the dyed image of the individual to be predicted into the prediction model to obtain a prediction result. And the rapid ADTKD-UMOD prediction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of auxiliary diagnosis of ADTKD-UMOD, and in particular to an auxiliary diagnosis system of ADTKD-UMOD based on deep learning. Background Art

[0002] Autosomal dominant tubulointerstitialkidney disease (ADTKD) is one of the most common monogenic inherited kidney diseases. Patients usually develop renal impairment in adolescence and gradually progress to end-stage renal disease. Timely diagnosis of this type of disease can help patients avoid excessive examinations and treatments, and facilitate future life planning for patients and their families. ADTKD-UMOD, caused by mutations in the uromodulin (UMOD) gene, is the most common subtype of ADTKD and is currently the only type of ADTKD with characteristic pathological manifestations. Specifically, the abnormal UMOD protein caused by the mutation aggregates in the cytoplasm of the renal tubular epithelial cells in the thick ascending limb of the loop of Henle to form positive staining inclusions with diagnostic significance, such as Figure 2 The identification of inclusion bodies requires careful observation and identification of pathological sections by experienced pathologists. However, these pathological changes are extremely subtle and have a limited distribution. Unless they are considered in advance, pathologists will not perform a full scan of each patient to search for characteristic lesions. This leads to a high rate of missed diagnosis and misdiagnosis, especially in patients without a family history.

[0003] Genetic testing is the gold standard for diagnosing ADTKD, but it is expensive and far from universally available. Using pathological biopsies for screening and diagnosis would not only address the limited coverage of genetic testing and improve the diagnostic rate, but also allow for retrospective analysis of previous biopsies.

[0004] In recent years, with the development of artificial intelligence, deep learning algorithms have demonstrated significant advantages in medical image recognition and classification. Automated segmentation algorithms have been developed for the automatic identification and segmentation of normal renal structures (such as glomeruli, tubules, and arterioles), as well as for the automatic classification and scoring of common diseases (such as IgA nephropathy and lupus nephritis). However, these studies have focused on the automated segmentation of common renal structures (such as glomeruli, tubules, and arterioles), with a lack of research on the diagnosis of ADTKD-UMOD. Summary of the Invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide an auxiliary diagnosis system for ADTKD-UMOD based on deep learning, so as to solve the problem of the existing lack of intelligent algorithms to assist in the diagnosis of ADTKD-UMOD.

[0006] In one aspect, an embodiment of the present invention provides an auxiliary diagnosis system for ADTKD-UMOD based on deep learning, comprising:

[0007] A data acquisition module, used to collect stained images and multi-channel fluorescent staining images of renal tubular pathological sections of multiple individuals;

[0008] A contour delineation module is used to obtain the contour of the thick ascending limb of the loop of Henle in the stained image of each individual based on the multi-channel fluorescent staining image of the renal tubule pathological section of each individual;

[0009] A training sample construction module is used to construct a training sample set based on a staining image outlining the outline of the renal tubules of the thick ascending limb of the loop of Henle; the training sample set includes labeled samples and unlabeled samples;

[0010] A model training module is used to construct a deep learning network model; based on the training sample set, the deep learning network model is semi-supervised trained to obtain an ADTKD-UMOD prediction model;

[0011] The prediction module is used to input the renal tubule outline image in the stained image of the individual to be predicted into the ADTKD-UMOD prediction model to obtain the ADTKD-UMOD prediction result of the individual to be predicted.

[0012] Based on the further improvement of the above solution, the deep learning network model includes:

[0013] A multi-mode enhancement module is configured to perform first-mode enhancement and second-mode enhancement on unlabeled samples to obtain first enhanced images and second enhanced images, and input the obtained first enhanced images and second enhanced images into the first branch and the second branch; perform second-mode enhancement on labeled samples to obtain second enhanced images, and input the obtained second enhanced images into the first branch and the second branch;

[0014] The first branch is used to extract features of the enhanced image using the Transformer network to obtain the first feature of the enhanced image, and perform ADTKD-UMOD prediction based on the first feature;

[0015] The second branch is used to extract features of the enhanced image using a convolutional network to obtain the second features of the enhanced image, and perform ADTKD-UMOD prediction based on the second features.

[0016] Based on the further improvement of the above scheme, the following formula is used to calculate the training loss of the deep learning network model:

[0017] Loss=Loss1+λLoss2

[0018] Among them, Loss1 represents the loss of labeled samples, Loss2 represents the loss of unlabeled samples, and λ represents the weight parameter.

[0019] Based on the further improvement of the above scheme, the loss of labeled samples is calculated using the following formula:

[0020]

[0021] Among them, x i represents the input image of the i-th labeled sample, y i represents the label of the i-th labeled sample, N l Indicates the number of labeled samples in the current training batch, represents the loss function, g + (·) indicates the second mode enhancement, f T (·) represents the Transformer network, f C (·) denotes a convolutional network.

[0022] Based on the further improvement of the above scheme, the following formula is used to calculate the loss of unlabeled samples:

[0023]

[0024] Among them, x j represents the input image of the jth unlabeled sample, N u Indicates the number of unlabeled samples in the current training batch, represents the loss function, represents the category of the first enhanced image of the jth unlabeled sample predicted by the second branch, g + (·) indicates the second mode enhancement, f T (·) represents the Transformer network, f C (·) represents the convolutional network, I[·] represents the characteristic function, and p j represents the probability that the first enhanced image of the jth unlabeled sample belongs to each category after being predicted by the second branch, <·,·> represents the similarity, Indicates the filtering threshold.

[0025] Based on a further improvement of the above solution, the Transformer network of the first branch includes:

[0026] The image block encoding module is used to segment the enhanced image and obtain the corresponding embedded representation of each image block;

[0027] Multiple Transformer blocks connected in sequence are used to perform multi-layer feature extraction on the embedded representation of image blocks;

[0028] The spatial feature extraction module is used to extract features of the enhanced image based on the residual convolutional network to obtain spatial features;

[0029] A feature interaction module is used to inject spatial features into the first Transformer block and extract multi-scale features after the Transformer block; for other Transformer blocks, the multi-scale features extracted after the previous Transformer block are extracted and injected into the Transformer block, and multi-scale features are extracted after the Transformer block; the multi-scale features extracted after the last Transformer block are the first features of the enhanced image;

[0030] The first prediction module is used to predict ADTKD-UMOD based on the first feature.

[0031] Based on the further improvement of the above scheme, the multi-channel fluorescent staining image includes a DAPI fluorescent staining channel, an Opal 480 fluorescent staining channel and an Opal 690 fluorescent staining channel;

[0032] Based on the multi-channel fluorescent staining images of the renal tubule pathological sections of each individual, the contours of the thick ascending limb of the loop of Henle in the staining images of each individual are obtained, including:

[0033] Perform image segmentation on the stained image to obtain the outline of the renal tubules in the stained image;

[0034] Overlay the images of the DAPI fluorescent staining channel and the Opal 480 fluorescent staining channel; match the stained image and the overlaid image to obtain matching point pairs for each effective color development area in the stained image;

[0035] For each effective color development area, determining an affine transformation matrix corresponding to each renal tubule contour in the effective color development area based on the matching point pairs in the effective color development area;

[0036] For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubular contour is valid, the area corresponding to the renal tubular contour in the multi-channel fluorescence staining image is obtained based on the affine transformation matrix corresponding to the renal tubular contour, and based on the intensity of the area corresponding to the renal tubular contour in the multi-channel fluorescence staining image in the Opal 690 fluorescence staining channel, it is determined whether the renal tubular contour is the renal tubular contour of the thick ascending limb of the loop of Henle.

[0037] Based on the further improvement of the above scheme, the stained image and the superimposed image are matched to obtain the matching point pairs for each effective color development area in the stained image, including:

[0038] Performing initial matching on the stained image and the superimposed image to obtain initial matching point pairs and an initial affine transformation matrix;

[0039] For each effective color development area in the dyed image, the effective color development area is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair of each effective color development area.

[0040] Based on a further improvement of the above solution, the effective color display area is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair of each effective color display area, including:

[0041] Perform affine transformation calculation on the initial matching point pairs of the effective color display area to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0042] If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 is obtained as the affine transformation matrix corresponding to the effective color display area; otherwise, the initial affine transformation matrix is ​​used as the affine transformation matrix corresponding to the effective color display area;

[0043] A sliding window is used to slide in the effective color display area, and an area corresponding to the image in each sliding window in the superimposed image is determined based on an affine transformation matrix corresponding to the effective color display area;

[0044] Matching is performed between each image within the sliding window and the corresponding area of ​​the image within the sliding window in the superimposed image to obtain a matching point pair of the effective color display area.

[0045] Based on a further improvement of the above solution, the affine transformation matrix corresponding to each renal tubule outline in the effective color display area is determined based on the matching point pairs in the effective color display area, including:

[0046] Determine the image block to which each renal tubule outline in the effective color development area belongs;

[0047] For each image block, the affine transformation matrix corresponding to the image block is obtained based on the matching point pairs within the image block;

[0048] The affine transformation matrix corresponding to the image block is the affine transformation matrix corresponding to the renal tubule contour belonging to the image block.

[0049] Compared with the existing technology, the present invention collects staining images and multi-channel fluorescence staining images of renal tubular pathological sections of multiple individuals through a data acquisition module, determines the contour of the thick renal tubules of the ascending limb of the loop of Henle in the stained image of each individual through a contour outlining module, thereby efficiently constructing a training sample set, and obtains an ADTKD-UMOD prediction model by semi-supervising training of the constructed deep learning network model based on the constructed training sample set through a model training module, thereby eliminating the need to label all samples, and accurate prediction of ADTKD-UMOD can still be performed when the labeled data is limited. For the individual to be predicted, the renal tubular contour map in its stained image is input into the ADTKD-UMOD prediction model to quickly obtain the ADTKD-UMOD prediction result of the individual to be predicted. Renal pathology diagnosis is a difficult point in specialized pathology diagnosis. Rare genetic diseases represented by ADTKD-UMOD have a high rate of missed diagnosis. The invention can fill the gap in the intelligent recognition system for renal pathological lesions that are easily missed. It can not only be used for screening and diagnosis in grassroots hospitals, but also in renal pathology diagnosis centers to assist pathologists in their daily work, make up for the lack of genetic testing coverage, and improve diagnostic efficiency and accuracy.

[0050] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0052] Figure 1 This is a block diagram of an auxiliary diagnosis system for ADTKD-UMOD based on deep learning according to an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of inclusion bodies;

[0054] Figure 3 is a dyed image according to an embodiment of the present invention;

[0055] Figure 4 Schematic diagram of the process of determining the image block to which the renal tubule contour belongs according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0057] A specific embodiment of the present invention discloses an auxiliary diagnosis system for ADTKD-UMOD based on deep learning, such as Figure 1 As shown, including:

[0058] A data acquisition module, used to collect stained images and multi-channel fluorescent staining images of renal tubular pathological sections of multiple individuals;

[0059] A contour delineation module is used to obtain the contour of the thick ascending limb of the loop of Henle in the stained image of each individual based on the multi-channel fluorescent staining image of the renal tubule pathological section of each individual;

[0060] A training sample construction module is used to construct a training sample set based on a staining image outlining the outline of the renal tubules of the thick ascending limb of the loop of Henle; the training sample set includes labeled samples and unlabeled samples;

[0061] A model training module is used to construct a deep learning network model; based on the training sample set, the deep learning network model is semi-supervised trained to obtain an ADTKD-UMOD prediction model;

[0062] The prediction module is used to input the renal tubule outline image in the stained image of the individual to be predicted into the ADTKD-UMOD prediction model to obtain the ADTKD-UMOD prediction result of the individual to be predicted.

[0063] If the doctor directly checks the UMOD inclusion bodies in the stained image in each field under a high-power objective lens and performs pixel-level annotation, it is very time-consuming. In addition, although the staining can show the brush border, the difference in cytoplasmic eosinophilia between the distal tubules and the proximal tubules is easily affected by the staining batch or tissue fixation quality under a conventional light microscope. The annotation personnel are easily confused by the color difference, and it is even more difficult to distinguish the segments therein, such as the thick ascending limb of the loop of Henle. Therefore, the present invention first determines the outline of the thick ascending limb of the loop of Henle in the stained image through the contour delineation module, and then constructs a training sample set, thereby improving the efficiency and accuracy of sample construction.

[0064] When implemented, the plurality of individuals includes individuals with ADTKD-UMOD and individuals without ADTKD-UMOD.

[0065] During implementation, the stained image is a PAS stained image. Since PAS staining is shorter in both staining time and cost than fluorescent staining, stained images are usually used to train the neural network model. Figure 3 As shown, the black box marks one of the color-developed areas.

[0066] Compared with the existing technology, the deep learning-based ADTKD-UMOD auxiliary diagnosis system provided in this embodiment collects staining images and multi-channel fluorescence staining images of renal tubular pathological sections of multiple individuals through a data acquisition module, and determines the contour of the thick renal tubules of the ascending limb of the loop of Henle in the staining image of each individual through a contour outlining module, thereby efficiently constructing a training sample set, and performing semi-supervised training on the constructed deep learning network model based on the constructed training sample set through a model training module to obtain an ADTKD-UMOD prediction model, so that there is no need to label all samples, and accurate prediction of ADTKD-UMOD can still be performed when the labeled data is limited. For the individual to be predicted, the renal tubular contour map in its staining image is input into the ADTKD-UMOD prediction model to quickly obtain the ADTKD-UMOD prediction result of the individual to be predicted. Renal pathology diagnosis is a difficult point in specialized pathology diagnosis. Rare genetic diseases represented by ADTKD-UMOD have a high rate of missed diagnosis. The invention can fill the gap in the intelligent recognition system for renal pathological lesions that are easily missed. It can not only be used for screening and diagnosis in grassroots hospitals, but also in renal pathology diagnosis centers to assist pathologists in their daily work, make up for the lack of genetic testing coverage, and improve diagnostic efficiency and accuracy.

[0067] Since DAPI fluorescent staining and Opal 480 fluorescent staining can display cell nuclei and epithelial cells, which helps with registration, and Opal690 can mark the thick ascending limb of the loop of Henle, the multi-channel fluorescent staining image includes DAPI fluorescent staining channel, Opal 480 fluorescent staining channel, and Opal 690 fluorescent staining channel.

[0068] Specifically, the contouring module obtains the contour of the thick ascending limb of the loop of Henle in the stained image of each individual based on the multi-channel fluorescent staining image of the renal tubule pathological section of each individual in the following manner:

[0069] S1. Segment the stained image to obtain the outline of the renal tubules in the stained image;

[0070] S2. Overlaying the DAPI fluorescent staining channel image and the Opal 480 fluorescent staining channel image; matching the stained image with the overlaid image to obtain a matching point pair for each effective color development area in the stained image;

[0071] S3. For each effective color development area, determine an affine transformation matrix corresponding to each renal tubule contour in the effective color development area based on the matching point pairs in the effective color development area;

[0072] S4. For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubular contour is valid, then based on the affine transformation matrix corresponding to the renal tubular contour, obtain the area corresponding to the renal tubular contour in the multi-channel fluorescence staining image, and based on the intensity of the area corresponding to the renal tubular contour in the Opal 690 fluorescence staining channel in the multi-channel fluorescence staining image, obtain whether the renal tubular contour is the renal tubular contour of the thick ascending limb of the loop of Henle.

[0073] During implementation, the existing kidney organ image segmentation method can be used to segment the stained image to obtain the renal tubule outline in the stained image.

[0074] The images of the DAPI fluorescent staining channel and the Opal 480 fluorescent staining channel are superimposed to obtain an overlay image, and then the stained image and the overlay image are matched to obtain matching point pairs for each effective color development area in the stained image, specifically including:

[0075] S21, performing initial matching on the stained image and the superimposed image to obtain initial matching point pairs and an initial affine transformation matrix;

[0076] S22. For each effective color development area in the dyed image, match the effective color development area with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain a matching point pair for each effective color development area.

[0077] Specifically, performing initial matching on the stained image and the superimposed image to obtain initial matching point pairs and an initial affine transformation matrix includes the following steps:

[0078] S211, performing morphological processing on the superimposed image;

[0079] S212, performing multi-angle rotation transformation on the processed superimposed image to obtain multiple rotation images;

[0080] S213, matching the stained image with the processed superimposed image and the rotation image respectively, and taking the matching point pair of the image pair with the highest matching degree as the initial matching point pair;

[0081] S214: Calculate the affine transformation matrix between the initial matching point pairs to obtain an initial affine transformation matrix.

[0082] During implementation, morphological processing is first performed on the superimposed image to remove the interference of noise.

[0083] During implementation, morphological processing includes background removal, erosion, and dilation. For example, after sorting the pixel intensities in the overlay image from largest to smallest, the top 30% of pixel intensities are considered valid signals, while the remainder are considered background. The corresponding pixel intensities are set to the background intensity value, such as 0, thereby removing some noise interference. The overlay image with the background removed is first eroded and then dilated. The area of ​​each colored region is then reduced tenfold, and the maximum value is taken as the threshold. Colored regions with an area smaller than the threshold are removed, while significant large regions are retained, further eliminating interference from noise and small fragmented regions. Erosion and dilation can use existing erosion and dilation methods in image science.

[0084] During implementation, the processed superimposed image is subjected to multi-angle rotation transformation to obtain multiple rotation images. For example, the processed superimposed image is rotated 90 degrees clockwise and 90 degrees counterclockwise to obtain corresponding rotation images.

[0085] During implementation, existing image matching algorithms are used to match the stained image with the processed overlay image, and also to match the stained image with each rotated image. For example, the loftr matching algorithm is used to match images. Matching point pairs are found in the image pairs. The number of matching point pairs represents the degree of match; a greater number indicates a higher degree of match. The matching point pair with the highest degree of match is selected as the initial matching point pair.

[0086] It should be noted that if the stained image and the superimposed image rotated 90 degrees clockwise have the highest match, the coordinates of the matching points in the 90-degree clockwise rotated image need to be converted to the coordinates of the original image. Calculate the affine transformation matrix between the initial matching point pairs to obtain the initial affine transformation matrix.

[0087] The initial matching is a rough matching of the global image, and fine-grained matching is further performed based on the initial matching point pairs and initial affine transformation matrix obtained from the initial matching.

[0088] Fine-grained matching is first performed within the color-coded regions. Therefore, it is necessary to first determine which diseased color-coded regions are valid. For each color-coded region in the stained image, if there is an initial matching point in that region, then that region is considered a valid color-coded region.

[0089] Specifically, for each effective color development area in the stained image, the effective color development area is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair of each effective color development area, including:

[0090] S221, performing affine transformation calculation on the initial matching point pairs of the effective color display area to obtain an affine transformation matrix M1 and the number of successfully registered point pairs N1;

[0091] S222: If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 is obtained as the affine transformation matrix corresponding to the effective color display area; otherwise, the initial affine transformation matrix is ​​used as the affine transformation matrix corresponding to the effective color display area;

[0092] S223: Slide a sliding window across the effective color development area, and determine the area in the superimposed image corresponding to the image within each sliding window based on the affine transformation matrix corresponding to the effective color development area;

[0093] S224 , matching each image within the sliding window with the corresponding area of ​​the image within the sliding window in the superimposed image to obtain an effective color region matching point pair of the effective color region.

[0094] During implementation, for an effective color display area, firstly, affine transformation calculation is performed on the initial matching point pairs in the effective color display area to obtain the corresponding affine transformation matrix M1 and the number of successfully aligned point pairs N1.

[0095] During implementation, the affine transformation calculation can be performed using OpenCV's estimateAffine2D function, which returns the number of successfully aligned point pairs.

[0096] If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 calculated for the initial matching point pairs within the effective color rendering area is used as the affine transformation matrix corresponding to the effective color rendering area, making subsequent matching more accurate. If the number of successfully registered point pairs N1 does not exceed the second threshold, the initial affine transformation matrix is ​​used as the affine transformation matrix corresponding to the effective color rendering area. During implementation, the second threshold is set based on the matching accuracy requirements.

[0097] Due to the displacement and deformation of the pathology strips, registration shifts occur. Therefore, the present invention addresses this registration shift by employing a stripe-by-strip matching approach. During implementation, a non-overlapping sliding window is used, sliding across each valid color region. For each image within the sliding window, the corresponding region in the superimposed image is determined based on the affine transformation matrix corresponding to that valid color region.

[0098] Specifically, the area corresponding to the image in each sliding window in the superimposed image is determined in the following manner:

[0099] S2231, performing affine transformation on the vertex coordinates of the image in the sliding window based on the affine transformation matrix corresponding to the effective color display area to obtain the points corresponding to the vertices in the superimposed image;

[0100] S2232: The area formed by the bounding box of the point corresponding to the vertex in the superimposed image is the area corresponding to the image in the sliding window in the superimposed image.

[0101] During implementation, the coordinates of the four vertices of the image in the sliding window (top left, top right, bottom right, and bottom left) are calculated based on the affine transformation matrix corresponding to the effective color display area. The area formed by the bounding box of these corresponding points is the area in the superimposed image corresponding to the image in the sliding window.

[0102] Obtain the area in the superimposed image corresponding to the image in the sliding window, match the image in the sliding window with the corresponding area in the superimposed image (for example, using the loftr matching algorithm for image matching), and obtain matching point pairs corresponding to the image in the sliding window; the matching point pairs corresponding to all the images in the sliding window in the effective color display area constitute the matching point pairs of the effective color display area.

[0103] The effective color display area is matched in fine-grained manner through a sliding window to improve the accuracy of registration.

[0104] After obtaining the matching point pairs of each effective color display area, determining the affine transformation matrix corresponding to each renal tubule contour in the effective color display area based on the matching point pairs of the effective color display area specifically includes:

[0105] S31, determining the image block to which each renal tubule outline in the effective color development area belongs;

[0106] S32. For each image block, obtain an affine transformation matrix corresponding to the image block based on the matching point pairs in the image block;

[0107] S32. The affine transformation matrix corresponding to the image block is the affine transformation matrix corresponding to the renal tubule contour belonging to the image block.

[0108] Specifically, the image block to which each renal tubule outline in the effective color display area belongs is determined in the following manner:

[0109] S311, for each renal tubule outline in the valid color development area, to which no image block is determined, the outer bounding box of the renal tubule outline is used as the current area;

[0110] S312, obtaining a current window with the upper left corner vertex of the current region as the upper left corner of the window and a preset length as the side length; if the current window contains a renal tubule contour that is outside the current region and to which no image block has been determined, taking the common circumscribed border of the renal tubule within the current region and the renal tubule contour within the current window but outside the current region and to which no image block has been determined as the current region; if not, keeping the current region unchanged;

[0111] S313, obtaining a current window with the upper right corner vertex of the current region as the upper right corner of the window and a preset length as the side length; if the current window contains a renal tubule contour that is outside the current region and to which no image block has been determined, taking the common circumscribed border of the renal tubule within the current region and the renal tubule contour within the current window but outside the current region and to which no image block has been determined as the current region; if not, keeping the current region unchanged;

[0112] S314, obtaining a current window with the lower right corner vertex of the current region as the lower right corner of the window and a preset length as the side length; if the current window contains a renal tubule contour that is outside the current region and to which no image block has been determined, then taking the common circumscribed border of the renal tubule within the current region and the renal tubule contour within the current window but outside the current region and to which no image block has been determined as the current region; if not, keeping the current region unchanged;

[0113] S315, obtaining a current window with the lower left corner vertex of the current area as the lower left corner of the window and a preset length as the side length; the area of ​​the current window is regarded as an image block; all renal tubule contours in the image block to which they do not belong in the current window belong to the image block.

[0114] by Figure 4 Take this as an example to illustrate. Figure 4 In (a), the circle is the outline of the renal tubule of the image block to which it is not determined, and the solid line box is the outer bounding box of the renal tubule outline. First, the outer bounding box of the renal tubule outline of one of the undetermined corresponding image blocks is used as the current area, that is, the solid line box in the figure is the current area, the upper left corner vertex of the current area is used as the upper left corner of the window, and the preset length is used as the side length to obtain the current window, that is, Figure 4 In the dotted box in (a), there are renal tubule contours in the current window that are not in the current area (solid box) and have no determined image block to which they belong. The common external frame of the renal tubule contours of these two renal tubules is used as the current area, for example Figure 4 The solid line box in (b).

[0115] Then with Figure 4 The upper right corner of the solid line frame (current area) in (b) is used as the upper right corner vertex of the window, and the preset length is the side length to draw the window to get the current window, that is, Figure 4 The dotted box in (b) of the current window. If there is no renal tubule outline that is not in the current area and whose image block is not determined, the current area remains unchanged. The current window is obtained by taking the lower right corner vertex of the current area as the lower right corner of the window and the preset length as the side length, as shown in the following example: Figure 4 The dotted box in (c) of the current window still has no renal tubule contour that is not in the current area and whose image block is not determined. The current window is obtained by taking the lower left corner vertex of the current area as the lower left corner of the window and the preset length as the side length, as shown in the following example: Figure 4In the dotted box of (d), the area of ​​the current window is regarded as an image block; all the renal tubule contours in the current window that have not been determined to belong to the image block belong to this image block.

[0116] The above process for determining the image blocks that belong to the renal tubular outline does not uniformly divide the pathological strip into equal-sized blocks. Instead, the blocks are determined based on the density of the renal tubules. The blocks corresponding to areas with dense renal tubular outlines are larger than those corresponding to areas with sparse renal tubular outlines. Areas with dense renal tubular outlines are treated as one block, reducing the number of blocks without compromising registration quality. Areas without renal tubular shapes do not require block determination, resulting in fewer blocks than those obtained by uniform division, thereby improving computational efficiency.

[0117] After determining the image block to which each renal tubule contour in the effective color display area belongs, for each image block, the affine transformation matrix corresponding to the image block is obtained based on the matching point pairs within the image block, and then the area corresponding to the renal tubule contour belonging to the image block in the superimposed image is determined based on the obtained affine transformation matrix.

[0118] Specifically, the affine transformation matrix corresponding to each image block is obtained based on the matching point pairs within the image block, including:

[0119] S321, performing affine transformation calculation based on the matching point pairs within the renal tubule contour of the image block to obtain an affine transformation matrix M2 and the number of successfully registered point pairs N2;

[0120] S322, determining the area corresponding to the image block in the superimposed image based on the affine transformation matrix M2;

[0121] S323 : Obtain an affine transformation matrix corresponding to the image block based on the image block and the region corresponding to the image block in the superimposed image.

[0122] During implementation, for an image block, an affine transformation is first calculated based on the matching point pairs located in the renal tubule contour among the matching point pairs in the image block. The affine transformation calculation can use the aforementioned estimateAffine2D function to obtain the affine transformation matrix M2 and the number of successfully aligned point pairs N2.

[0123] It should be noted that if the number of matching point pairs located within the renal tubular contour of the image block is less than a preset threshold, for example, less than 50 pairs, then the point pairs closest to the renal tubular contour are selected from the matching points located in the image block but not within the renal tubular contour as the matching point pairs located within the renal tubular contour of the image block to supplement the number of matching point pairs located within the renal tubular contour of the image block, thereby facilitating more accurate matching.

[0124] Then, the area corresponding to the image block in the superimposed image is determined based on the affine transformation matrix M2.

[0125] During implementation, an affine transformation is performed on the vertex coordinates of the image block based on the affine transformation matrix M2 to obtain the points corresponding to the vertices in the overlay image. The area formed by the bounding box of the points corresponding to the vertices of the image block in the overlay image is the area corresponding to the image block in the overlay image. For the specific process, please refer to steps S2231 to S2232.

[0126] Then, based on the image block and the area corresponding to the image block in the superimposed image, an affine transformation matrix corresponding to the image block is obtained, specifically including:

[0127] S3231, matching the image block with the corresponding region of the image block in the superimposed image to obtain a matching point pair C2; performing affine transformation calculation on the matching point pair C2 to obtain an affine transformation matrix M3 and the number of successfully registered point pairs N3;

[0128] S3232: If the number of matching point pairs C2 located within the renal tubule contour exceeds a third threshold (e.g., 40 pairs), then

[0129] Perform affine transformation calculation on the matching point pairs C2 that are located within the renal tubule contour to obtain the affine transformation matrix M4 and the number of successfully registered point pairs N4;

[0130] If the number of successfully registered point pairs N4 exceeds a fourth threshold (e.g., 20 pairs), the affine transformation matrix M4 is used as the valid affine transformation matrix corresponding to the image block; otherwise, if the number of successfully registered point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is an invalid affine transformation matrix corresponding to the image block;

[0131] If the number of matching point pairs in the matching point pairs C2 that are located within the contour of the renal tubule does not exceed the third threshold (40 pairs), then: if the number of successfully aligned point pairs N3 exceeds the fourth threshold, the affine transformation matrix M3 is used as the valid affine transformation matrix corresponding to the image block; otherwise, the affine transformation matrix M3 is the invalid affine transformation matrix corresponding to the image block.

[0132] During implementation, the matching algorithm of step S3231 adopts the above-mentioned loftr algorithm to obtain the matching point pair C2, and uses the above-mentioned estimateAffine2D function to perform affine transformation calculation on the matching point pair C2 to obtain the affine transformation matrix M3 and the number of successfully aligned point pairs N3.

[0133] If the number of matching point pairs in the matching point pairs C2 that are located within the renal tubule contour exceeds a third threshold value (e.g., 40 pairs), an affine transformation calculation is performed on the matching point pairs in the matching point pairs C2 that are located within the renal tubule contour to obtain an affine transformation matrix M4 and the number of successfully registered point pairs N4. It is then determined whether the number of successfully registered point pairs N4 exceeds a fourth threshold value (e.g., 20 pairs). If it exceeds the fourth threshold value, the affine transformation matrix M4 is used as the affine transformation matrix corresponding to the image block, which can achieve more accurate matching. Therefore, the affine transformation matrix M4 is used as the affine transformation matrix corresponding to the image block. If it is successfully determined that the number of successfully registered point pairs N4 does not exceed the fourth threshold value, it is determined whether the number of successfully registered point pairs N3 exceeds the fourth threshold value. If it does, the affine transformation matrix M3 is used as the affine transformation matrix corresponding to the image block. If N3 also does not exceed the fourth threshold value, the affine transformation matrix M3 is used as the invalid affine transformation matrix corresponding to the image block.

[0134] If the number of matching point pairs in the matching point pairs C2 that are located within the contour of the renal tubule does not exceed the third threshold, then check whether the number of successfully aligned point pairs N3 exceeds the fourth threshold. If it exceeds, the affine transformation matrix M3 is used as the affine transformation matrix corresponding to the image block. If N3 also does not exceed the fourth threshold, the affine transformation matrix M3 is used as the invalid affine transformation matrix corresponding to the image block.

[0135] When the number of successfully registered point pairs N3 is small, the calculation will have a large error. Therefore, M3 is the invalid affine transformation matrix corresponding to the image block.

[0136] During implementation, the third threshold and the fourth threshold are set according to the matching accuracy requirement.

[0137] The affine transformation matrix corresponding to the image block is the affine transformation matrix corresponding to all renal tubule contours belonging to the image block.

[0138] By determining the image block corresponding to the tubule contour and calculating the affine transformation matrix corresponding to the image block, it is no longer necessary to calculate the affine transformation matrix for each tubule contour, thereby improving the calculation efficiency.

[0139] For each renal tubular outline, if the affine transformation matrix corresponding to the renal tubular outline is valid, then based on the affine transformation matrix corresponding to the renal tubular outline, the area corresponding to the renal tubular outline in the multi-channel fluorescent staining image is obtained. During implementation, the coordinates of the four vertices of the circumscribed frame of the renal tubular outline are affine transformed according to the affine transformation matrix corresponding to the renal tubular outline to obtain the points corresponding to the vertices in the multi-channel fluorescent staining image; the area formed by the circumscribed frame of the points corresponding to the vertices in the multi-channel fluorescent staining image is the area corresponding to the renal tubular outline in the multi-channel fluorescent staining image. For the specific process, please refer to steps S2231 to S2232.

[0140] If the renal tubule outline is within the corresponding area of ​​the multi-channel fluorescence staining image and the intensity of the Opal 690 fluorescence staining channel exceeds the preset threshold, it is the renal tubule outline of the thick ascending limb of the loop of Henle.

[0141] After obtaining the outline of the thick renal tubules of the ascending limb of the loop of Henle in the stained image, a training sample set is constructed based on the stained image outlining the outline of the thick renal tubules of the ascending limb of the loop of Henle.

[0142] During implementation, a professional doctor will identify and judge whether there are inclusion bodies on the contour of the thick ascending limb of the loop of Henle (i.e., whether there is ADTKD-UMOD). Since the full supervised annotation form in the pathological section images at the million-cell level leads to extremely high costs, some stained images can be annotated, that is, whether there are inclusion bodies in the thick ascending limb of the loop of Henle. The image of the contour of the renal tubules of the thick ascending limb of the loop of Henle and the image of the contour of the renal tubules of other non-thick ascending limbs of the loop of Henle in the stained image are intercepted as the input images of the samples to construct a training sample set. The sample set includes labeled samples and unlabeled samples. The labeled samples are labeled samples, and the sample label is whether the renal tubule contour contains inclusion bodies. The unlabeled samples are samples without labels. Construct a deep learning network model; perform semi-supervised training on the deep learning network model based on the training sample set to obtain an ADTKD-UMOD prediction model.

[0143] Specifically, the constructed deep learning network model includes:

[0144] A multi-mode enhancement module is configured to perform first-mode enhancement and second-mode enhancement on unlabeled samples to obtain first enhanced images and second enhanced images, and input the obtained first enhanced images and second enhanced images into the first branch and the second branch; perform second-mode enhancement on labeled samples to obtain second enhanced images, and input the obtained second enhanced images into the first branch and the second branch;

[0145] The first branch is used to extract features of the enhanced image using the Transformer network to obtain the first feature of the enhanced image, and perform ADTKD-UMOD prediction based on the first feature;

[0146] The second branch is used to extract features of the enhanced image using a convolutional network to obtain the second feature of the enhanced image, and perform ADTKD-UMOD prediction based on the second feature.

[0147] During implementation, the first enhancement mode is weak enhancement, such as image flipping and translation. During implementation, the second enhancement mode is strong enhancement mode, and the enhancement strategy can be CTAugment or RandAugment.

[0148] For unlabeled samples, the first mode enhancement and the second mode enhancement are performed respectively to obtain the corresponding first enhanced image and the second enhanced image. For labeled samples, only the second mode enhancement is performed on the sample image to obtain the corresponding second enhanced image.

[0149] The obtained enhanced images are input into the first branch and the second branch respectively.

[0150] Specifically, the first branch uses the Transformer network to extract features from the enhanced image to obtain the first feature of the enhanced image. The Transformer network of the first branch includes:

[0151] The image block encoding module is used to segment the enhanced image and obtain the corresponding embedded representation of each image block;

[0152] Multiple Transformer blocks connected in sequence are used to perform multi-layer feature extraction on the embedded representation of image blocks;

[0153] The spatial feature extraction module is used to extract features of the enhanced image based on the residual convolutional network to obtain spatial features;

[0154] A feature interaction module is used to inject spatial features into the first Transformer block and extract multi-scale features after the Transformer block; for other Transformer blocks, the multi-scale features extracted after the previous Transformer block are extracted and injected into the Transformer block, and multi-scale features are extracted after the Transformer block; the multi-scale features extracted after the last Transformer block are the first features of the enhanced image;

[0155] The first prediction module is used to predict ADTKD-UMOD based on the first feature.

[0156] In implementation, the block encoding module first divides the enhanced image into multiple blocks, for example, a 256×256 image into multiple 16×16 blocks. Each block is flattened and projected into D dimensions. The resulting D vector is then added to the block's positional encoding to produce an embedded representation for each block.

[0157] The embedded representation of each image block is passed through multiple Transformer blocks connected in sequence for feature extraction.

[0158] In order to supplement more local image information, spatial features are extracted through the spatial feature extraction module, and the spatial features are injected into the Transformer block through the feature interaction module to improve the performance of the model.

[0159] Specifically, the spatial feature extraction module uses a residual convolutional network to extract multi-scale features from the enhanced image to obtain spatial features. The residual convolutional network can be, for example, ResNet. During implementation, multiple feature maps of different resolutions are extracted through the residual convolutional network, and the obtained multiple feature maps are flattened and spliced ​​to obtain spatial features, which are expressed as

[0160] Specifically, the feature interaction module uses the following formula to inject spatial features into the i-th Transformer block:

[0161]

[0162] Among them, γ i represents the learnable parameters corresponding to the i-th Transformer block, Attention(·) represents the attention mechanism, LN(·) represents layer normalization, and when i>1, Represents the multi-scale features extracted after the i-1th Transformer block, when i=1, Represents spatial features; if i=1, represents the embedded representation, if i>1, Represents the output features of the i-1th Transformer block; Represents the input features of the i-th Transformer block.

[0163] When implemented, spatial features are injected into the Transformer block through the attention mechanism.

[0164] Specifically, the feature interaction module uses the following formula to extract multi-scale features after the i-th Transformer block:

[0165]

[0166] in, represents the multi-scale features extracted after the i-1th Transformer block, Attention(·) represents the attention mechanism, LN(·) represents layer normalization, represents the output features of the i-th Transformer block, represents the intermediate parameters, FFN(·) represents the feedforward neural network module, represents the multi-scale features extracted after the i-th Transformer block.

[0167] When implementing, extract the multi-scale features after the i-th Transformer block After that, there is the i+1th Transformer block, then As part of the input features of the i+1th Transformer block, it is injected into the i+1th Transformer block. Otherwise, as the first feature of the enhanced image.

[0168] The first prediction module performs ADTKD-UMOD prediction based on the first feature. During implementation, the first prediction module may use an existing classification head.

[0169] The second branch uses a convolutional network to extract features from the enhanced image, obtains a second feature of the enhanced image, and performs ADTKD-UMOD prediction based on the second feature. The second branch can use an existing convolutional network structure.

[0170] During implementation, in order to integrate the inductive bias of the convolutional network into the Transformer network, enhance the spatial information of the convolutional network, and share knowledge, the deep learning network model also includes an information interaction module to interact with the extracted features of the convolutional network and the Transformer network.

[0171] Specifically, the information interaction module uses the following methods to exchange information between the two branches:

[0172]

[0173] in, represents the features of the j-th image block output by the i-th Transformer block in the Transformer network, M C,j represents the features of the area corresponding to the j-th image block in the feature map of the convolutional network, LN(·) represents layer normalization, pooling(·) represents downsampling, align(·) represents mapping the features to the same dimensional space, BN(·) represents batch normalization, and upsample(·) represents upsampling.

[0174] It should be noted that there are as many information interactions as there are Transformer blocks. That is, in the i-th information interaction, the features output by the i-th Transformer block are fused with the features extracted by the convolutional network, and then the features extracted by the convolutional network are added to the features extracted by the i-th Transformer block, which are used as the features extracted by the convolutional network after the i-th information interaction.

[0175] After the first feature extraction branch and the second feature branch extract the first feature and the second feature, the prediction module predicts whether the sample has ADTKD-UMOD based on the first feature and the second feature respectively.

[0176] The loss is calculated based on the prediction results of the prediction module and the parameters of the deep learning network model are updated.

[0177] Specifically, the training loss of the deep learning network model is calculated using the following formula:

[0178] Loss=Loss1+λLoss2

[0179] Among them, Loss1 represents the loss of labeled samples, Loss2 represents the loss of unlabeled samples, and λ represents the weight parameter.

[0180] For labeled samples, the loss is calculated based on the prediction results based on the first feature, the prediction results based on the second feature, and the label information.

[0181] Specifically, the loss of labeled samples is calculated using the following formula:

[0182]

[0183] Among them, x i represents the input image of the i-th labeled sample, y i represents the label of the i-th labeled sample, N l Indicates the number of labeled samples in the current training batch, represents the loss function, g + (·) indicates the second mode enhancement, f T (·) represents the Transformer network, f C (·) denotes a convolutional network.

[0184] The labeled samples are only enhanced in the second mode. Therefore, the loss is calculated by the loss calculated by the prediction results obtained by the two branches of the second enhanced image and the label of the sample. Existing loss calculation functions can be used, such as the cross entropy loss function.

[0185] For unlabeled samples, since there is no label information, an unsupervised loss is calculated. The input image of the unlabeled sample is enhanced in two ways. Since weak enhancement retains more information of the original image, the prediction result of the second branch on the first enhanced image is used as the pseudo label, and training is performed using the pseudo label pair.

[0186] Specifically, the loss of unlabeled samples is calculated using the following formula:

[0187]

[0188] Among them, x j represents the input image of the jth unlabeled sample, N uIndicates the number of unlabeled samples in the current training batch, represents the loss function, represents the category of the first enhanced image of the jth unlabeled sample predicted by the second branch, g + (·) indicates the second mode enhancement, f T (·) represents the Transformer network, f C (·) represents the convolutional network, I[·] represents the characteristic function, and p j represents the probability of the first enhanced image of the j-th unlabeled sample belonging to each category after being predicted by the second branch, <·,·> represents the similarity, and t represents the filtering threshold.

[0189] It should be noted that the label of the sample is a one-hot encoding vector, and the category of the first enhanced image of the i-th unlabeled sample is predicted by the second branch Is the one-hot encoding vector of the predicted maximum probability corresponding to the category. The probability p of the first enhanced image of the i-th unlabeled sample belonging to each category predicted by the second branch j The predicted probability vector of each category. Filter threshold In order to enhance the reliability of pseudo labels, only when the similarity is greater than The loss is calculated only for the pseudo labels that are not sure, thus filtering out the uncertain pseudo labels.

[0190] The parameters of the deep learning network model are trained based on the loss. When the change in loss reaches the preset accuracy or the number of iterations reaches the preset number, the training is stopped and the ADTKD-UMOD prediction model is obtained.

[0191] For the individual to be predicted, each renal tubule outline in the stained image is input into the obtained ADTKD-UMOD prediction model, and whether the renal tubule outline is ADTKD-UMOD can be quickly predicted.

[0192] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0193] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A deep learning-based auxiliary diagnosis system for ADTKD-UMOD, characterized by: include: A data acquisition module, used to collect stained images and multi-channel fluorescent staining images of renal tubular pathological sections of multiple individuals; A contour delineation module is used to obtain the contour of the thick ascending limb of the loop of Henle in the stained image of each individual based on the multi-channel fluorescent staining image of the renal tubule pathological section of each individual; A training sample construction module is used to construct a training sample set based on a staining image outlining the outline of the renal tubules of the thick ascending limb of the loop of Henle; the training sample set includes labeled samples and unlabeled samples; A model training module is used to construct a deep learning network model; based on the training sample set, the deep learning network model is semi-supervised trained to obtain an ADTKD-UMOD prediction model; The prediction module is used to input the renal tubule outline image in the stained image of the individual to be predicted into the ADTKD-UMOD prediction model to obtain the ADTKD-UMOD prediction result of the individual to be predicted.

2. The deep learning-based ADTKD-UMOD auxiliary diagnosis system according to claim 1, characterized in that: The deep learning network model includes: A multi-mode enhancement module is configured to perform first-mode enhancement and second-mode enhancement on unlabeled samples to obtain first enhanced images and second enhanced images, and input the obtained first enhanced images and second enhanced images into the first branch and the second branch; perform second-mode enhancement on labeled samples to obtain second enhanced images, and input the obtained second enhanced images into the first branch and the second branch; The first branch is used to extract features of the enhanced image using the Transformer network to obtain the first feature of the enhanced image, and perform ADTKD-UMOD prediction based on the first feature; The second branch is used to extract features of the enhanced image using a convolutional network to obtain the second features of the enhanced image, and perform ADTKD-UMOD prediction based on the second features.

3. The ADTKD-UMOD auxiliary diagnosis system based on deep learning according to claim 2 is characterized in that: The training loss of the deep learning network model is calculated using the following formula: Loss=Loss1+λLoss2 Among them, Loss1 represents the loss of labeled samples, Loss2 represents the loss of unlabeled samples, and λ represents the weight parameter.

4. The ADTKD-UMOD auxiliary diagnosis system based on deep learning according to claim 3 is characterized in that: The loss of labeled samples is calculated using the following formula: Among them, x i represents the input image of the i-th labeled sample, y i represents the label of the i-th labeled sample, N l Indicates the number of labeled samples in the current training batch, represents the loss function, g + (·) indicates the second mode enhancement, f T (·) represents the Transformer network, f C (·) denotes a convolutional network.

5. The ADTKD-UMOD auxiliary diagnosis system based on deep learning according to claim 3 is characterized in that: The loss of unlabeled samples is calculated using the following formula: Among them, x j represents the input image of the jth unlabeled sample, N u Indicates the number of unlabeled samples in the current training batch, represents the loss function, represents the category of the first enhanced image of the jth unlabeled sample predicted by the second branch, g + (·) indicates the second mode enhancement, f T (·) represents the Transformer network, f C (·) represents the convolutional network, I[·] represents the characteristic function, and p j represents the probability that the first enhanced image of the jth unlabeled sample belongs to each category after being predicted by the second branch, <·,·> represents the similarity, Indicates the filtering threshold.

6. The ADTKD-UMOD auxiliary diagnosis system based on deep learning according to claim 2, characterized in that: The Transformer network of the first branch includes: The image block encoding module is used to segment the enhanced image and obtain the corresponding embedded representation of each image block; Multiple Transformer blocks connected in sequence are used to perform multi-layer feature extraction on the embedded representation of image blocks; The spatial feature extraction module is used to extract features of the enhanced image based on the residual convolutional network to obtain spatial features; A feature interaction module is used to inject spatial features into the first Transformer block and extract multi-scale features after the Transformer block; for other Transformer blocks, the multi-scale features extracted after the previous Transformer block are extracted and injected into the Transformer block, and multi-scale features are extracted after the Transformer block; the multi-scale features extracted after the last Transformer block are the first features of the enhanced image; The first prediction module is used to predict ADTKD-UMOD based on the first feature.

7. The deep learning-based ADTKD-UMOD auxiliary diagnosis system according to claim 1, characterized in that: The multi-channel fluorescence staining image includes a DAPI fluorescence staining channel, an Opal 480 fluorescence staining channel, and an Opal 690 fluorescence staining channel; Based on the multi-channel fluorescent staining images of the renal tubule pathological sections of each individual, the contours of the thick ascending limb of the loop of Henle in the staining images of each individual are obtained, including: Perform image segmentation on the stained image to obtain the outline of the renal tubules in the stained image; Overlay the images of the DAPI fluorescent staining channel and the Opal 480 fluorescent staining channel; match the stained image and the overlaid image to obtain matching point pairs for each effective color development area in the stained image; For each effective color development area, determining an affine transformation matrix corresponding to each renal tubule contour in the effective color development area based on the matching point pairs in the effective color development area; For each renal tubular contour, if the affine transformation matrix corresponding to the renal tubular contour is valid, the area corresponding to the renal tubular contour in the multi-channel fluorescence staining image is obtained based on the affine transformation matrix corresponding to the renal tubular contour, and based on the intensity of the area corresponding to the renal tubular contour in the multi-channel fluorescence staining image in the Opal 690 fluorescence staining channel, it is determined whether the renal tubular contour is the renal tubular contour of the thick ascending limb of the loop of Henle.

8. The deep learning-based ADTKD-UMOD auxiliary diagnosis system according to claim 7, characterized in that: Match the stained image with the overlay image to obtain matching point pairs for each effective color development area in the stained image, including: Performing initial matching on the stained image and the superimposed image to obtain initial matching point pairs and an initial affine transformation matrix; For each effective color development area in the dyed image, the effective color development area is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain the matching point pair of each effective color development area.

9. The deep learning-based ADTKD-UMOD auxiliary diagnosis system according to claim 8, characterized in that: The effective color display area is matched with the superimposed image based on the initial matching point pair and the initial affine transformation matrix to obtain a matching point pair for each effective color display area, including: Perform affine transformation calculation on the initial matching point pairs of the effective color display area to obtain the affine transformation matrix M1 and the number of successfully registered point pairs N1; If the number of successfully registered point pairs N1 exceeds the second threshold, the affine transformation matrix M1 is obtained as the affine transformation matrix corresponding to the effective color display area; otherwise, the initial affine transformation matrix is ​​used as the affine transformation matrix corresponding to the effective color display area; A sliding window is used to slide in the effective color display area, and an area corresponding to the image in each sliding window in the superimposed image is determined based on an affine transformation matrix corresponding to the effective color display area; Matching is performed between each image within the sliding window and the corresponding area of ​​the image within the sliding window in the superimposed image to obtain a matching point pair of the effective color display area.

10. The deep learning-based ADTKD-UMOD auxiliary diagnosis system according to claim 7, characterized in that: Determining an affine transformation matrix corresponding to each renal tubule contour in the effective color display area based on the matching point pairs in the effective color display area includes: Determine the image block to which each renal tubule outline in the effective color development area belongs; For each image block, the affine transformation matrix corresponding to the image block is obtained based on the matching point pairs within the image block; The affine transformation matrix corresponding to the image block is the affine transformation matrix corresponding to the renal tubule contour belonging to the image block.

Images