Renal tubule and interstitial lesion grading system
By combining the SegFormer architecture with multi-stage feature extraction of the self-attention mechanism, the problems of low computational efficiency and insufficient accuracy in grading renal tubular and interstitial lesions are solved, achieving efficient and accurate lesion grading suitable for renal pathology imaging.
Patent Information
- Application Number
- CN202511004814.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-10-10
AI Technical Summary
Existing technologies for grading renal tubular atrophy and interstitial fibrosis in renal biopsy pathological images have problems such as low computational efficiency, limited feature extraction, neglect of dynamic interactions, and strong subjective dependence, resulting in insufficient grading accuracy.
A grading system for renal tubular and interstitial lesions is adopted, including a preprocessing module, a slice processing module, a lesion segmentation network module and a grading network module. The SegFormer architecture and self-attention mechanism are used, combined with lesion segmentation mask and scale prior information, to achieve lesion grading through multi-stage feature extraction and classification.
The accuracy and efficiency of IFTA lesion grading in renal pathology WSIs were improved, and it was able to identify 91.18% of lesions in ultra-high-resolution images, reduce the computational burden, be suitable for small sample data, and have good clinical applicability and generalization ability.
Smart Images
Figure CN120765628A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of grading of renal tubular and interstitial lesions, and particularly relates to a grading system for renal tubular and interstitial lesions. BACKGROUND
[0002] At present, the grading of renal tubular atrophy and interstitial fibrosis (IFTA) in kidney biopsy pathological images mainly depends on the manual evaluation of pathologists, and there are problems of strong subjectivity and low efficiency. In the prior art, the existing automatic technology generally includes a U-Net segmentation network (used for medical image segmentation, but the large number of parameters leads to heavy computing burden), a DeepLab v2 (segmenting lesion tissues in a kidney biopsy WSI, but lacking context understanding of global features of the lesion), and a continuous CNN architecture (evaluating interstitial fibrosis and renal tubular atrophy respectively, but failing to comprehensively capture the dynamic interaction of the two).
[0003] For the prior art, the following problems exist: First, the computing efficiency is low. For example, a high-parameter network such as a U-Net segmentation network is difficult to process a super-high-resolution WSI.
[0004] Second, the feature extraction is limited. The existing model cannot capture both global and local features of the lesion, resulting in insufficient grading accuracy.
[0005] Third, the dynamic interaction is ignored. That is, the synergistic effect of interstitial fibrosis and renal tubular atrophy is not modeled.
[0006] Fourth, the subjectivity is strong. Manual evaluation has poor consistency, affecting the reliability of the grading results. SUMMARY
[0007] To solve the technical problems in the related art, the present application provides a grading system for renal tubular and interstitial lesions.
[0008] To achieve the above purpose, the technical scheme adopted by the present application comprises: According to a first aspect of the present application, a grading system for renal tubular and interstitial lesions is provided, comprising: a preprocessing module configured to segment the renal cortex and renal medulla regions in a kidney pathological whole-image, and to remove the renal medulla region; a slice processing module configured to segment the preprocessed renal cortex region image into slice images of a first resolution; a lesion segmentation network module based on a SegFormer architecture and integrating self-attention and convolutional layers, configured to input the slice images and output IFTA lesion segmentation masks; a mask splicing module, which splices the lesion segmentation mask into a mask image of a second resolution; a hierarchical network module, which comprises a three-stage feature extraction unit, an input end of which receives the slice image, the lesion segmentation mask, the spliced mask image, and lesion proportion prior information, and an output end of which is connected to a classification unit to generate an IFTA lesion grading result.
[0009] Optionally, the lesion segmentation network module comprises: an encoder unit, which extracts features in four stages using an overlapping block embedding operation, each stage comprising a segmentation feature extraction submodule; a decoder unit, which performs unified dimension processing, upsampling, and splicing on the four-stage features using a multi-layer perception, and outputs a segmentation mask; The segmentation feature extraction submodule integrates a self-attention mechanism and a 3x3 deep separable convolution layer, and introduces a scaling factor to reduce computational complexity.
[0010] Optionally, the three-stage structure of the hierarchical network module is as follows: First stage: comprising a hierarchical feature extraction submodule and a lesion-aware attention submodule, an input end of which receives a slice image of a first resolution and a lesion segmentation mask; Second stage: receiving the feature map output by the first stage, adjusting the feature map through an MLP layer, and inputting the adjusted feature map into the hierarchical feature extraction submodule and the lesion-aware attention submodule; Third stage: receiving the feature map output by the second stage, adjusting the feature map through an MLP layer, and inputting the adjusted feature map into the hierarchical feature extraction submodule, and combining lesion proportion prior information to generate a grading result.
[0011] Optionally, the lesion-aware attention submodule comprises: a mask processing unit, which performs Sigmoid transformation and normalization operations on the input lesion segmentation mask; a feature map upsampling unit, which adjusts the size of the feature map to match the size of the mask; a weighting unit, which performs pixel-by-pixel matrix multiplication operations between the normalized mask and the upsampled feature map; a global max pooling unit, which compresses the weighted feature map and outputs it to the next stage.
[0012] Optionally, the mask processing unit performs: to smoothly transition the binary mask to the [0, 1] interval; In the above formula, is the input segmentation mask, is the segmentation mask obtained by Sigmoid transformation, is the final segmentation mask.
[0013] Optionally, the hierarchical feature extraction submodule initializes weights through a pre-trained network, where the pre-trained network comprises: The online network and the target network receive different enhanced variants of the same kidney pathology slice; Feature extractor and MLP layer, used to generate feature vectors; Similarity loss calculation unit, according to the formula Update the network weights, where The operation represents L2 normalization; yes The mathematical expectation of It is the prediction head Online network feature information The prediction results, It is an enhanced sample Feature information extracted by online network; yes The mathematical expectation of Another enhanced sample Feature information extracted by the target network; The operation represents the calculation of the inner product of the two; denote the updateable weights of the online network and the target network, respectively.
[0014] Optionally, the classification unit of the hierarchical network module includes: Fully connected layer and SoftMax function, output probability distribution ; Cross entropy loss calculation unit, according to the formula Optimize parameters; In the above formula, represents the probability that the i-th sample is predicted to be category c, Indicates that the hierarchical network predicts the i-th sample as the c-class output prediction value, k represents the k-th category, is the number of categories for lesion grading, Indicates that the hierarchical network predicts the i-th sample as the output prediction value of the k-th category, Is a symbolic function that indicates whether the i-th sample belongs to category c. If it does, it takes 1, and if it does not, it takes 0.
[0015] Optionally, the lesion ratio prior information calculation unit performs: And the scalar Expand to an auxiliary vector of the same length as the feature map and input it into the third stage; In the above formula, is the area of IFTA lesion, is the area of the renal cortex.
[0016] Optionally, the renal tubular and interstitial lesions grading system further includes a visualization output module, which is used to: The lesion segmentation mask is superimposed on the original pathological image, and the quantification results and the lesion area ratio are displayed simultaneously.
[0017] Optionally, the grading results include T0, T1 and T2; wherein, the standard for T0 is: the proportion of IFTA lesion area is <25%; the standard for T1 is: 25%≤IFTA lesion area proportion≤50%; the standard for T2 is: the proportion of IFTA lesion area is >50%.
[0018] Beneficial effects: 1. Through the above technical solution, the present invention discloses a multi-level IFTA lesion grading network, which, combined with the lesion area information predicted by the IFTA segmentation network and the prior knowledge of the lesion ratio, can accurately grade the lesions according to the severity of the lesions. For ultra-high-resolution kidney pathology WSI, the lesion grading network of the present invention is designed to extract mapping features layer by layer, which can pay attention to global feature information and local lesion feature information at the same time. In order to address the problem that the feature distribution of IFTA is scattered and close to the background features, a lesion perception sub-attention module is introduced to enhance the relevant features of the lesion area. The hierarchical network model proposed in the present invention can effectively identify the severity of IFTA lesions in kidney pathology WSI, and provide efficient and accurate lesion grading assessment information for kidney pathologists and clinicians.
[0019] Specifically, through the above technical solution, the system of the present invention can significantly improve grading accuracy. By simultaneously inputting slice images (local details), lesion segmentation masks (spatial positioning), medium-resolution masks (regional correlation), and lesion proportion priors (global severity), the system forms a multi-dimensional feature complementarity, effectively achieving a 91.18% grading accuracy for IFTA lesions in renal pathology WSIs.
[0020] Second, the system of the present invention can effectively optimize computational efficiency (a bottleneck in ultra-high-resolution image processing). The system of the present invention proactively removes meaningless renal medullary regions through a preprocessing module, effectively reducing the amount of subsequent data processing. Furthermore, the system slices the WSI into first-resolution images that can be processed in parallel, enabling efficient batch processing on the GPU, further improving processing efficiency. Furthermore, the lesion segmentation network of the present invention utilizes a full MLP decoder (SegFormer architecture), which has fewer parameters than the U-Net network, enabling a lightweight decoding design and further improving processing efficiency.
[0021] Third, the system of the present invention has good clinical applicability. Through a hierarchical network module, the system of the present invention can simultaneously output lesion grade and lesion proportion based on the slice image, lesion segmentation mask, spliced mask image, and prior information on lesion proportion, thus adapting to clinical needs.
[0022] Fourth, the lesion segmentation network module and hierarchical network module of the system of the present invention can improve the generalization ability of the system of the present invention through self-supervised pre-training (the hierarchical feature extraction submodule in the hierarchical network module can learn the common pathological features of unlabeled data through similarity loss) and mask-guided attention (using the segmentation mask as the attention weight so that the model can still focus on key areas with a small amount of annotation), so that it can be applied to small samples and overcome the problem of scarce labeled data.
[0023] In general, the system of the present invention, through its integrated architecture of "multi-source input-hierarchical processing-lesion classification", can achieve simultaneous breakthroughs in computational accuracy, computational efficiency, clinical adaptability, and generalization capability. In particular, it can effectively address the processing bottlenecks of ultra-high-resolution WSI (through segmentation + renal medulla elimination), the problem of lesion feature dispersion (through lesion perception attention + prior information guidance), and the subjectivity of manual evaluation (using standard quantitative output) in existing related technologies.
[0024] 2. Other beneficial effects or advantages of the present invention will be described in detail in the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative labor.
[0026] in: Figure 1 1 is a schematic diagram of a process framework of a grading system for renal tubular and interstitial lesions provided by an exemplary embodiment of the present invention; Figure 2 FIG1 is an example of a visualization result of lesion segmentation provided by an exemplary embodiment of the present invention, in which the dark block is the IFTA area; Figure 3 1 is a schematic diagram of the MLP layer framework of an IFTA segmentation network provided by an exemplary embodiment of the present invention; Figure 4 1 is a schematic diagram of an IFTA segmentation network feature extraction module framework provided by an exemplary embodiment of the present invention; Figure 5 1 is a schematic diagram of a network framework of an IFTA hierarchical feature extraction module pre-training method provided by an exemplary embodiment of the present invention; Figure 6 is a schematic diagram of a network framework of a lesion-aware attention module provided by an exemplary embodiment of the present invention; Figure 7 1 is an example of a kidney pathology image corresponding to three lesion grades provided by an exemplary embodiment of the present invention. The dark block in this figure is the IFTA area. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.
[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.
[0029] In order to facilitate relevant technical personnel to have a clearer and more accurate understanding of the technical solution of the present invention, the existing related technologies and the technical problems existing therein are first described in more detail below.
[0030] Renal biopsy is the gold standard for diagnosing kidney disease. Its primary focus is microscopic observation of tissue sections and pathological analysis of the specimens. With the advent of digital scanners, information from renal biopsy tissue has been digitally converted from glass slides to whole-slide images (WSIs). As ultra-high-resolution pathological images, WSIs contain a wealth of information on tissue morphology and staining.
[0031] Among the various histopathological manifestations of renal failure caused by chronic kidney disease, tubular atrophy and interstitial fibrosis (IFTA) are key indicators of disease severity and prognosis. IFTA is commonly seen in progressive renal diseases, including diabetic nephropathy, hypertensive nephrosclerosis, IgA nephropathy, and glomerulonephritis. It involves structural changes in the renal interstitium and tubules, reflecting the underlying pathology of the kidney. Tubular atrophy refers to a decrease in the number and size of functional tubules, leading to decreased renal function. Interstitial fibrosis refers to the excessive accumulation of fibrous connective tissue in the interstitial spaces between tubules, replacing normally functioning renal tissue. This causes compression and damage to adjacent tubules, disrupting their blood supply and ultimately leading to their atrophy. The resulting loss of functional tubules may lead to a decrease in the production of tubular factors that help maintain the health of the surrounding interstitial tissue, disrupting the normal balance of interstitial cell signaling and repair mechanisms, and ultimately leading to increased fibrosis. Interstitial fibrosis and tubular atrophy interact with each other, forming a vicious cycle, accelerating the progression of chronic renal failure and ultimately leading to renal impairment.
[0032] Therefore, understanding and accurately assessing the extent of IFTA lesions is crucial for the diagnosis and prognosis of patients with chronic renal failure. The Banff renal transplant pathology scoring system is an international consensus classification system for evaluating renal transplant pathology. This system provides a standardized diagnosis and grading method for pathological changes after renal transplantation.
[0033] Deep Convolutional Neural Networks (DCNNs) can achieve end-to-end automatic learning and extract image features at various levels, making significant progress in medical image processing. The paper "Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation [C] Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015: 234-241" first proposed the U-net architecture for segmentation. Using skip connections to fuse multi-level features between the encoder and decoder, it advanced semantic pixel-wise segmentation in medical images. However, the large number of parameters introduced by U-net increases the computational burden. DCNNs have also been used to analyze glomeruli and other pathological tissues in pathological images of renal biopsies (WSIs). The paper "Ginley B, Jen KY, Han SS, et al. Automated computational detection of interstitial fibrosis, tubular atrophy, and glomerulosclerosis[J]. Journal of the American Society of Nephrology, 2021, 32(4): 837-850" uses DeepLab v2 to automatically segment and quantify atrophic tubules, fibrotic tubulointerstitial tissue, non-sclerotic glomeruli, and sclerotic glomeruli in WSI image patches. However, the DeepLab v2 network lacks a global and contextual understanding of tissue lesion features. When lesions contain multiple tissue components, the network has difficulty accurately segmenting them.The paper "Marechal E, Jaugey A, Tarris G, et al. Automatic evaluation of histological prognostic factors using two consecutive convolutional neural networks on kidney samples[J]. Clinical Journal of the American Society of Nephrology, 2022, 17(2): 260-270." designed two consecutive CNNs to obtain renal histological prognostic features and predict the severity of interstitial fibrosis and tubular atrophy respectively. However, interstitial fibrosis and tubular atrophy often coexist and interact with each other. Comprehensively evaluating IFTA lesions can capture the dynamic interactions between lesions for a more comprehensive analysis.
[0034] Currently, the assessment and grading of IFTA primarily relies on renal pathology reports generated by pathologists through microscopic interpretation of renal biopsy sections. Manual assessment of IFTA is time-consuming and labor-intensive, and the evaluation process is highly subjective. Achieving consistent assessment results across different laboratories or medical institutions is difficult because pathologists' interpretation and scoring of the extent of IFTA lesions may vary based on individual differences and time, leading to a lack of consistency and reproducibility in the assessment results. Furthermore, interstitial fibrosis and atrophic tubules often coexist in renal disease and interact to accelerate the progression of chronic renal failure. Comprehensive assessment of IFTA lesions can capture the dynamic interactions between lesions, allowing for a more comprehensive analysis.
[0035] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.
[0036] like Figures 1 to 7As shown, according to the first aspect of the present invention, a grading system for renal tubular and interstitial lesions is provided, comprising a preprocessing module, a slice processing module, a lesion segmentation network module, a mask splicing module and a grading network module. The preprocessing module is used to segment the renal cortex and renal medulla regions in the full-volume renal pathology image and to exclude the renal medulla region. The slice processing module is used to segment the preprocessed renal cortex region image into a slice image of a first resolution. The lesion segmentation network module is based on the SegFormer architecture and integrates self-attention and convolution layers, inputs a slice image and outputs an IFTA lesion segmentation mask. The mask splicing module splices the lesion segmentation mask into a mask image of a second resolution. The grading network module includes a three-stage feature extraction unit, whose input end receives the slice image, the lesion segmentation mask, the spliced mask image and the lesion ratio prior information, and the output end is connected to the classification unit to generate the IFTA lesion grading result.
[0037] Through the above-mentioned technical solution, the present invention discloses a multi-level IFTA lesion grading network. Combining lesion region information predicted by the IFTA segmentation network and prior knowledge of lesion proportion, it can accurately grade lesions according to their severity. For ultra-high-resolution renal pathology WSIs, the proposed lesion grading network is designed to extract mapped features layer by layer, simultaneously focusing on global and local lesion features. To address the problem of IFTA features being scattered and close to background features, a lesion-aware sub-attention module is introduced to enhance lesion region-related features. Experimental results demonstrate that the proposed method can accurately assess the severity of IFTA lesions, achieving a 91.18% accuracy rate for grading IFTA lesions in renal pathology WSIs. The recognition performance of the proposed method for three lesion categories based on three evaluation metrics (recall, precision, and F1 score) is shown in Table 1 below. This demonstrates that the proposed grading network model can effectively identify the severity of IFTA lesions in renal pathology WSIs, providing efficient and accurate lesion grading information for renal pathologists and clinicians.
[0038] Table 1 Classification accuracy table Specifically, through the above technical solution, the system of the present invention can significantly improve grading accuracy. By simultaneously inputting slice images (local details), lesion segmentation masks (spatial positioning), medium-resolution masks (regional correlation), and lesion proportion priors (global severity), the system forms a multi-dimensional feature complementarity, effectively achieving a 91.18% grading accuracy for IFTA lesions in renal pathology WSIs.
[0039] Second, the system of the present invention can effectively optimize computational efficiency (a bottleneck in ultra-high-resolution image processing). The system of the present invention proactively removes meaningless renal medullary regions through a preprocessing module, effectively reducing the amount of subsequent data processing. Furthermore, the system slices the WSI into first-resolution images that can be processed in parallel, enabling efficient batch processing on the GPU, further improving processing efficiency. Furthermore, the lesion segmentation network of the present invention utilizes a full MLP decoder (SegFormer architecture), which has fewer parameters than the U-Net network, enabling a lightweight decoding design and further improving processing efficiency.
[0040] Third, the system of the present invention has good clinical applicability. Through a hierarchical network module, the system of the present invention can simultaneously output lesion grade and lesion proportion based on the slice image, lesion segmentation mask, spliced mask image, and prior information on lesion proportion, thus adapting to clinical needs.
[0041] Fourth, the lesion segmentation network module and hierarchical network module of the system of the present invention can improve the generalization ability of the system of the present invention through self-supervised pre-training (the hierarchical feature extraction submodule in the hierarchical network module can learn the common pathological features of unlabeled data through similarity loss) and mask-guided attention (using the segmentation mask as the attention weight so that the model can still focus on key areas with a small amount of annotation), so that it can be applied to small samples and overcome the problem of scarce labeled data.
[0042] In general, the system of the present invention, through its integrated architecture of "multi-source input-hierarchical processing-lesion classification", can achieve simultaneous breakthroughs in computational accuracy, computational efficiency, clinical adaptability, and generalization capability. In particular, it can effectively address the processing bottlenecks of ultra-high-resolution WSI (through segmentation + renal medulla elimination), the problem of lesion feature dispersion (through lesion perception attention + prior information guidance), and the subjectivity of manual evaluation (using standard quantitative output) in existing related technologies.
[0043] The system of the present invention is described below with reference to an exemplary embodiment.
[0044] like Figure 1 The figure shows the overall process framework of lesion-assisted assessment provided by the present invention. The technical solutions adopted by the present invention include: 1. IFTA Segmentation Network First, the renal pathology WSI is input into a pre-segmentation network (preprocessing module) for segmentation of the renal medulla and renal cortex. By eliminating the renal medulla region in the WSI, the renal medulla region containing complex structures and tissues is prevented from interfering with subsequent segmentation and classification tasks. The pre-segmentation network uses the U-Net framework. This step is the preprocessing of renal pathology WSIs.
[0045] First, the size of the kidney pathology WSI slices after removing the renal medulla area is selected, with the length as H and the width as W as the benchmark, and the ultra-high resolution kidney pathology WSI slices are divided into The low-resolution image is then input into the IFTA segmentation network to obtain the predicted mask of the low-resolution image.
[0046] The IFTA segmentation network segments the IFTA lesion area based on the SegFormer semantic segmentation framework. SegFormer adopts a hierarchical design, combining the Transformer architecture and a lightweight multi-layer perceptron (MLP) decoder, and is capable of multi-scale feature learning. In this invention, the self-attention feature extraction module in the encoder is replaced by a feature extraction module of self-attention plus convolutional layers to capture global and local feature information simultaneously. Its network architecture is as follows Figure 1 The IFTA segmentation network region is shown in Figure 4.
[0047] The encoder of the IFTA segmentation network uses overlapping patch embedding and merging operations instead of non-overlapping patch embedding and merging in the standard Transformer. The entire encoder processes the image in four stages, with different feature map sizes and number of channels in each stage. The size of the input image is First, the image is segmented into blocks through the overlapping block embedding layer, and then enters the segmentation feature extraction module after downsampling, which is the first stage. Then, the features are downsampled and enter the next stage again. Each stage has a segmentation feature extraction module. The feature map size extracted by each feature extraction module is as follows: Where, is the number of the feature extraction module, is the number of channels, and .
[0048] The structure of the lesion segmentation network module is as follows Figure 4 As shown in the figure, the input feature map is first normalized, then fed into the self-attention module and convolutional layer to capture global context information, and the input features are summed. The expression of self-attention is shown in the following formula.
[0049] in, They are the query vector, key vector, and value vector in the weight calculation, which are all obtained by linearly transforming the input elements through the weight matrix; is the dimension of the key vector; Represents the softmax function, which converts the attention score into a value between 0 and 1 and sums it to 1, thereby obtaining the attention weight. The present invention introduces a scaling factor from the attention module , for the sequence Reduction is performed to reduce computational complexity.
[0050] First, the size is sequence After the Reshape operation, the feature map has a size of The feature map is then passed through the linear layer Will The size is converted to The output features are then passed through the Mix-FNN layer and the feature image blocks are merged. The difference from the standard Transformer feedforward network is that a The depthwise separable convolutional layer can capture the positional relationship between image blocks without position encoding. The Mix-FNN operation expression is as follows: Among them, GELU() is the Gaussian error linear unit activation function, is the input of the Mix-FNN layer, which is the sum of the features output by the self-attention module and the convolutional layer.
[0051] The decoder of the IFTA segmentation network is a lightweight full MLP design with the following structure: Figure 3 First, the multi-scale features output by the encoder in the four stages are received, and each feature map is processed by a separate MLP layer to unify the feature maps of different dimensions into a dimension Then upsample the feature maps of different sizes to get four groups of sizes. Then concatenate the upsampled feature maps into a set of size Finally, an MLP layer is used to map the fused features to semantic categories to obtain the segmentation mask of the IFTA lesion.
[0052] The loss function used in the optimization process of the segmentation network is a composite loss function, which uses the cross entropy loss function, the focus loss function and the recall loss function. The expression is: Where, is the standard cross entropy loss, is the focal loss, To recover the loss, 、 and is an adjustable parameter. The IFTA segmentation network predicts the lesion area for all slices and outputs the segmentation mask prediction result. By adding the predicted segmentation mask to the original image, the visualization result of the lesion segmentation can be obtained, such as Figure 2 shown.
[0053] The training model used ground-truth images of the renal cortex and medulla, as well as images of IFTA lesions, annotated by professional renal pathologists and clinicians. Both segmentation network models predicted segmentation results that were largely consistent with the ground-truth regions.
[0054] 2. IFTA Classification Network like Figure 1 As shown. The low-resolution image after slicing is input into the lesion segmentation network module of the first stage, and the lesion segmentation mask obtained by the IFTA segmentation network is input into the lesion perception sub-attention module of the first stage; The lesion segmentation mask is spliced to obtain a size of The lesion segmentation mask is used as the input of the medium-resolution lesion perception sub-attention module in the second stage. The lesion segmentation mask is spliced into a complete lesion segmentation mask corresponding to the original pathological image, and the proportion of the lesion area is input into the hierarchical network of the third stage as prior information.
[0055] Phase 1: This phase takes as input the low-resolution pathological image after sectioning and the lesion segmentation mask obtained in the IFTA segmentation network. The hierarchical feature extraction network is used to obtain context information, and the lesion perception sub-attention module uses the lesion mask information for feature weighting. The input feature map dimension of this phase is , the output feature map dimension after the lesion-aware sub-attention module is .
[0056] The second stage: The feature map extracted in the previous stage is transformed in size and dimension through the MLP layer to obtain The feature map is input into the hierarchical feature extraction network of the second stage, and the network weight of the lesion area is further strengthened by the lesion perception sub-attention module. The final output feature map dimension is .
[0057] The third stage: Similarly, the image features extracted in the previous stage are transformed into different sizes and dimensions through the MLP layer to obtain The feature map is input into the feature extraction network of the third stage, which extracts global and local features. The lesions are finally classified by combining the input lesion prior information in the classification layer. The prior knowledge is the lesion ratio of the renal cortex segmentation mask obtained by the IFTA segmentation network pre-segmentation to the IFTA lesion area mask. , The calculation expression is: in, Indicates the IFTA lesion area, Represents the renal cortex area. The input method is to initialize a priori knowledge auxiliary vector with the same length as the feature vector, and all elements of the vector are , concatenated with the eigenvector.
[0058] The feature extraction module is first trained on a pretrained network using self-supervised contrastive learning, and the weights of this network are transferred to the IFTA classification network. The training process fine-tunes only the classification layer and the feature extraction module in the third stage. This allows the classification model to more effectively adapt to the classification task while retaining the common features of pathology images learned during pretraining. The final SoftMax function constrains the output predictions to be between 0 and 1 and converts them into classification probabilities: Where, represents the probability that the i-th sample is predicted to be category c, Indicates that the hierarchical network predicts the i-th sample as the c-class output prediction value; k represents the k-th category, The number of categories representing the lesion grade, It indicates that the hierarchical network predicts the i-th sample as the output prediction value of the k-th category. In this invention, the lesions are divided into three levels: T0, T1 and T2. , the specific cross entropy loss function is as follows.
[0059] in, Is a sign function, indicating whether the i-th sample belongs to category c, if it does, it takes 1, and if it does not, it takes 0; The base of is e.
[0060] 3. Pre-training of lesion segmentation network module The lesion segmentation network module uses a self-supervised contrastive learning method to train unlabeled kidney pathology slice images. Figure 6The framework of the pre-trained network is presented. Each slice image is enhanced to generate two variants using different enhancement methods. A similarity loss function is used to update the network weights to enhance the feature similarity of identical pathological tissues and reduce the feature similarity of different pathological tissues, thereby improving the network's ability to extract effective features of renal pathology. Finally, the weights of the trained hierarchical feature extraction network are transferred to the lesion grading network of the present invention. The specific training process is as follows: First, input an unlabeled kidney pathology slice image, assuming a slice image is , after two different random enhancement methods, the enhanced samples are obtained and Two enhanced samples and Input the online network and target network respectively. Among them, the weight of the online network is , the weight of the target network is .sample First, a hierarchical feature extraction network is used in the online network Extract the characteristics of kidney pathology and map them to the feature space to obtain feature information ; Then, through the MLP layer Map the features to get Similarly, another enhanced sample Through the target network, first by Extract features and map them to feature space to obtain feature information , and then pass through the corresponding MLP layer Get the mapping When the feature space mapping of two samples and After all is done, the feature map of the online network It passes through an additional MLP layer Processing is performed to obtain the target network feature mapping prediction results.
[0061] During the pre-training process, similarity loss is used to update the network weights. Similarity loss is the mean square error between the normalized online network predicted feature map and the target network actual map, and its expression is as follows: in, The operation represents L2 normalization; yes The mathematical expectation of It is the prediction head Online network feature information The prediction results, It is an enhanced sample feature information extracted by the online network; is the mathematical expectation, is another augmented sample feature information extracted by the target network; represents calculating the inner product of both; respectively represent the updatable weights of the online network and the target network.
[0062] In training, the network weights of the online network are updated by the similarity loss. The update of the online network will affect the weight adjustment of the same architecture part in the target network. Specifically, the following formula is used to update the parameters of the online network: ; the following formula is used to update the parameters of the target network: ; wherein, represents a network optimizer; represents the mean square similarity loss of the gradient of the online network parameters ; represents the learning rate of network training; represents the target decay rate. The target decay rate of each network iteration can be calculated by the following formula: ; in the formula, represents the initial value of the target decay rate, represents the current training round; represents the maximum training round.
[0063] IV. Lesion-aware sub-attention module The lesion-aware sub-attention module performs a weighted attention operation using the segmentation mask of the lesion area, Figure 6 is a network structure diagram of the lesion-aware sub-attention module. In this way, the model can allocate different attention weights to different regions in the image according to the mask information of the lesion area, thereby strengthening the representation ability of the lesion area features and improving the importance of these features in the model.
[0064] The input of this module includes the lesion segmentation mask and the pathological feature vector extracted from the lesion segmentation network module, and the feature map obtained after adjusting the scale through the MLP layer. The input mask is a binary mask, which defines a clear boundary for the lesion area in the pathological image. However, in IFTA, there is usually some ambiguity and uncertainty in the boundary between the lesion area and the non-lesion area. Therefore, the lesion-aware sub-attention module performs a Sigmoid transformation on the input segmentation mask to obtain , and then performs normalization processing to make the pixel value of the final segmentation mask smoothly transition between 0 and 1. The specific transformation process is as follows: This smooth transition process makes the boundary of the lesion area softer, reduces the uncertainty of the boundary, and helps the attention mechanism focus on the lesion area more accurately.
[0065] Next, the input feature map Upsample to match the normalized segmentation mask Match, then perform pixel-by-pixel multiplication and apply the mask weighting to the feature map to obtain the mask-weighted attention feature map Finally, the output feature map Perform a global maximum pooling operation to obtain the output feature map of the lesion-aware attention network , the specific process is as follows.
[0066] in, Represents the global maximum pooling operation; Represents matrix lattice operations.
[0067] In one embodiment of the present invention, the lesion segmentation network module may include an encoder unit and a decoder unit. The encoder unit uses overlapping block embedding to extract features in four stages, each stage including a segmentation feature extraction submodule. The decoder unit uses a multilayer perceptron to uniformly dimension, upsample, and concatenate the four-stage features, outputting a segmentation mask. The segmentation feature extraction submodule integrates a self-attention mechanism with a 3×3 depthwise separable convolutional layer and introduces a scaling factor to reduce computational complexity.
[0068] In this implementation, the four-stage encoder design integrates multi-scale features, addressing the recognition challenges of the variable (dispersed / diffuse) morphology of IFTA lesions. Furthermore, the introduction of a scaling factor reduction sequence effectively reduces the amount of self-attention computation, thereby optimizing computational efficiency. Furthermore, a 3×3 depthwise separable convolutional layer replaces the positional encoding, addressing the Transformer's insensitivity to image patch position, preserving positional information and improving the accuracy of renal tubule boundary segmentation.
[0069] In one embodiment of the present invention, the three-stage structure of the hierarchical network module of the present invention may be: The first stage includes a hierarchical feature extraction submodule and a lesion perception attention submodule, and receives a slice image of the first resolution and a lesion segmentation mask as input. The second stage receives the feature map output by the first stage, adjusts it through the MLP layer, and inputs it into the hierarchical feature extraction submodule and the lesion perception attention submodule; The third stage: Receive the feature map output from the second stage, adjust it through the MLP layer, input it into the grading feature extraction submodule, and generate the grading result in combination with the prior information of the lesion ratio.
[0070] In this implementation, the first stage input features are slice images and lesion segmentation masks, which capture local lesion details and thus compress the feature map size. The second stage input features are medium-resolution masks, which establish regional associations and thus reduce the number of parameters. The third stage input features are global masks and lesion ratio prior information, which integrate global context with standard quantization values to improve classification accuracy.
[0071] In one embodiment of the present invention, the lesion-aware attention submodule of the present invention may include a mask processing unit, a feature map upsampling unit, a weighting unit, and a global maximum pooling unit. The mask processing unit performs a Sigmoid transform and normalization operation on the input lesion segmentation mask. The feature map upsampling unit adjusts the feature map size to match the mask size. The weighting unit performs a pixel-by-pixel matrix multiplication operation between the normalized mask and the upsampled feature map. The global maximum pooling unit compresses the weighted feature map and outputs it to the next stage.
[0072] In this implementation, the lesion-aware attention submodule performs Sigmoid transformation and normalization operations on the lesion segmentation mask, so that the lesion boundary weight transitions continuously from 0 to 1, which is more in line with the actual pathology and improves feature discrimination. At the same time, the global maximization unit can compress the feature map into a vector, avoiding high-dimensional matrix operations and improving the calculation rate.
[0073] In one embodiment of the present invention, the mask processing unit of the present invention performs: So that the binary mask smoothly transitions to the [0,1] interval; In the above formula, is the input segmentation mask, for The segmentation mask obtained by Sigmoid transformation, is the final segmentation mask.
[0074] In this way, the final segmentation mask generated by this implementation enables the attention mechanism to give higher weights to the lesion core area and medium weights to the transition area, realize boundary uncertainty modeling, and reduce the false activation of similar tissues such as sclerotic glomeruli.
[0075] In one embodiment of the present invention, the hierarchical feature extraction submodule of the present invention initializes weights through a pre-trained network, which includes an online network, a target network, a feature extractor, an MLP layer, and a similarity loss calculation unit. The online network and the target network receive different enhanced variants of the same kidney pathology slice. The feature extractor and the MLP layer are used to generate feature vectors. The similarity loss calculation unit is based on the formula Update the network weights, where The operation represents L2 normalization; yes The mathematical expectation of It is the prediction head Online network feature information The prediction results, It is an enhanced sample Feature information extracted by online network; yes The mathematical expectation of Another enhanced sample Feature information extracted by the target network; The operation represents the calculation of the inner product of the two; denote the updateable weights of the online network and the target network, respectively.
[0076] In this embodiment, the similarity loss By forcibly enhancing variant features, the model can achieve higher accuracy with a smaller proportion of labeled data. At the same time, the pre-trained feature extraction module can identify interference factors, improve feature generalization, and reduce the impact of clinical data differences.
[0077] In one embodiment of the present invention, the classification unit of the hierarchical network module of the present invention may include a fully connected layer, a SoftMax function and a cross entropy loss calculation unit. The fully connected layer and the SoftMax function output probability distribution The cross entropy loss calculation unit is based on the formula Optimize the parameters. represents the probability that the i-th sample is predicted to be category c, Indicates that the hierarchical network predicts the i-th sample as the c-class output prediction value, k represents the k-th category, is the number of categories for lesion classification, Indicates that the hierarchical network predicts the i-th sample as the output prediction value of the k-th category, Is a symbolic function that indicates whether the i-th sample belongs to category c. If it does, it takes 1, and if it does not, it takes 0.
[0078] In this embodiment, the cross entropy loss calculation unit weightedly processes the category imbalance problem, which can effectively improve the recall rate.
[0079] In one embodiment of the present invention, the lesion ratio prior information calculation unit of the present invention performs: And the scalar Expand to an auxiliary vector of the same length as the feature map and input it into the third stage; In the above formula, is the area of IFTA lesion, is the area of the renal cortex.
[0080] In this embodiment, the scalar Converting it into an auxiliary vector of the same length as the feature map enables the model to display the learning threshold boundary (for example, the Banff threshold boundary), so that samples of different levels can be correctly distinguished, avoiding the "critical misjudgment" common in manual evaluation.
[0081] In one embodiment of the present invention, the grading system for renal tubular and interstitial lesions of the present invention may further include a visualization output module, which is used to superimpose the lesion segmentation mask onto the original pathological image and synchronously display the quantification results and the lesion area ratio value.
[0082] This implementation effectively improves clinical diagnostic efficiency. For example, by superimposing a lesion segmentation mask on the original pathology image, doctors can quickly locate the IFTA area. Simultaneously, the quantification results and the lesion area ratio are displayed simultaneously, reducing the doctor's quantitative calculation time and enabling the direct generation of corresponding pathology reports.
[0083] In one embodiment of the present invention, the grading results of the present invention include T0, T1 and T2 levels; wherein, the standard for T0 level is: the proportion of IFTA lesion area is <25%; the standard for T1 level is: 25%≤IFTA lesion area proportion≤50%; the standard for T2 level is: the proportion of IFTA lesion area is >50%.
[0084] Binding to the Banff standard (25% / 50%) can effectively improve clinical acceptance and make the grading results consistent with the judgment of pathologists.
[0085] The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.
Claims
1. A grading system for renal tubular and interstitial lesions, characterized in that: include: The preprocessing module is used to segment the renal cortex and medulla in the full-volume renal pathology images and remove the renal medulla area; a slice processing module, configured to segment the preprocessed renal cortex region image into slice images of a first resolution; A lesion segmentation network module, based on the SegFormer architecture and integrating self-attention and convolutional layers, inputs the slice image and outputs the IFTA lesion segmentation mask; a mask splicing module, splicing the lesion segmentation masks into a mask image of a second resolution; The grading network module includes a three-stage feature extraction unit, an input end of which receives the slice image, the lesion segmentation mask, the spliced mask image and the lesion ratio prior information, and an output end is connected to the classification unit to generate the IFTA lesion grading result.
2. The grading system for renal tubular and interstitial lesions according to claim 1, wherein: The lesion segmentation network module includes: The encoder unit,uses overlapping block embedding operation to extract features in four stages,,each stage contains a segmentation feature extraction submodule; The decoder unit uses a multi-layer perceptron to unify the dimensions of the four-stage features, upsample and concatenate them, and output a segmentation mask; The segmentation feature extraction submodule integrates the self-attention mechanism with a 3×3 depth-separable convolutional layer, and introduces a scaling factor to reduce computational complexity.
3. The grading system for renal tubular and interstitial lesions according to claim 1, wherein: The three-stage structure of the hierarchical network module is: The first stage includes a hierarchical feature extraction submodule and a lesion perception attention submodule, and receives a slice image of the first resolution and a lesion segmentation mask as input. The second stage receives the feature map output by the first stage, adjusts it through the MLP layer, and inputs it into the hierarchical feature extraction submodule and the lesion perception attention submodule; The third stage: Receive the feature map output from the second stage, adjust it through the MLP layer, input it into the grading feature extraction submodule, and generate the grading result in combination with the prior information of the lesion ratio.
4. The grading system for renal tubular and interstitial lesions according to claim 3, wherein: The lesion-aware attention submodule includes: The mask processing unit performs Sigmoid transformation and normalization operations on the input lesion segmentation mask; Feature map upsampling unit, which adjusts the feature map size to match the mask size; The weighting unit performs the pixel-by-pixel matrix multiplication of the normalized mask and the upsampled feature map; The global maximum pooling unit compresses the weighted feature map and outputs it to the next stage.
5. The grading system for renal tubular and interstitial lesions according to claim 4, characterized in that: The mask processing unit performs: So that the binary mask smoothly transitions to the [0,1] interval; In the above formula, is the input segmentation mask, for The segmentation mask obtained by Sigmoid transformation, is the final segmentation mask.
6. The grading system for renal tubular and interstitial lesions according to claim 3, wherein: The hierarchical feature extraction submodule initializes weights through a pre-trained network, which includes: The online network and the target network receive different enhanced variants of the same kidney pathology slice; Feature extractor and MLP layer, used to generate feature vectors; Similarity loss calculation unit, according to the formula Update the network weights, where The operation represents L2 normalization; yes The mathematical expectation of It is the prediction head Online network feature information The prediction results, It is an enhanced sample Feature information extracted by online network; yes The mathematical expectation of Another enhanced sample Feature information extracted by the target network; The operation represents the calculation of the inner product of the two; denote the updateable weights of the online network and the target network, respectively.
7. The grading system for renal tubular and interstitial lesions according to claim 3, wherein: The classification unit of the hierarchical network module includes: Fully connected layer and SoftMax function, output probability distribution ; Cross entropy loss calculation unit, according to the formula Optimize parameters; In the above formula, represents the probability that the i-th sample is predicted to be category c, Indicates that the hierarchical network predicts the i-th sample as the c-class output prediction value, k represents the k-th category, is the number of categories for lesion classification, Indicates that the hierarchical network predicts the i-th sample as the output prediction value of the k-th category, Is a sign function, indicating whether the i-th sample belongs to category c. If it does, it takes 1, and if it does not, it takes 0.
8. The grading system for renal tubular and interstitial lesions according to claim 1, wherein: The lesion ratio prior information calculation unit performs: And the scalar Expand to an auxiliary vector of the same length as the feature map and input it into the third stage; In the above formula, is the area of IFTA lesion, is the area of the renal cortex.
9. The grading system for renal tubular and interstitial lesions according to claim 1, wherein: The renal tubular and interstitial lesion grading system further includes a visualization output module, which is used to: The lesion segmentation mask is superimposed on the original pathological image, and the quantification results and the lesion area ratio are displayed simultaneously.
10. The grading system for renal tubular and interstitial lesions according to claim 1, wherein: The grading results include T0, T1 and T2 levels; among them, the standard for T0 level is: the proportion of IFTA lesion area is <25%; the standard for T1 level is: 25%≤IFTA lesion area proportion≤50%; the standard for T2 level is: IFTA lesion area proportion>50%.