A multi-slice hierarchical glomerulus identification and matching method based on deep learning

Abstract

A multi-section hierarchical glomerulus identification and matching method based on deep learning comprises the following steps: performing glomerular instance segmentation on a multi-section hierarchical image to be identified using a glomerular instance segmentation model; based on the glomerular instance segmentation results, matching glomeruli between two layers of the multi-section hierarchy using the glomerular matching model, thereby achieving matching of glomeruli at each section hierarchical level; wherein the glomerular instance segmentation model and the glomerular matching model are trained using renal biopsy sample images with multi-section hierarchies. The identification and matching method of the present invention achieves accurate identification and matching performance of glomeruli at multi-section hierarchies, significantly reducing doctors' time in searching for glomeruli, thereby helping them focus more on lesion diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of tissue image processing, and in particular to a multi-section hierarchical glomerulus recognition and matching method based on deep learning. Background Art

[0002] Chronic kidney disease (CKD) is one of the leading causes of death worldwide, leading to renal failure or cardiovascular complications and threatening one-tenth of the world's population. Risk factors for CKD include diabetes, hypertension, and hyperlipidemia. The key to treatment is identifying and controlling the cause to mitigate kidney damage. Due to its complexity and progressive nature, accurate diagnosis of the cause is difficult based on clinical manifestations, laboratory tests, and imaging studies. However, renal biopsy histopathology can provide strong support.

[0003] Glomerular lesions provide crucial information on the etiology of kidney disease, and identifying various pathological manifestations within the glomerulus, including inflammation, sclerosis, proliferation, and necrosis, is crucial. Generally speaking, the diameter of an adult glomerulus is approximately 100 microns, while the distance between consecutive pathological sections is approximately 3-5 microns. Therefore, preparing as many consecutive sections as possible can help avoid missing important pathological changes. Furthermore, different histological stains, such as H&E, PAS, Masson's trichrome (MT), and PASM stains, can highlight the distinct histological structures of the glomerulus, thereby distinguishing different types of pathological changes. Physicians typically identify corresponding glomeruli across multiple sections and stains (multistained images). Based on the positional correspondence between these multiple levels, glomeruli are matched, thereby aggregating lesions within a single glomerulus across multiple levels. Multiple levels serve as a reference, highlighting pathological changes that might be overlooked on a single level. Furthermore, the three-dimensional structural information contained in these multiple levels can reveal pathological changes in spatial dimensions that are not readily discernible on a single plane.

[0004] Multiple levels and staining are necessary but limit the overall efficiency of glomerular disease diagnosis. Increasing the number of tissue sections and integrating glomerular lesions from various levels is time-consuming, forcing renal pathologists to spend significant time identifying and matching glomeruli across multiple levels. Furthermore, this carries the potential risk of missed diagnoses. Because multiple glomeruli must be identified and matched, renal pathologists are unable to prioritize the diagnosis and integration of more valuable lesions.

[0005] Deep learning has been widely applied in several medical fields. It has achieved performance comparable to that of human pathologists in pathological image analysis, enabling basic tasks such as classification, segmentation, and detection, as well as aiding diagnosis. Its advantage lies in handling repetitive and tedious tasks, thereby improving diagnostic efficiency. Currently, the application of deep learning in renal pathology is limited to basic tasks such as segmentation, detection, and classification. Most research focuses on glomeruli, including segmentation and detection. Analyzing each stain or level in isolation is limited and fails to model the complete diagnostic workflow of renal pathologists, thereby improving efficiency. Some work has leveraged cross-modality to transfer information between different stains, including virtual staining, stain enhancement, and universal domain adaptation. These methods improve the generalization of models across different stains, reducing the burden of tedious visual inspection and brain analysis on pathologists. However, these methods lack the ability to match glomeruli across multiple levels or stains, requiring manual position-based glomerular matching across multiple levels, which is inefficient. Observation of multiple sections and multistained whole slide images (WSI) of a case has been unavoidable, time-consuming and laborious so far.

[0006] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a multi-section hierarchical glomerulus identification and matching method based on deep learning.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A multi-section hierarchical glomerulus identification and matching method based on deep learning includes the following steps:

[0010] Performing glomerular instance segmentation on the image to be identified at multiple slice levels by using the glomerular instance segmentation model;

[0011] Based on the glomerulus instance segmentation result, matching the glomeruli between two layers in the multi-section hierarchy by the glomerular matching model, and then achieving matching of the glomeruli in each section hierarchy;

[0012] The glomerulus instance segmentation model and the glomerulus matching model are trained using renal biopsy sample images with multiple slice levels;

[0013] Among them, the matching of glomeruli between the two levels includes a coarse matching stage and a fine matching stage. The coarse matching stage determines the matching relationship between glomeruli at two different levels through coherent point drift (CPD) based on the Gaussian mixture model, thereby determining the most likely matching source glomerulus for each target glomerulus. The fine matching stage determines the optimal matching glomerulus for each target glomerulus through distance calculation in the feature space based on the coarse matching results, and obtains the matching matrix of glomerular matching between the two levels; the matching matrices are fused to obtain the final matching matrix of glomeruli at each section level.

[0014] Further:

[0015] The multi-section layer includes 12 layers, and the staining of the first 6 and last 6 layers is H&E, PAS, MT, PASM, H&E, and PAS, respectively.

[0016] The glomerulus instance segmentation model is an instance segmentation model (MTM) based on the Mask R-CNN framework and the Transformer as the backbone. The MTM detects and segments glomeruli in renal biopsy images through a backbone network, a region proposal network (RPN), region of interest pooling (ROIPooling), and a multi-task prediction head.

[0017] The glomerulus instance segmentation includes:

[0018] The backbone network is used to extract features of renal biopsy images with different staining related to glomerular structure;

[0019] The outputs of each stage in the backbone network are fused through the feature pyramid network FPN, and the features of different scales are integrated into the region proposal network RPN;

[0020] The region proposal network (RPN) uses anchor points to calculate the obtained feature map and generate region proposals. The region of interest pooling extracts the remaining region proposals after position correction, elimination, and maximum value suppression (NMS).

[0021] Use a multi-task prediction head to classify and segment the extracted region proposals, and use bounding box regression to correct the proposed positions again;

[0022] Thus, the detection box position and semantic segmentation results of each glomerulus are obtained.

[0023] The overall optimization goal of the glomerulus instance segmentation model is as follows:

[0024] min L=L cls +L box +L mask

[0025] Among them, L cls and L box represents the classification and bounding box regression loss, L mask The binary cross entropy function is used, and its expression is as follows:

[0026]

[0027] Where N is the number of categories, y i and are the label and prediction results of the i-th category respectively.

[0028] Performing the coarse matching based on the coherent point drift (CPD) and the fine matching based on the hybrid distance metric includes:

[0029] For multiple aspect levels, select one level as the target level T, and the remaining levels as the source levels S. Assume that T and S have T k and S k glomeruli, then the positions of the glomeruli on T and S are respectively recorded as T glo ={t i |i∈[1,T k ]} and S g1o ={s j |j∈[1,S k ]};

[0030] The coarse matching determines the relationship between each T glo The most likely match N k S glo , that is, find Using CPD based on Gaussian mixture model GMM, the overall likelihood function of CPD is as follows:

[0031]

[0032] in, f θ As the drift function, an affine transformation function is used. During the drift process, the affine transformation parameter θ is solved to obtain the matching probability of each glomerulus on the source level and the specific glomerulus on the target level, thereby obtaining the overall posterior probability;

[0033] Through the fine matching, N obtained from the coarse matching k The best matching glomerulus is determined among the glomeruli, i.e. Finally got M S→T ={(s j ,t i )|j∈[1,S k ],i∈[1,T k ]}, where HD is a mixed distance metric space;

[0034] Get the matching matrix M with the i-th level as the target level i ∈R Tki*12 , where Tkn represents the total number of glomeruli detected at the i-th level, M i Each row represents the serial number of the same glomerulus at each level, and each level is taken as the target level in turn to obtain multiple matching matrices.

[0035] The mixed distance metric space is characterized by the Euclidean distance f ed , deep general feature f d and handcrafted features f h constitute;

[0036] The Euclidean distance feature f ed : The Euclidean distance feature score is defined as N k The reciprocal of the normalized distance between the glomerulus and the target glomerulus, i.e. ||st||2;

[0037] The deep general feature f d : reflects the low-level semantic features of the glomerulus. The deep universal feature similarity between two glomeruli is defined as the 1-norm of the feature difference, i.e. ||f ds -f dt ||;

[0038] The hand-crafted feature f h : Construct hand-crafted features based on instance segmentation to obtain contours, including Hu moment, area, perimeter, aspect ratio of the minimum circumscribed rectangle, and circumference of the minimum circumscribed circle, a total of 11 dimensional features f h ∈R 11 , the manual feature similarity score is defined as the 1-norm of the feature difference, i.e. ||f hs -f ht ||;

[0039] The distance in feature space is as follows:

[0040] HD(·)=α||st||2+β||f ds -f dt ||+γ||f hs -f ht ||.

[0041] The VCM method is used to cross-fuse each matching matrix to obtain the final matching matrix; specifically, it includes:

[0042] Initialize the count list based on each matching matrix and create a two-dimensional zero vector for each matching matrix;

[0043] Traverse all matching matrices and calculate the number of matching sequence numbers between the reference row and each row of the source matrix;

[0044] The voting method was used to take the mode in the glomerular number dimension and combine them into the final matching matrix.

[0045] A multi-section hierarchical glomerulus identification and matching system based on deep learning, comprising a processor, characterized in that the processor is configured to execute the multi-section hierarchical glomerulus identification and matching method based on deep learning.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-section hierarchical glomerulus identification and matching method based on deep learning.

[0047] The present invention has the following beneficial effects:

[0048] The present invention provides a multi-faceted hierarchical glomerulus identification and matching method based on deep learning, and constructs a glomerulus instance segmentation and matching system based on deep learning. Each case is cut into multiple continuous layers, which can include multiple stainings. The identification and matching method of the present invention has better performance indicators than other existing models. The proposed glomerulus instance segmentation and matching method achieves accurate identification and matching performance, which can save doctors a lot of time searching for glomeruli, allowing doctors to focus more on the diagnosis of lesions. The method proposed by the present invention can achieve accurate identification and matching of 12-level multistained glomeruli.

[0049] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of a multi-section hierarchical glomerulus identification and matching method based on deep learning according to an embodiment of the present invention.

[0051] Figure 2 A comparison chart of the visualization results of instance segmentation according to an embodiment of the present invention.

[0052] Figure 3 This is a matching effect diagram of an embodiment of the present invention.

[0053] Figure 4 Schematic diagram of a human-machine interface according to an embodiment of the present invention.

[0054] Figure 5 This is the multi-level glomerular output result after the human-machine interface is magnified according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0056] See Figure 1 In some embodiments, the present invention provides a multi-faceted hierarchical glomerulus identification and matching method based on deep learning, which may include the following steps:

[0057] Use renal biopsy sample images with multiple slice levels to train glomerular instance segmentation models and glomerular matching models;

[0058] Performing glomerular instance segmentation on the image to be identified at multiple slice levels by using the glomerular instance segmentation model;

[0059] Based on the glomerulus instance segmentation result, matching the glomeruli between two layers in the multi-section hierarchy by the glomerular matching model, and then achieving matching of the glomeruli in each section hierarchy;

[0060] Among them, the matching of glomeruli between the two levels includes a coarse matching stage and a fine matching stage. The coarse matching stage determines the matching relationship between glomeruli at two different levels through coherent point drift (CPD) based on the Gaussian mixture model, thereby determining the most likely matching source glomerulus for each target glomerulus. The fine matching stage determines the optimal matching glomerulus for each target glomerulus through distance calculation in the feature space based on the coarse matching results, and obtains the matching matrix of glomerular matching between the two levels; the matching matrices are fused to obtain the final matching matrix of glomeruli at each section level.

[0061] Specific embodiments of the present invention are further described below.

[0062] In a specific embodiment, a multi-faceted hierarchical glomerulus identification and matching method based on deep learning may include:

[0063] In step S0, renal biopsy samples were collected to construct a dataset containing healthy subjects and 13 types of chronic kidney disease. Each case was then cut into 12 consecutive sections (levels), with the first and last six levels stained identically, resulting in six slides for each case.

[0064] In step S1 , a high-throughput microscopy scanning instrument is used for imaging to obtain a whole slide image (WSI).

[0065] Step S2 constructs four datasets: Instance-Train, Instance-Test, Match-Train, and Match-Test. The WSI-level images are cut into 1024*1024 pixel patches, which are used for training the glomerular instance segmentation model, performance evaluation of the glomerular instance segmentation model, training the glomerular matching model, and performance evaluation of the glomerular matching method, respectively.

[0066] Step S3 uses the Mask R-CNN framework (MTM) embedded in the multistained transformer to implement the task of glomerulus instance segmentation.

[0067] In step S4, glomerular matching between two levels is achieved based on the CHDM algorithm, and accurate 12-level multistained glomerular matching is achieved based on the VCM algorithm.

[0068] Step S5 realizes fully automatic glomerulus recognition through a human-machine interface. At the same time, the human-machine interface can also automatically match the 12-level multistained glomerulus images and present the matching results.

[0069] A multi-faceted hierarchical glomerulus identification and matching method based on deep learning, the specific implementation steps include:

[0070] Step S0-0: 600 renal biopsy samples were collected from Xijing Hospital in Xi'an, China, covering a wide range of healthy subjects and 13 types of chronic kidney disease.

[0071] Step S0-1: To more comprehensively and accurately evaluate glomerular lesions and chronic kidney disease, each case was sequentially divided into 12 levels (named levels 1-12). The staining for the first 6 and last 6 levels was H&E, PAS, MT, PASM, H&E, PAS, etc., respectively.

[0072] In step S1-0, imaging was performed using a 600-slide high-throughput microscope scanner from Shenzhen Shengqiang Technology Co., Ltd., China, to obtain whole-slide images (WSIs) with an imaging resolution of 0.105 μm / pixel (80x equivalent magnification). The cases included 1200 H&E, 1200 PAS, 600 MT, and 600 PASM WSIs.

[0073] Step S2-0 constructs four datasets: Instance-Train, Instance-Test, Match-Train, and Match-Test. The WSI-level images are cut into 1024*1024 pixel patches, which are used for training the glomerular instance segmentation model, performance evaluation of the glomerular instance segmentation model, training the glomerular matching model, and performance evaluation of the glomerular matching method, respectively.

[0074] In step S3-0, in order to realize the position recognition and contour extraction of 12-level multistained glomeruli, an instance segmentation model MTM is proposed with Mask R-CNN as the framework and Transformer as the backbone.

[0075] MTM detects and segments glomeruli in renal biopsy images through backbone, region proposal network (RPN), region of interest pooling (ROI Pooling) and prediction head.

[0076] Step S3-1: First, features of differently stained renal biopsy images related to glomerular structures are efficiently extracted through the backbone.

[0077] Secondly, the outputs of different stages in the backbone are fused by the Feature Pyramid Network (FPN), that is, features of different scales are integrated into the input of RPN.

[0078] Then, RPN uses anchors to calculate the obtained feature map and generate region proposals. ROIPooling extracts the remaining region proposals after position correction, culling and maximum value suppression (NMS).

[0079] Finally, the multi-task prediction head is used to classify and segment the proposals, and the bounding box regression is used to correct the position of the proposals again.

[0080] After the above process, the detection box position and semantic segmentation results of each glomerulus are obtained. The overall optimization goal of the instance segmentation model is shown in the following formula.

[0081] min L=L cls +L box +L mask

[0082] Among them, L cls and L box represents the classification and bounding box regression loss. Lmask The binary cross entropy function is used, and its expression is shown in the following formula.

[0083]

[0084] Where N is the number of categories, y i and are the labels and prediction results of the i-th class respectively. y0 represents the background and y1 represents the glomerular pixels.

[0085] In order to assist pathologists in matching all corresponding glomeruli and thus reduce their burden, step S4-0 proposes a new feature matching algorithm, including a pairwise matching algorithm, namely, a coarse matching based on coherent point drift and a fine matching based on hybrid distance metric (CHDM), as well as a voting-based cross-fusion matching (VCM).

[0086] Step S4-1 CHDM process is defined as follows: for 12 levels, select one level as target level T, and the remaining levels as source level S. Assume that T and S have T k and S k glomeruli, then the positions of the glomeruli on T and S are respectively recorded as T glo ={t i |i∈[1,T k ]} and S glo ={s j |j∈[1,S k ]}. CHDM is divided into two stages, namely coarse matching and fine matching.

[0087] The purpose of step S4-2 rough matching is to determine the glo The most likely match N k S glo , that is, find Using CPD based on Gaussian mixture model (GMM), the overall likelihood function of CPD is shown as follows.

[0088]

[0089] in, f θ As the drift function, this embodiment uses an affine transformation function. Based on the Expectation-Maximization (EM) algorithm, during the drift process, the affine transformation parameter θ is solved to obtain the matching probability of each glomerulus on the source level and a specific glomerulus on the target level, thereby obtaining the overall posterior probability.

[0090] The purpose of step S4-3 fine matching is to obtain a more refined matching result. k The best matching glomerulus is determined among the glomeruli, i.e. Finally got M S→T ={(s j ,t i )|j∈[1,S k ],i∈[1,T k ]}. Among them, HD is a mixed distance metric space, which is composed of Euclidean distance features (f ed ), deep general features (f d ) and hand-crafted features (f h )constitute.

[0091] (1)f ed : In execution After that, t i and s j The closer the Euclidean distance between them, the higher the matching probability. The Euclidean distance feature score is defined as N k The reciprocal of the normalized distance between the first glomerulus and the target glomerulus, i.e. ||st||2.

[0092] (2)f d :This feature reflects the low-level semantic features of the glomerulus, thereby predicting the matching glomerulus from properties such as structure, color, and shape. The deep universal feature similarity between two glomeruli is defined as the 1-norm of the feature difference, i.e. ||f ds -f dt ||.

[0093] (3)f h :Since the morphological parameters of the same glomerulus at different levels are closer than those of different glomeruli as a whole, we constructed a manual feature of the contour obtained based on instance segmentation, including Hu moment, area, perimeter, aspect ratio of the minimum circumscribed rectangle and the minimum circumscribed circle circumference, a total of 11 dimensions of features f h ∈R 11 The manual feature similarity score is defined as the 1-norm of the feature difference, i.e. ||f hs -f ht ||.

[0094] The result of fine matching can be obtained from the feature space constructed by the above three features. The distance in the feature space is shown in the following formula.

[0095] HD(·)=α||st||2+β||f ds -f dt ||+γ||f hs -f ht ||

[0096] Step S4-4 is based on CHDM, and the matching matrix M with the i-th level as the target level can be obtained. i ∈R Tki *12 Among them, Tkn represents the total number of glomeruli detected at the i-th level, M i Each row represents the serial number of the same glomerulus on 12 levels. Taking level 1-12 as the target level, 12 matching matrices {M i |i∈[1,12]}. To further integrate the matching results between each level, the 12 matching matrices are fused to obtain the final matching matrix M. As a specific cross-fusion method, the following VCM algorithm is proposed.

[0097] Step S4-5 VCM algorithm process is as follows: There is a matching matrix set {M i |i∈[1,12]}, where Indicates T based on the i-th level ki Matching relationships, that is, each line is the serial number of the matching glomerulus from the 1st section to the 12th level; initialize a counting list Each element of the list is a two-dimensional zero vector.

[0098] Step S4-6 takes a matching matrix as the target and all other matching matrices as the source. Take each row of the target as the reference, traverse each row of the source, and obtain the number of matching numbers between the reference row and all rows of the current source. If N(i t ,i s ) within the maximum value maxN(i t ,i s ) is greater than the sum of the counting matrix elements and exist and Record the matching glomerulus numbers and in . And so on.

[0099] Step S4-7 uses a voting method to take the mode in the glomerular number dimension to obtain the optimal glomerular number matching result and combine it into the final matching matrix M. Each row of M represents the glomerular number matched on different sections, and each column of M represents the glomerular number on the same section.

[0100] Step S5-0 realizes fully automatic glomerulus recognition through a human-computer interface. At the same time, the human-computer interface can also automatically match 12-level multistained glomerulus images and present the matching results.

[0101] The embodiments of the present invention propose a multi-section hierarchical glomerulus identification and matching method based on deep learning. In some of the above embodiments, MTM (Multistained Transformer-based Mask R-CNN) is used as an instance segmentation model to quickly extract the position and contour information of the glomeruli. Then, a feature matching (CHDM) algorithm consisting of coarse matching based on coherent point drift and fine matching based on hybrid feature distance metric is used to achieve matching of glomeruli between two levels. Finally, accurate 12-level multistained glomerulus matching is achieved through a voting-based cross matching method (VCM).

[0102] Performance Testing

[0103] The present invention has excellent glomerulus instance segmentation performance. MTM is compared with other models, including CNNM [MASK RCNN], SOLOV2, YOLACT, Condinst, etc. The results are shown in the following table.

[0104]

[0105] Overall, MTM achieved the best performance on instance segmentation. For object detection, MTM outperformed other models by at least 2 percentage points on mAP@50, mAP@75, and mAP@50.95. This demonstrates that the proposed MTM is capable of accurate glomerulus detection. For semantic segmentation, MTM outperformed the next-best performer, CNNM, by 1.4%, 2.4%, and 3.1% on mAP@50, mAP@75, and mAP@50.95, respectively. MTM demonstrated a significant advantage on the more challenging mAP metric.

[0106] Comparison of instance segmentation visualization results Figure 2 shown.

[0107] The present invention has an excellent effect in glomerular matching. In order to verify the influence of different modules on the performance of glomerular matching, the matching results under different drift modes and different metric space construction modes are compared.

[0108] (1) Results of different CPDs: The weight remains 1:1:1

[0109] method SMR@6:12 SMR@6 SMR@7 SMR SMR@9 SMR@10 SMR@11 SMR@12 Rigid 0.7379 0.9790 0.9182 0.8007 0.7754 0.6825 0.5869 0.4223 Affine 0.7459 0.9825 0.8783 0.8141 0.7990 0.7018 0.6210 0.4249 Deformable 0.3407 0.7340 0.4771 0.3610 0.3147 0.2325 0.1628 0.1025

[0110] The performance of different drift modes is related to the distribution of glomeruli at different levels. During the filming process, slices at different levels undergo relative displacement and deformation. The CPD method can model this process. Drift modes include rigid, affine, and deformable.

[0111] The table above shows that affine performs best overall, achieving leading results in all metrics except OMR and SMN@6. Deformabe performs poorly, with its OMR just exceeding 50%. Rigid has a clear advantage in SMN@7, but falls short of affine in more challenging metrics.

[0112] (2) Different weight combinations: Maintaining affine

[0113]

[0114] Different hybrid distance feature spaces determine matching performance. Using affine as the fixed drift mode, we compared different hybrid distance configurations. The table above shows that when using three features, matching results are optimal for all metrics except SMN@6 and SMN@7. Using only two features results in suboptimal performance. Using only a single feature rarely results in an OMR exceeding 0.7.

[0115] The matching effect diagram of the embodiment of the present invention is as follows Figure 3 shown. Figure 4 The human-machine interface according to the embodiment of the present invention is shown. Figure 5 The image shows the magnified multi-level glomerular output of the human-computer interface of an embodiment of the present invention. In general, the matching method of the present invention can achieve accurate 12-level multistained matching for glomeruli of varying shapes and sizes. The matching method proposed in the present invention can identify matching glomeruli based on a hybrid distance metric and perform cross-matching to obtain accurate results, which is of great significance for simplifying the workflow of pathologists. The present invention can significantly save doctors' time searching for glomeruli, thereby helping them focus more on lesion diagnosis.

[0116] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0117] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0118] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0119] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0120] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0121] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0123] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0124] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0125] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0126] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0127] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0128] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A multi-faceted hierarchical glomerulus identification and matching method based on deep learning, characterized in that: The steps include: Glomerular instance segmentation is performed on multi-section images to be identified using a glomerular instance segmentation model. The glomerular instance segmentation model is an instance segmentation model (MTM) based on the Mask R-CNN framework and the Transformer backbone. The MTM detects and segments glomerular instances in renal biopsy images through a backbone network, a region proposal network (RPN), a region of interest pooling (ROI) and a multi-task prediction head. Based on the glomerulus instance segmentation result, matching the glomeruli between two layers in the multi-section hierarchy by the glomerular matching model, and then achieving matching of the glomeruli in each section hierarchy; The glomerulus instance segmentation model and the glomerulus matching model are trained using renal biopsy sample images with multiple slice levels; Among them, the matching of glomeruli between the two levels includes a coarse matching stage and a fine matching stage. The coarse matching stage determines the matching relationship between glomeruli at two different levels through coherent point drift (CPD) based on the Gaussian mixture model, thereby determining the most likely matching source glomerulus for each target glomerulus. The fine matching stage determines the optimal matching glomerulus for each target glomerulus through distance calculation in the feature space based on the coarse matching results, and obtains the matching matrix of glomerular matching between the two levels; the matching matrices are fused to obtain the final matching matrix of glomeruli at each section level.

2. The multi-faceted hierarchical glomerulus identification and matching method based on deep learning according to claim 1, characterized in that: The multi-section layer includes 12 layers, and the staining of the first 6 and last 6 layers is H&E, PAS, MT, PASM, H&E, and PAS, respectively.

3. The multi-faceted hierarchical glomerulus identification and matching method based on deep learning according to claim 1, wherein: The glomerulus instance segmentation includes: The backbone network is used to extract features of renal biopsy images with different staining related to glomerular structure; The outputs of each stage in the backbone network are fused through the feature pyramid network FPN, and the features of different scales are integrated into the region proposal network RPN; The region proposal network (RPN) uses anchor points to calculate the obtained feature map and generate region proposals. The region of interest pooling extracts the remaining region proposals after position correction, elimination, and maximum value suppression (NMS). Use a multi-task prediction head to classify and segment the extracted region proposals, and use bounding box regression to correct the proposed positions again; Thus, the detection box position and semantic segmentation results of each glomerulus are obtained.

4. The multi-faceted hierarchical glomerulus identification and matching method based on deep learning according to any one of claims 1 to 3, wherein: The overall optimization goal of the glomerulus instance segmentation model is as follows: ； in, and represents the classification and bounding box regression losses, The binary cross entropy function is used, and its expression is as follows: ； Where N is the number of categories, and are the label and prediction results of the i-th category respectively.

5. The multi-faceted hierarchical glomerulus identification and matching method based on deep learning according to any one of claims 1 to 3, wherein: Performing the coarse matching based on the coherent point drift (CPD) and the fine matching based on the hybrid distance metric includes: For multiple aspect levels, select one level as the target level T, and the remaining levels as the source levels S. Assume that T and S have and glomeruli, then the positions of the glomeruli on T and S are recorded as and ; The coarse matching determines the Most likely match indivual , that is, find ; Using CPD based on Gaussian mixture model GMM, the overall likelihood function of CPD is as follows: ； in, , As the drift function, the affine transformation function is used. During the drift process, the affine transformation parameters are solved. , we obtain the matching probability of each glomerulus on the source level with a specific glomerulus on the target level, and thus obtain the overall posterior probability; Through the fine matching, the The best matching glomerulus is determined among the glomeruli, i.e. , and finally get ,in, is a mixed distance metric space; Get the matching matrix with the i-th level as the target level ,in, represents the total number of glomeruli detected at the i-th level, Each row represents the serial number of the same glomerulus at each level, and each level is taken as the target level in turn to obtain multiple matching matrices.

6. The multi-faceted hierarchical glomerulus identification and matching method based on deep learning according to claim 5, characterized in that: The mixed distance metric space is characterized by Euclidean distance , deep universal features and handcrafted features constitute; The Euclidean distance feature : The Euclidean distance feature score is defined as The reciprocal of the normalized distance between the glomerulus and the target glomerulus, that is, ; The deep universal features : reflects the low-level semantic features of the glomerulus. The deep universal feature similarity between two glomeruli is defined as the 1-norm of the feature difference, i.e. ; The manual features : Constructs manual features of the contour obtained based on instance segmentation, including Hu moment, area, perimeter, aspect ratio of the minimum circumscribed rectangle and the minimum circumscribed circle perimeter, a total of 11 dimensions of features , the manual feature similarity score is defined as the 1-norm of the feature difference, i.e. ; The distance in feature space is as follows: 。 7. The multi-faceted hierarchical glomerulus identification and matching method based on deep learning according to any one of claims 1 to 3, characterized in that: The voting-based cross-matching method (VCM) is used to cross-fuse each matching matrix to obtain the final matching matrix. Specifically, it includes: Initialize the count list based on each matching matrix and create a two-dimensional zero vector for each matching matrix; Traverse all matching matrices and calculate the number of matching sequence numbers between the reference row and each row of the source matrix; The voting method was used to take the mode in the glomerular number dimension and combine them into the final matching matrix.

8. A multi-faceted hierarchical glomerulus identification and matching system based on deep learning, comprising a processor, characterized in that: The processor is configured to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the multi-section hierarchical glomerulus identification and matching method based on deep learning as described in any one of claims 1 to 7.

Images