Pathological grading method, device, storage medium and computer program product

Abstract

The application discloses a pathological grading method and device, a storage medium and a computer program product, relates to the technical field of medical image processing, and relates to the technical field of medical image processing. The pathological grading method comprises the following steps: acquiring a light microscope image of a kidney biopsy, and detecting the light microscope image by using a preset glomerulus recognition model to obtain a section image of each glomerulus; according to the section image, determining a proportion result of each type of glomerulus in the light microscope image; and according to the proportion result and a preset pathological grading rule, determining a pathological grading result. The application automatically identifies and classifies the glomerulus in the biopsy image, and automatically classifies according to the clinical pathological grading standard, thereby reducing the work load of the clinician in the pathological grading, and improving the accuracy and efficiency of the pathological grading.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a pathology grading method, device, storage medium, and computer program product. Background Art

[0002] Systemic lupus erythematosus (SLE) is an autoimmune disease that affects multiple organs and systems. Lupus nephritis (LN) is one of the most serious target organ damages in SLE and is highly prevalent in Asian populations. 60% to 80% of children with SLE develop LN. Early and accurate LN diagnosis and prognostic assessment can help physicians better tailor treatment to children with LN after diagnosis, provide early focused attention and intervention for high-risk patients, and improve survival rates.

[0003] Currently, the diagnosis of LN mainly relies on pathological examination, that is, pathologists manually read the pathological images of renal biopsies to grade, stage and analyze the prognosis of LN. However, due to diagnostic differences between different doctors, the consistency of manual visual reading is low, and there are large differences in multi-resolution images. As well as the poor prediction performance of conventional single-model, single-resolution methods, the accuracy of the pathological grading of lupus nephritis is reduced.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a pathological grading method, device, storage medium and computer program product, aiming to improve the accuracy of pathological grading of lupus nephritis.

[0006] To achieve the above objectives, the present application proposes a pathology grading method, which comprises:

[0007] Obtaining a light microscopic image of a renal biopsy of the patient, and detecting the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus;

[0008] Determining the percentage of glomeruli of each type in the light microscope image according to the slice image;

[0009] The pathology grading result is determined based on the proportion result and the preset pathology grading rules.

[0010] In one embodiment, before the step of detecting the light microscopy image using a preset glomerulus recognition model, the method further includes:

[0011] Perform data enhancement on the acquired labeled training samples to obtain enhanced labeled samples;

[0012] Using a preset target detection model to learn the enhanced labeled samples to obtain a glomerulus recognition model;

[0013] The glomerulus recognition model was fine-tuned using renal biopsy annotation data.

[0014] In one embodiment, the step of performing data enhancement on the acquired labeled training samples to obtain enhanced labeled samples includes:

[0015] Randomly cropping the labeled training samples, and migrating the labels of the labeled training samples to the cropped images;

[0016] performing a color dithering operation on each of the cropped images;

[0017] Randomly flipping each of the cropped images horizontally;

[0018] Each of the cropped images is rotated at a random angle to obtain an enhanced labeled sample.

[0019] In one embodiment, before the step of detecting the light microscopy image using a preset glomerulus recognition model, the method further includes:

[0020] According to the recall rate of glomerulus recognition in historical training data, the confidence threshold of the glomerulus recognition model is adjusted, wherein the confidence threshold is positively correlated with the recall rate of the glomerulus recognition.

[0021] In one embodiment, the step of determining the proportion of glomeruli of each type in the light microscopy image based on the slice image includes:

[0022] Determining the classification result of each glomerulus according to the slice image and a preset glomerular classification model;

[0023] The proportion result is determined according to the typing result.

[0024] In one embodiment, before the step of determining the classification result of each glomerulus according to the slice image and a preset glomerular classification model, the method further includes:

[0025] Using the acquired unlabeled renal biopsy data, a masked autoencoder (MAE) is trained, wherein the MAE includes a second encoder and a second decoder constructed based on a visual transformer (ViT);

[0026] The parameters of the trained second encoder of the MAE are determined as initial parameters of the first encoder of the glomerular typing model, wherein the glomerular typing model is trained based on the initial parameters of the first encoder.

[0027] In one embodiment, before the step of training the masked autoencoder MAE, the method further includes:

[0028] Build a data-efficient image converter (DEiT) architecture, where the DEiT architecture includes a teacher model and a student model. The teacher model is trained on a YOLO classification model, and the student model is built based on ViT.

[0029] Providing soft labels to the student model through the teacher model so that the student model learns image representations of various types of glomeruli;

[0030] Parameters of the student model are determined as initial parameters of the second encoder, wherein the MAE is trained based on the initial parameters of the second encoder.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a pathology grading device, which includes:

[0032] A glomerulus recognition module is used to obtain a light microscopic image of a patient's renal biopsy and detect the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus;

[0033] a proportion determination module, configured to determine the proportion of each type of glomerulus in the light microscope image based on the slice image;

[0034] The pathology grading module is used to determine the pathology grading result based on the proportion result and the preset pathology grading rules.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a pathology grading device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the pathology grading method described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the pathological grading method described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the pathology grading method described above.

[0038] One or more technical solutions proposed in this application have at least the following technical effects: obtaining a light microscopy image of a patient's renal biopsy, and using a preset glomerulus recognition model to detect the light microscopy image, obtaining a slice image of each glomerulus, and automatically and accurately sorting out the slice image of the glomerulus from the complex light microscopy image, effectively avoiding the subjectivity and omissions of manual screening, and improving the efficiency and accuracy of image processing; then, based on the slice image, determining the proportion of each type of glomerulus in the light microscopy image, ensuring the objectivity of the proportion result, and providing reliable data support for subsequent pathological grading; finally, determining the pathological grading result based on the proportion result and the preset pathological grading rules, making the pathological grading process more objective and standardized, and reducing the influence of subjective judgment. This application improves the objectivity and accuracy of the proportion result by automatically identifying and typing the glomeruli, thereby improving the accuracy of the pathological grading. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 A schematic diagram of the process flow provided for Example 1 of the pathological grading method of this application;

[0042] Figure 2 This is a schematic diagram of image cropping provided in Example 2 of this application;

[0043] Figure 3 A schematic diagram of another method for image cropping provided in Example 2 of the present application;

[0044] Figure 4 This is a schematic diagram of the random angle rotation operation provided in Example 2 of the present application;

[0045] Figure 5 This is a schematic diagram of lesion shielding provided in Example 3 of this application;

[0046] Figure 6 This is the overall architecture diagram of the lupus nephritis grading model provided in Example 3 of this application;

[0047] Figure 7 This is a schematic diagram of the module structure of the pathology grading device according to an embodiment of the present application;

[0048] Figure 8Schematic diagram of the device structure of the hardware operating environment involved in the pathology grading method in the embodiment of the present application. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] In this embodiment, for ease of description, the following description is made with the terminal as the execution subject.

[0052] In clinical practice, the pathological grading, staging, and prognostic analysis of lupus nephritis primarily rely on physicians' manual interpretation of pathological images. While this process can reflect the pathological characteristics of lupus nephritis to a certain extent, it places significant pressure on physician resources. Furthermore, diagnosis is limited by the physician's subjective judgment and experience, making it difficult to fully guarantee the consistency and accuracy of diagnostic results. Furthermore, the progression of the disease is continuous, while pathological grading standards are discrete. In the pathological grading performed by manual typing, many one-size-fits-all criteria exist for the purpose of standardization. These categorizations flatten the disease information reflected in the pathological images, similarly reducing the accuracy of the pathological grading.

[0053] The present application provides a solution. First, light microscopy images of the patient's renal biopsy are obtained, and the light microscopy images are detected by a fine-tuned glomerular recognition model, so as to automatically and accurately separate the slice images of the glomeruli from the complex light microscopy images, effectively avoiding the subjectivity and omission problems that may exist in the traditional manual screening process, and improving the efficiency and accuracy of image processing; then, based on each slice image and the glomerular typing model, the typing of each glomerulus is determined, and the glomerular typing is performed through the model, which ensures the objectivity of the typing results. At the same time, the pathological condition of the kidney can be more accurately evaluated, and a more accurate pathological grading diagnosis can be provided to the patient; finally, the proportion of each type of glomerulus is determined, and based on the proportion of each type of glomerulus and the pathological grading rules, the pathological grading results of the patient are determined, making the pathological grading process more objective and standardized, reducing the influence of subjective judgment, and thus improving the accuracy of pathological grading.

[0054] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions. This embodiment and the following embodiments will be described below using a terminal as the execution subject.

[0055] Based on this, the present invention provides a pathological grading method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the pathology grading method of this application.

[0056] In this embodiment, the pathology grading method includes steps S10 to S30:

[0057] Step S10, obtaining a light microscopic image of a renal biopsy of the patient, and detecting the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus;

[0058] It should be noted that the glomerulus recognition model refers to a machine learning algorithm that automatically identifies and locates glomeruli in images. It is learned through a large amount of training data and can efficiently and accurately identify glomeruli in light microscopy images.

[0059] For example, based on the YOLO11 target detection model, complete biopsy light microscopy images of corresponding diseases are used as training data to train and fine-tune a glomerulus recognition model; then, the complete light microscopy images of the patient's renal biopsy are input into the fine-tuned glomerulus recognition model, and all glomeruli are identified and labeled by the model, usually by drawing bounding boxes to achieve labeling and positioning; according to the labeling, the identified glomeruli are cropped out from the original light microscopy image to obtain a slice image of the identified glomerulus.

[0060] For example, after acquiring a light microscopy image, data enhancement can be performed before recognition. Conventional data enhancement steps include random cropping, color dithering (including brightness, contrast, saturation, and hue), Gaussian blurring, and horizontal flipping. However, experiments have shown that transformations such as cropping and Gaussian blurring can cause the loss of image features that are essential for glomerular typing, retaining only the changes from color dithering and horizontal flipping. Therefore, the acquired light microscopy image is color-dithered and horizontally flipped before being input into the glomerular recognition model for detection and recognition.

[0061] It is understandable that by automatically identifying and extracting glomerular images, the workload of doctors is reduced, and human errors are also reduced, thereby improving the efficiency and accuracy of lupus nephritis pathological grading.

[0062] Step S20, determining the proportion of glomeruli of each type in the light microscopy image based on the slice image;

[0063] In one feasible embodiment, slice images containing glomeruli are input into a pre-trained glomerular classification model, and features are extracted from each slice image using the model's internal algorithm. The glomeruli in the slice images are classified into different types based on their features, such as glomeruli with no obvious proliferation of intrinsic cells, glomeruli with simple mesangial proliferation, glomeruli with endocapillary cell proliferation, and glomeruli with crescent formation, etc. The proportion of each type of glomeruli is statistically analyzed.

[0064] For example, the slice image can be preprocessed by denoising, contrast enhancement and other preprocessing operations to improve image quality; then, the characteristic information of the glomeruli, such as morphology, size, color, etc., can be extracted using image processing technology; based on the extracted characteristic information, the glomeruli can be classified and identified using machine learning or deep learning algorithms; finally, the number of glomeruli of each type can be counted, and their proportion can be calculated.

[0065] It can be understood that by automatically classifying the glomeruli in the slice images and counting the proportion results, the time for manual analysis and judgment is reduced, the classification accuracy of the glomerular images is improved, and thus the accuracy of the pathological grading of lupus nephritis is improved.

[0066] Step S30: determining the pathology grading result according to the proportion result and the preset pathology grading rules.

[0067] In a feasible embodiment, the pathological grading result of the patient is determined according to the proportion of each type of glomerulus in all glomeruli and the lesion grading rules of lupus nephritis.

[0068] Exemplarily, the proportion of each type of glomerulus in the patient's biopsy image is counted, and patients with only intrinsic cells without obvious hyperplasia of glomeruli are divided into grade I, patients with simple mesangial hyperplasia of glomeruli but no endocapillary cell hyperplasia of glomeruli are divided into grade II, and patients with endocapillary cell hyperplasia of glomeruli are divided into grade III or IV. Patients whose sum of the proportion of endocapillary cell hyperplasia glomeruli and crescent-forming glomeruli does not exceed 50% are classified as grade III, and patients whose sum exceeds 50% are classified as grade IV.

[0069] For example, electron microscopy images can also be combined to provide more information for the diagnosis of lupus nephritis, such as observing the morphology and arrangement of podocytes, the increase or decrease of mesangial matrix, and the deposition of immune complexes in the mesangial region, thereby reducing the occurrence of missed diagnosis or misdiagnosis and improving the accuracy and reliability of the pathological grading of lupus nephritis.

[0070] This embodiment provides a pathology grading method that automatically identifies and types glomeruli in biopsy images and automatically grades them according to clinical pathology grading standards. This reduces the workload of clinicians in pathology grading, reduces the impact of human error and subjective judgment, and improves the accuracy of pathology grading.

[0071] In a feasible implementation manner, before step S10, the method further includes:

[0072] Step S01 : adjusting the confidence threshold of the glomerulus recognition model according to the recall rate of glomerulus recognition in historical training data, wherein the confidence threshold is positively correlated with the recall rate of glomerulus recognition.

[0073] It's important to note that the recall rate for glomerular identification refers to the proportion of glomeruli correctly identified by the model in historical training data. It is used to assess whether the glomerular identification model can accurately identify all glomeruli in biopsy images. Confidence indicates the model's degree of certainty about its identified glomeruli. A higher confidence level indicates greater confidence in the model's predictions. Slice images with a confidence level exceeding a preset threshold are then sent to the next step for glomerular classification.

[0074] During the training process, the recall rate of the model is negatively correlated with the confidence threshold of the model. The lower the confidence threshold of the model, the higher the recall rate of glomerulus recognition, because the model will also determine some low-confidence images as glomerular slice images, which may or may not contain glomeruli, which will cause the accuracy of the model to decrease. Therefore, during the model adjustment process, the confidence threshold can be adjusted in a positive correlation according to the recognition rate of the model to keep the accuracy and recall rate of the model as balanced as possible.

[0075] For example, in historical training data, at a confidence threshold of 0.25, the recall rate for glomerulus identification was 79%. This meant that only 79% of the glomeruli in the biopsy images progressed to the subsequent classification stage, resulting in reduced accuracy in pathological grading. To ensure the integrity of the data acquired by the glomerular classification model during the classification stage, the confidence threshold can be appropriately lowered. While maintaining a certain level of accuracy, meaning that a small number of slice images without glomeruli are passed to the classification model, the model's recall rate can be improved, ensuring that the majority of glomeruli in the biopsy images are accurately identified and progress to the classification stage.

[0076] In this embodiment, by adjusting the confidence threshold, glomeruli can be identified more accurately, the risk of misdiagnosis and missed diagnosis can be reduced, and the accuracy of pathological grading can be improved.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated hereafter. On this basis, before step S10, it also includes:

[0078] Step S01, performing data enhancement on the acquired labeled training samples to obtain enhanced labeled samples;

[0079] It should be noted that data augmentation refers to increasing the diversity of the data set by applying a series of random transformations, which is used to simulate possible changes in the training data and improve the generalization ability of the model. These include horizontal flipping, color transformation, etc.

[0080] For example, the PAS (Periodic Acid-Schiff) staining dataset of animal kidney slices obtained and published in October 2024 is used as annotated training samples. In order to reduce the impact of the differences between rodent and human kidney samples on the model, data enhancement can be performed on the annotated training samples, and each image in the annotated training samples can be horizontally flipped and randomly scaled. While generating more training samples, the model's adaptability to image changes is improved.

[0081] Due to the small sample size, a data augmentation module was added to enhance the data. This includes random cropping of complete renal biopsy images, while preserving glomerular integrity, and automatic label migration. This allows for the acquisition of slice data with varying image sizes and numbers of glomeruli. The slices are then rotated, flipped, and subjected to transformations such as brightness, contrast, and hue to achieve data enhancement. Experiments have shown that pre-training the model on data processed by the data augmentation module, followed by fine-tuning on the complete renal biopsy dataset, can achieve a certain optimization effect on the model, increasing its recall rate from 83.2% to 85.7%.

[0082] Step S02: Using a preset target detection model to learn the enhanced labeled samples to obtain a glomerulus recognition model;

[0083] It should be noted that the target detection model is used to identify and locate specific target images in an image.

[0084] For example, the YOLO11 model is selected as the target detection model. After receiving the input of the enhanced labeled samples, the model reads the image and label information therein, continuously adjusts its internal parameters to fit the training data, and finally obtains a glomerulus recognition model that can accurately identify glomerulus images in biopsy images.

[0085] Step S03: fine-tune the glomerulus recognition model using the renal biopsy annotation data.

[0086] It should be noted that renal biopsy annotated data refers to a dataset that accurately labels glomeruli in renal biopsy images. During fine-tuning, the model receives this annotated data as input and analyzes the glomerular features within this data to improve the model's accuracy in identifying glomeruli in real renal biopsy data.

[0087] In this embodiment, data enhancement is used to enable the model to learn more diverse feature representations and thus generalize to unseen data; fine-tuning is used to make the model more suitable for glomerulus recognition in renal biopsy images, thereby improving the accuracy of glomerulus recognition and, in turn, the accuracy of pathological grading.

[0088] In a feasible implementation, step S01 includes:

[0089] Step A01: randomly cropping labeled training samples and transferring the labels of the labeled training samples to the cropped images;

[0090] For example, the PAS staining dataset of animal kidney slices contains more than 10,000 normal and diseased glomeruli of rodents annotated with masks. After converting the mask image into a rectangular frame, it is used as the labeled training sample. When cropping, the cropping range is determined according to the rectangular frame of the labeled training sample. Please refer to Figure 2 The left side shows the result of the initial random division of the cropping area. Before the cropping begins, the original image is randomly divided into six areas: A, B, C, D, E, and F. During the inspection, it was found that there were glomeruli that were cut between areas B and C, and between areas B and E. In order to ensure the integrity of the glomeruli, the corresponding cutting line (the dividing line of the area that cuts the glomerulus) is moved into the area. Figure 2 The right side is the result after the cutting line is moved. The cutting line between areas B and E is moved into the original area B, and the cutting line between areas B and C is moved into the original area B. In order to avoid duplication of cutting areas, other areas are adaptively adjusted to obtain the re-divided areas B', C' and E'. After cropping is completed, the labels at the corresponding positions on the original image are automatically migrated to the corresponding cropped image.

[0091] For example, the user input is received through LNDA (Lupus Nephritis Data Augmentation, lupus nephritis data enhancement module), please refer to Figure 3 , Figure 3 A schematic diagram of another image cropping method is provided. According to the received input (pic, a, b), the input image (pic) is cropped into equal parts, divided into a blocks in the horizontal direction and b blocks in the vertical direction, and the input image is divided into a×b=ab image blocks; then, the LNDA module is used to detect boundary fragmentation task targets on the segmented image, and the boundary fragmentation task target detection is eliminated by re-cropping to increase data diversity and the generalization ability of the model.

[0092] It can be understood that random cropping provides glomeruli of different sizes and positions for model training, which increases the diversity of samples and improves the model's adaptability to new samples.

[0093] Step A02, performing a random dithering operation on each cropped image;

[0094] It should be noted that color dithering refers to the color transformation of an image, including modifying brightness, contrast, saturation, and hue. Brightness transformation refers to increasing or decreasing the image brightness on the original basis to simulate different lighting conditions in a real environment. However, considering that excessive brightness will lead to overexposure and color distortion of the image, while excessively low brightness will lead to loss of image details, the brightness transformation is limited to the range of 0.8 to 1.2 to obtain the complete image characteristics of the glomerulus.

[0095] For example, the brightness of each area can be automatically adjusted based on the local features of the image, and the brightness, contrast, color and other features of each area can be extracted. Through image processing and machine learning, a suitable brightness adjustment value can be calculated for each area.

[0096] Step A03, randomly flipping each cropped image horizontally;

[0097] It should be noted that random horizontal flipping refers to flipping the image 180 degrees in the horizontal direction, and the flipping probability is random.

[0098] Step A04: rotate each cropped image at a random angle to obtain enhanced labeled samples.

[0099] It should be noted that based on the symmetry characteristics of the glomerular contour, randomly rotating the cropped image of the glomerulus can help the model learn to recognize glomeruli in different directions and improve the accuracy of diagnosis.

[0100] For example, please refer to Figure 4 , Figure 4 A schematic diagram of the random angle rotation operation is provided. First, the cropped image of the glomerulus is rotated at a random angle, then the rotated image is complemented to its original shape, and then the complemented image is center-cropped to obtain a glomerular image with the same size and shape as the original cropped image of the glomerulus, ensuring diversity while ensuring that the glomerular features are not lost.

[0101] In this embodiment, data enhancement techniques such as random cropping, color jittering, horizontal flipping, and random angle rotation are used to increase the diversity of the training set, which helps reduce the overfitting of the model to the training data and improves the performance of the model on unknown data, thereby improving the accuracy of subsequent models in glomerulus identification.

[0102] Based on the first and / or second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to above and will not be described in detail. On this basis, step S20 includes:

[0103] Step S21, determining the classification result of each glomerulus based on the slice image and the preset glomerular classification model;

[0104] For example, a patient's glomerular slice image is obtained, and the glomerular slice image is converted into a low-dimensional, compact feature representation, including the shape, size, texture, etc. of the glomerulus, through the encoder of the glomerular typing model. The encoder can be constructed based on convolutional neural networks, ResNet (Residual Neural Network) and ViT (Vision Transformer); then, the feature representation is passed to the decoder of the glomerular typing model, and the decoder converts the feature representation into the glomerular typing result.

[0105] Step S22: Determine the proportion result according to the typing result.

[0106] In this embodiment, by automatically extracting features and decoding classifications, glomerular typing information can be obtained more quickly, thereby improving diagnostic efficiency. At the same time, it reduces errors in human typing, ensures the accuracy of the proportion results, and provides reliable data support for subsequent pathological grading.

[0107] In a feasible implementation manner, before step S20, the method further includes:

[0108] Step B01: using the acquired unlabeled renal biopsy data, training a masked autoencoder (MAE), wherein the MAE includes a second encoder and a second decoder constructed based on a visual transformer (ViT);

[0109] In one feasible embodiment, each image in the unlabeled renal biopsy data is randomly masked and the masked image is input into the MAE; the masked image is encoded into a latent representation by the MAE encoder, and the original image (the masked part) is reconstructed from the latent representation by the MAE decoder; through continuous iteration and optimization, the MAE encoder can learn an effective representation of the glomerular image.

[0110] It should be noted that MAE (Masked Autoencoders) is a self-supervised learning model that learns the latent features of data through a masking and reconstruction process. MAE consists of an encoder and a decoder. The encoder is responsible for processing visible image patches and learning useful image representations, while the decoder attempts to reconstruct the masked image.

[0111] Since the actual collected patient sample data has significant differences from the public dataset in terms of image quality, lesion type and sample diversity, and there are also differences in resolution and quality between samples from different centers, building an encoder and decoder based on ViT can effectively reduce the MAE model's demand for large amounts of labeled data, and through global feature capture, enhance the feature representation capability of the MAE encoder, thereby improving the performance of the MAE model.

[0112] For example, the relevant parameters of the model can also be adjusted according to the characteristics of the glomerular image. Considering that the classification features are discrete and may appear in local positions in the image, for example, the identification of simple mesangial hyperplasia glomeruli depends on finding continuous cell nuclei wrapped by mesangium, and the identification of capillary hyperplasia glomeruli depends on observing whether there are changes in the morphology of capillary loops. These features may only be composed of a few cells. If the patch ratio is too high, the problem of obscuring the lesion may occur. Please refer to Figure 5 Many detailed features of the glomerulus are obscured, and it is difficult for the decoder to reconstruct the glomerular image based on the remaining features, resulting in poor training results. After repeated experiments, the patch ratio was modified to 1 / 32, which is exactly the ratio of the size of a cell nucleus to the size of the glomerulus, effectively avoiding the obstruction of continuous lesions and improving the accuracy of the model.

[0113] Step B02: Determine the parameters of the trained second encoder of the MAE as initial parameters of the first encoder of the glomerular typing model, wherein the glomerular typing model is trained based on the initial parameters of the first encoder.

[0114] By migrating the parameters of the already trained MAE model, the glomerular classification model can obtain a better starting point at the beginning of training, thereby accelerating the training process; at the same time, due to the powerful feature extraction capability of the MAE model, migrating the MAE encoder parameters helps to improve the accuracy of the glomerular classification model.

[0115] In this embodiment, reasonable parameter migration is used to accelerate the training process of the glomerular typing model and improve the model performance.

[0116] In a feasible implementation manner, before step B01, the method further includes:

[0117] Step B03: Build a data-efficient image converter DEiT architecture, where the DEiT architecture includes a teacher model and a student model. The teacher model is obtained by training the YOLO classification model, and the student model is built based on ViT.

[0118] It should be noted that the DEiT (Data-Efficient Image Transformer) architecture can use limited training data to quickly and accurately learn image feature representations through the knowledge distillation process of the teacher-student model.

[0119] Step B04: providing soft labels to the student model through the teacher model, so that the student model can learn the image representation of various glomeruli;

[0120] It should be noted that the soft label refers to the output of the teacher model in the form of probability distribution given to the input data, rather than a single category label.

[0121] For example, a teacher model is obtained by training the YOLO classification model, which can accurately identify the category of glomeruli in the image. The input glomerulus image is classified by the teacher model, and the predicted probability distribution (soft label) is output. Then, a student model is constructed based on ViT, and the image features and representation of the glomeruli are learned by imitating the output of the teacher model.

[0122] It is understandable that through the teacher-student framework, the student model can learn more effectively from limited data, improving data utilization efficiency; at the same time, the student model not only learns the knowledge of the teacher model, prompting the Transformer model to learn the unique features of the convolutional neural network, but also can adapt to new data distributions, enhancing the generalization ability of the model.

[0123] Step B05: Determine the parameters of the student model as the initial parameters of the second encoder, wherein the MAE is trained based on the initial parameters of the second encoder.

[0124] The parameters were then transferred to the MAE model for self-supervised learning, further improving the feature extraction capabilities. Experiments showed that the introduction of MAE, based on the DEiT architecture, significantly improved the accuracy of the glomerular classification model, reaching 88.5%.

[0125] In this embodiment, data distillation can help the model learn faster, reduce the risk of model overfitting, and enhance the generalization ability of the model, so that it can perform well in real lupus nephritis detection and improve the accuracy of pathological grading.

[0126] For example, in order to help understand the implementation process of the pathological grading method obtained by combining this embodiment with the above embodiment 1, please refer to Figure 6 , Figure 6 A general framework diagram of a lupus nephritis grading model is provided, specifically:

[0127] In step S101, a light microscopy image of a renal biopsy of the patient is obtained. The glomeruli in the light microscopy image are identified and labeled using a glomerular recognition and segmentation model based on YOLO11. The glomerular image is cropped according to the annotations to obtain a slice image of the glomerulus. The slice image is then passed to the glomerular classification model for classification.

[0128] The glomerular classification model is obtained in steps S102 and S103. First, in step S102, the DEiT architecture is constructed, and the pre-trained YOLO-cls model (YOLO classification model) is used as the teacher model to teach the student model to be trained. The encoder and decoder of the student model are constructed through ViT. After each round of training, the student model will output category labels and distillation labels. The classification loss can be obtained by calculating the difference between the category label and the true label, and the distillation loss can be obtained by calculating the difference between the distillation label and the prediction result of the teacher model. The weighted sum of the distillation loss and the classification loss can be calculated to obtain the DEiT loss. By minimizing the loss function, the student model learns better feature representation. The classification loss ensures that the model can accurately predict the category of the image, while the distillation loss forces the student model to imitate the behavior of the teacher model, thereby achieving a balance between data efficiency and model performance to prevent overfitting. Next is step S103. After the DEiT model training is completed, the parameters of the ViT encoder of the student model are passed into the ViT encoder of the MAE model; then, based on the mask self-building capability of MAE, the input image is restored to the target image to further improve the feature extraction capability of the model; after the MAE model training is completed, the parameters of the MAE ViT encoder are passed into the ViT encoder of the glomerular typing model. After the glomerular typing model is trained, it is used for glomerular typing.

[0129] It should be noted that Figure 5 The YOLO11* and YOLO-cls* in the figure indicate that they are models obtained after fine-tuning on the glomerulus dataset after data augmentation. The data augmentation includes random color jittering (including brightness, contrast, saturation and hue), horizontal flipping and random angle rotation of the glomerulus images.

[0130] Step S104: The slice image is passed into the glomerular classification model for classification. First, the slice image is subjected to feature extraction by the ViT encoder, and then the extracted feature representation is converted into glomerular categories by the ViT decoder.

[0131] Step S105: After the classification is completed, the category of each glomerulus in the slice image is input into the data processing model for statistical analysis to obtain the proportion of each type of glomerulus. Combined with the pathological grading rules, the pathological grading result of the patient can be obtained.

[0132] In step S106 , clinical data may be combined to perform a three-level prognosis using an MLP model, and the prognosis result is expressed in the form of a probability vector.

[0133] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the pathological grading method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0134] The present application also provides a pathology grading device, please refer to Figure 7 , the pathological grading device includes:

[0135] The glomerulus recognition module 10 is used to obtain a light microscopic image of a patient's renal biopsy and detect the light microscopic image using the fine-tuned glomerulus recognition model to obtain a slice image of each glomerulus;

[0136] A proportion determination module 20 is used to determine the proportion of glomeruli of each type in the light microscope image based on the slice image;

[0137] The pathology grading module 30 is used to determine the pathology grading result according to the proportion result and the preset pathology grading rules.

[0138] The pathology grading device provided in the embodiments of this application utilizes the pathology grading method described in the above embodiments to improve the accuracy of pathology grading of lupus nephritis. Compared to the prior art, the beneficial effects of the pathology grading device provided in this application are the same as those of the pathology grading method described in the above embodiments. Other technical features of the pathology grading device are the same as those disclosed in the above embodiments and are not further described here.

[0139] An embodiment of the present application provides a pathology grading device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the pathology grading method of the above-mentioned embodiment 1.

[0140] Reference below Figure 8, which shows a schematic diagram of the structure of a pathology grading device suitable for implementing the embodiments of the present application. The pathology grading device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The pathological grading device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0141] like Figure 8 As shown, the pathology grading device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the pathology grading device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 can allow the pathology grading device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a pathology grading device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or provided instead.

[0142] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0143] The pathology grading device provided in the embodiments of this application utilizes the pathology grading method described in the above embodiments to improve the accuracy of pathology grading of lupus nephritis. Compared to the prior art, the beneficial effects of the pathology grading device provided in this application are the same as those of the pathology grading method described in the above embodiments. Other technical features of the pathology grading device are the same as those disclosed in the above embodiments and are not further described here.

[0144] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0146] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the pathology grading method in the above embodiment.

[0147] The computer-readable storage medium provided in the embodiments of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0148] The computer-readable storage medium may be included in the pathology grading device; or may exist independently without being incorporated into the pathology grading device.

[0149] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a pathology grading device, the pathology grading device: obtains a light microscopy image of a renal biopsy, and uses a preset glomerulus recognition model to detect the light microscopy image to obtain a slice image of each glomerulus; determines the proportion of each type of glomerulus in the light microscopy image based on the slice image; and determines the pathology grading result based on the proportion result and a preset pathology grading rule.

[0150] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0151] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0153] The computer-readable storage medium provided in the embodiments of this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned pathological grading method, thereby improving the accuracy of the pathological grading of lupus nephritis. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the pathological grading method provided in the aforementioned embodiments, and are not further elaborated here.

[0154] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of the above-mentioned pathology grading method when executed by a processor.

[0155] The computer program product provided in the embodiments of this application can improve the accuracy of pathological grading of lupus nephritis. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the pathological grading method provided in the above embodiments, and will not be repeated here.

[0156] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A pathological grading method, characterized in that: The pathological grading method includes: Obtaining a light microscopic image of the renal biopsy, and detecting the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus; Build a data-efficient image converter (DEiT) architecture, where the DEiT architecture includes a teacher model and a student model. The teacher model is trained on a YOLO classification model, and the student model is built based on a visual converter (ViT). Providing soft labels to the student model through the teacher model so that the student model learns image representations of various types of glomeruli; Determining the parameters of the student model as initial parameters of a second encoder of a masked autoencoder (MAE), wherein the second encoder is constructed based on a visual transformer (ViT); Using the acquired unlabeled renal biopsy data and the initial parameters of the second encoder, training the MAE, wherein the MAE includes a second decoder constructed based on the visual transformer ViT; Determining the parameters of the trained second encoder of the MAE as initial parameters of the first encoder of a preset glomerular typing model, wherein the glomerular typing model is trained based on the initial parameters of the first encoder; Determining the classification result of each glomerulus according to the slice image and the glomerular classification model; According to the classification results, determining the proportion of glomeruli of each classification in the light microscope image; The pathology grading result is determined based on the proportion result and the preset pathology grading rules.

2. The pathology grading method according to claim 1, wherein: Before the step of detecting the light microscope image using a preset glomerulus recognition model, the method further includes: Perform data enhancement on the acquired labeled training samples to obtain enhanced labeled samples; Using a preset target detection model to learn the enhanced labeled samples to obtain a glomerulus recognition model; The glomerulus recognition model was fine-tuned using renal biopsy annotation data.

3. The pathological grading method according to claim 2, wherein: The step of performing data enhancement on the acquired labeled training samples to obtain enhanced labeled samples includes: Randomly cropping the labeled training samples, and migrating the labels of the labeled training samples to the cropped images; performing a color dithering operation on each of the cropped images; Randomly flipping each of the cropped images horizontally; Each of the cropped images is rotated at a random angle to obtain an enhanced labeled sample.

4. The pathology grading method according to claim 1, wherein: Before the step of detecting the light microscope image using a preset glomerulus recognition model, the method further includes: According to the recall rate of glomerulus recognition in historical training data, the confidence threshold of the glomerulus recognition model is adjusted, wherein the confidence threshold is positively correlated with the recall rate of the glomerulus recognition.

5. A pathology grading device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the pathology grading method according to any one of claims 1 to 4.

6. A storage medium, characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the pathology grading method according to any one of claims 1 to 4 are implemented.

7. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the pathology grading method according to any one of claims 1 to 4 are implemented.

Images