Pathology grading method and device, storage medium and computer program product

Abstract

The invention discloses a pathology grading method and device, a storage medium and a computer program product, and relates to the technical field of medical image processing.The pathology grading method comprises the steps that a light microscope image of renal biopsy is obtained, a preset glomerulus recognition model is used for detecting the light microscope image, and slice images of all glomerulus are obtained; according to the slice image, determining a proportion result of each type of glomerulus in the light microscope image; and determining a pathology grading result according to the proportion result and a preset pathology grading rule. According to the method and the device, the glomerulus in the biopsy image is automatically identified and classified, and automatic classification is carried out according to the clinical pathology classification standard, so that the workload of a clinician in pathology classification is reduced, and the accuracy and the efficiency of pathology classification are improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a pathology grading method, device, storage medium and computer program product. Background Art

[0002] Systemic lupus erythematosus (SLE) is an autoimmune disease that affects multiple organs and systems. Lupus nephritis (LN) is one of the most serious target organ damages of SLE, which is highly prevalent in Asian populations. 60% to 80% of SLE children suffer from LN. Early and accurate diagnosis and prognostic assessment of LN can help doctors provide better individualized treatment for LN children after diagnosis, and focus on and intervene in high-risk patients as early as possible to improve survival rates.

[0003] Currently, the diagnosis of LN mainly relies on pathological examination, that is, pathologists manually read the pathological images of renal biopsy to grade, stage and analyze the prognosis of LN. However, due to differences in diagnosis among different doctors, the consistency of manual visual reading is low, there are large differences in multi-resolution images, and the conventional single-model, single-resolution method has poor prediction performance, all of which will lead to a decrease in the accuracy of pathological grading of lupus nephritis.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a pathological grading method, device, storage medium and computer program product, aiming to improve the accuracy of pathological grading of lupus nephritis.

[0006] To achieve the above objectives, the present application proposes a pathological grading method, which comprises:

[0007] Obtaining a light microscopic image of a patient's renal biopsy, and detecting the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus;

[0008] According to the slice image, determining the proportion of each type of glomerulus in the light microscope image;

[0009] The pathology grading result is determined according to the proportion result and the preset pathology grading rules.

[0010] In one embodiment, before the step of detecting the light microscope image using a preset glomerulus recognition model, the method further includes:

[0011] Perform data enhancement on the acquired labeled training samples to obtain enhanced labeled samples;

[0012] Using a preset target detection model to learn the enhanced labeled samples to obtain a glomerulus recognition model;

[0013] The glomerulus recognition model is fine-tuned using renal biopsy annotation data.

[0014] In one embodiment, the step of performing data enhancement on the acquired labeled training samples to obtain enhanced labeled samples includes:

[0015] Randomly cropping the labeled training samples, and migrating the labels of the labeled training samples to the cropped images;

[0016] Performing a color dithering operation on each of the cropped images;

[0017] Randomly flipping each of the cropped images horizontally;

[0018] Each of the cropped images is rotated at a random angle to obtain an enhanced labeled sample.

[0019] In one embodiment, before the step of detecting the light microscope image using a preset glomerulus recognition model, the method further includes:

[0020] According to the recall rate of glomerulus recognition in historical training data, the confidence threshold of the glomerulus recognition model is adjusted, wherein the confidence threshold is positively correlated with the recall rate of the glomerulus recognition.

[0021] In one embodiment, the step of determining the proportion of each type of glomerulus in the light microscope image according to the slice image comprises:

[0022] Determining the classification result of each glomerulus according to the slice image and a preset glomerular classification model;

[0023] According to the typing result, the proportion result is determined.

[0024] In one embodiment, before the step of determining the classification result of each glomerulus according to the slice image and the preset glomerular classification model, the method further includes:

[0025] Using the acquired unlabeled renal biopsy data, a masked autoencoder MAE is trained, wherein the MAE includes a second encoder and a second decoder constructed based on a visual converter ViT;

[0026] The parameters of the second encoder of the trained MAE are determined as the initial parameters of the first encoder of the glomerular classification model, wherein the glomerular classification model is trained based on the initial parameters of the first encoder.

[0027] In one embodiment, before the step of training the masked autoencoder MAE, the method further includes:

[0028] Construct a data-efficient image converter DEiT architecture, wherein the DEiT architecture includes a teacher model and a student model, wherein the teacher model is obtained by training a YOLO classification model, and the student model is constructed based on ViT;

[0029] Providing soft labels to the student model through the teacher model, so that the student model can learn image representations of various types of glomeruli;

[0030] The parameters of the student model are determined as initial parameters of the second encoder, wherein the MAE is trained based on the initial parameters of the second encoder.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a pathology grading device, which comprises:

[0032] A glomerulus recognition module is used to obtain a light microscopic image of a patient's renal biopsy and detect the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus;

[0033] A proportion determination module, used to determine the proportion of each type of glomerulus in the light microscope image according to the slice image;

[0034] The pathology grading module is used to determine the pathology grading result according to the proportion result and the preset pathology grading rules.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a pathology grading device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the pathology grading method described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the pathology grading method described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the pathology grading method described above are implemented.

[0038] One or more technical solutions proposed in this application have at least the following technical effects: obtaining the light microscopy image of the patient's renal biopsy, and using a preset glomerulus recognition model to detect the light microscopy image, obtaining the slice image of each glomerulus, and automatically and accurately sorting out the slice image of the glomerulus from the complex light microscopy image, effectively avoiding the subjectivity and omissions of manual screening, and improving the efficiency and accuracy of image processing; then, according to the slice image, determining the proportion of each type of glomerulus in the light microscopy image, ensuring the objectivity of the proportion result, and providing reliable data support for subsequent pathological grading; finally, determining the pathological grading result according to the proportion result and the preset pathological grading rules, making the pathological grading process more objective and standardized, and reducing the influence of subjective judgment. This application improves the objectivity and accuracy of the proportion result by automatically identifying and typing the glomeruli, thereby improving the accuracy of the pathological grading. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the pathological grading method of the present application;

[0042] Figure 2 A schematic diagram of image cropping provided in Example 2 of the present application;

[0043] Figure 3 A schematic diagram of another method for image cropping provided in Embodiment 2 of the present application;

[0044] Figure 4 This is a schematic diagram of the random angle rotation operation provided in the second embodiment of the present application;

[0045] Figure 5 A schematic diagram of lesion shielding provided in Example 3 of the present application;

[0046] Figure 6 This is the overall architecture diagram of the lupus nephritis grading model provided in Example 3 of the present application;

[0047] Figure 7 This is a schematic diagram of the module structure of the pathological grading device according to an embodiment of the present application;

[0048] Figure 8Schematic diagram of the device structure of the hardware operating environment involved in the pathology grading method in the embodiment of the present application. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] In this embodiment, for the convenience of description, the following description is made with the terminal as the execution subject.

[0052] Since the pathological grading, staging and prognosis analysis of lupus nephritis in clinical practice mainly rely on doctors' manual reading of pathological images, although this process can reflect the pathological characteristics of lupus nephritis to a certain extent, it will lead to great pressure on the demand for doctor resources, and the diagnosis is limited by the doctor's subjective judgment and experience level, resulting in the consistency and accuracy of the diagnosis results being difficult to fully guarantee. At the same time, the development of the disease is continuous, while the pathological grading standards are discrete. In the pathological grading performed by manual typing, there are many one-size-fits-all standards for the purpose of standardization and standardization. This classification standard will flatten the disease information reflected by the pathological image, and will also lead to a decrease in the accuracy of the pathological grading.

[0053] The present application provides a solution, which includes firstly obtaining a light microscopy image of a patient's renal biopsy, detecting the light microscopy image through a fine-tuned glomerular recognition model, and automatically and accurately separating the slice image of the glomerulus from the complex light microscopy image, thereby effectively avoiding the subjectivity and omission problems that may exist in the traditional manual screening process and improving the efficiency and accuracy of image processing; then, based on each slice image and the glomerular classification model, determining the classification of each glomerulus, and performing glomerular classification through the model, thereby ensuring the objectivity of the classification results and more accurately evaluating the pathological condition of the kidney, providing the patient with a more accurate pathological grading diagnosis; finally, determining the proportion of each type of glomerulus, and determining the patient's pathological grading result based on the proportion of each type of glomerulus and the pathological grading rules, thereby making the pathological grading process more objective and standardized, reducing the influence of subjective judgment, and thus improving the accuracy of pathological grading.

[0054] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the terminal as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the present application embodiment provides a pathological grading method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the pathology grading method of the present application.

[0056] In this embodiment, the pathological grading method includes steps S10 to S30:

[0057] Step S10, obtaining a light microscopic image of a renal biopsy of the patient, and detecting the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus;

[0058] It should be noted that the glomerulus recognition model refers to a machine learning algorithm that automatically identifies and locates glomeruli in images. It is learned through a large amount of training data and can efficiently and accurately identify glomeruli in light microscopy images.

[0059] Illustratively, based on the YOLO11 target detection model, complete biopsy light microscopy images of corresponding diseases are used as training data, and training and tuning are performed to obtain a glomerulus recognition model; then, the complete light microscopy images of the patient's renal biopsy are input into the fine-tuned glomerulus recognition model, and all glomeruli are identified and labeled by the model, usually by drawing bounding boxes to achieve labeling and positioning; according to the labeling, the identified glomeruli are cropped out from the original light microscopy image to obtain a slice image of the identified glomerulus.

[0060] For example, after acquiring the light microscopy image, data enhancement can be performed before recognition. Conventional data enhancement steps include random cropping, color jitter (including brightness, contrast, saturation and hue), Gaussian blur, horizontal flipping, etc. However, it has been found through experiments that transformations such as cutting and Gaussian blurring may lead to the loss of image features that glomerular typing depends on, and only the changes of color jitter and horizontal flipping are retained. Therefore, the acquired light microscopy image is color jittered and horizontally flipped, and then the light microscopy image is input into the glomerular recognition model for detection and recognition.

[0061] It can be understood that by automatically identifying and extracting glomerular images, the workload of doctors is reduced, and human errors are also reduced, thereby improving the efficiency and accuracy of lupus nephritis pathological grading.

[0062] Step S20, based on the slice image, determining the percentage of glomeruli of each type in the light microscope image;

[0063] In a feasible embodiment, a slice image containing glomeruli is input into a pre-trained glomerular classification model, and features are extracted for each slice image through the internal algorithm of the model. The glomeruli in the slice image are classified into different types according to their features, such as glomeruli with no obvious proliferation of intrinsic cells, glomeruli with simple mesangial proliferation, glomeruli with endocapillary cell proliferation, and glomeruli with crescent formation, etc. The proportion of each type of glomerulus is statistically analyzed.

[0064] Illustratively, the slice image can be subjected to preprocessing operations such as denoising and contrast enhancement to improve image quality; then, image processing technology is used to extract characteristic information of the glomeruli, such as morphology, size, color, etc.; based on the extracted characteristic information, machine learning or deep learning algorithms are used to classify and identify the glomeruli; finally, the number of glomeruli of each type is counted, and their proportion is calculated.

[0065] It can be understood that by automatically classifying the glomeruli in the slice images and counting the proportion results, the time for manual analysis and judgment is reduced, the classification accuracy of glomerular images is improved, and thus the accuracy of pathological grading of lupus nephritis is improved.

[0066] Step S30, determining the pathology grading result according to the proportion result and the preset pathology grading rule.

[0067] In a feasible embodiment, the pathological grading result of the patient is determined according to the proportion of each type of glomerulus in all glomeruli and the lesion grading rules of lupus nephritis.

[0068] Exemplarily, the proportion of each type of glomerulus in the patient's biopsy image is counted, and patients with only intrinsic cells without obvious hyperplastic glomeruli are divided into grade I, patients with simple mesangial hyperplasia glomeruli but no endocapillary cell hyperplasia glomeruli are divided into grade II, patients with endocapillary cell hyperplasia glomeruli are divided into grade III or IV, patients whose sum of the proportions of endocapillary cell hyperplasia glomeruli and crescent-forming glomeruli does not exceed 50% are classified as grade III, and patients whose sum of the proportions of endocapillary cell hyperplasia glomeruli and crescent-forming glomeruli exceed 50% are classified as grade IV.

[0069] For example, electron microscope images can be combined to provide more information for the diagnosis of lupus nephritis, such as observing the morphology and arrangement of podocytes, the increase or decrease of mesangial matrix, and the deposition of immune complexes in the mesangial region, thereby reducing the occurrence of missed diagnosis or misdiagnosis and improving the accuracy and reliability of the pathological grading of lupus nephritis.

[0070] This embodiment provides a pathology grading method, which reduces the workload of clinicians in pathology grading by automatically identifying and classifying glomeruli in biopsy images and automatically grading them according to clinical pathology grading standards, while reducing the impact of human errors and subjective judgments, thereby improving the accuracy of pathology grading.

[0071] In a feasible implementation manner, before step S10, the method further includes:

[0072] Step S01, adjusting the confidence threshold of the glomerulus recognition model according to the recall rate of glomerulus recognition in historical training data, wherein the confidence threshold is positively correlated with the recall rate of glomerulus recognition.

[0073] It should be noted that the recall rate of glomerular identification refers to the proportion of glomeruli correctly identified by the model in historical training data, which is used to evaluate whether the glomerular identification model can accurately find all glomeruli from biopsy images. The confidence level indicates the degree of certainty of the model in the glomerular results it has identified. The higher the confidence level, the more confident the model is in its prediction results. Slice images with confidence levels exceeding the preset confidence threshold are sent to the next step of glomerular classification.

[0074] During the training process, the recall rate of the model is negatively correlated with the confidence threshold of the model. The lower the confidence threshold of the model, the higher the recall rate of glomerulus recognition, because the model will also determine some low-confidence images as slice images of glomeruli, which may or may not contain glomeruli, which will cause the accuracy of the model to decrease. Therefore, during the model adjustment process, the confidence threshold can be adjusted in a positive correlation according to the recognition rate of the model to keep the accuracy and recall of the model as balanced as possible.

[0075] For example, in the historical training data, when the confidence threshold is 0.25, the recall rate of glomerulus identification is 79%, and only 79% of the glomeruli in the biopsy images enter the subsequent classification stage, resulting in a decrease in the accuracy of pathological grading. In order to ensure the integrity of the data acquisition of the glomerular classification model in the classification stage, the confidence threshold can be appropriately lowered, and the recall rate of the model can be improved while ensuring a certain accuracy rate, that is, a small number of slice images without glomeruli are passed into the classification model, to ensure that most of the glomeruli in the biopsy images can be accurately identified and enter the classification stage normally.

[0076] In this embodiment, by adjusting the confidence threshold, the glomerulus can be identified more accurately, the risk of misdiagnosis and missed diagnosis can be reduced, and the accuracy of pathological grading can be improved.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, before step S10, it also includes:

[0078] Step S01, performing data enhancement on the acquired labeled training samples to obtain enhanced labeled samples;

[0079] It should be noted that data augmentation refers to increasing the diversity of a data set by applying a series of random transformations to simulate possible changes in training data and improve the generalization ability of the model, including horizontal flipping, color transformation, etc.

[0080] Exemplarily, the PAS (Periodic Acid-Schiff) staining dataset of animal kidney sections published in October 2024 is used as annotated training samples. In order to reduce the impact of the differences between rodent and human kidney samples on the model, data enhancement can be performed on the annotated training samples, and each image in the annotated training samples can be horizontally flipped and randomly scaled. While generating more training samples, the model's ability to adapt to image changes can be improved.

[0081] Due to the small size of the patient sample, a data enhancement module was added to enhance the data, including random cropping on the complete renal biopsy image without destroying the integrity of the glomerulus, and automatically migrating the labels to obtain slice data of various image sizes and different numbers of glomeruli, and rotating, flipping, changing the brightness, contrast, hue and other transformations of the slices to achieve the effect of data enhancement. Experiments show that pre-training the model on the data processed by the data enhancement module and then fine-tuning the model on the complete renal biopsy sample data set can have a certain optimization effect on the model, increasing the recall rate of the model from the original 83.2% to 85.7%.

[0082] Step S02, using a preset target detection model to learn the enhanced labeled samples to obtain a glomerulus recognition model;

[0083] It should be noted that the target detection model is used to identify and locate specific target images in an image.

[0084] Exemplarily, the YOLO11 model is selected as the target detection model. After receiving the input of the enhanced labeled samples, the model reads the image and label information therein, continuously adjusts its internal parameters to fit the training data, and finally obtains a glomerulus recognition model that can accurately identify glomerulus images in biopsy images.

[0085] Step S03: fine-tune the glomerulus recognition model using the renal biopsy annotation data.

[0086] It should be noted that renal biopsy annotated data refers to a dataset that accurately annotates the glomeruli in renal biopsy images. During the fine-tuning process, the model receives renal biopsy annotated data as input, and improves the model's recognition accuracy of glomeruli in real renal biopsy data by analyzing the glomerular features in these data.

[0087] In this embodiment, through data enhancement, the model can learn more diverse feature representations and thus generalize to unseen data; through fine-tuning, the model is made more suitable for glomerulus recognition in renal biopsy images, thereby improving the accuracy of glomerulus recognition and further improving the accuracy of pathological grading.

[0088] In a feasible implementation, step S01 includes:

[0089] Step A01, randomly cropping the labeled training samples, and transferring the labels of the labeled training samples to the cropped images;

[0090] For example, the PAS staining dataset of animal kidney slices contains more than 10,000 normal and diseased glomeruli of rodents annotated with masks. After the mask image is converted into a rectangular frame, it is used as a labeled training sample. When cropping, the cropping range is determined according to the rectangular frame of the labeled training sample. Please refer to Figure 2 ,The left side shows the result of the initial random division of the cropping area. Before the cropping begins, the original image is randomly divided into six areas: A, B, C, D, E, and F. During the inspection, it was found that there were glomeruli cut between areas B and C, and between areas B and E. In order to ensure the integrity of the glomerulus, the corresponding cutting line (the dividing line of the area that cuts the glomerulus) is moved into the area. Figure 2 The right side is the result after the cutting line is moved. The cutting line between areas B and E is moved into the original area B, and the cutting line between areas B and C is moved into the original area B. In order to avoid duplication of cutting areas, other areas are adaptively adjusted to obtain the re-divided areas B', C' and E'. After cropping, the labels at the corresponding positions on the original image are automatically migrated to the corresponding cropped image.

[0091] For example, the user input is received through LNDA (Lupus Nephritis Data Augmentation, Lupus Nephritis Data Augmentation Module), please refer to Figure 3 , Figure 3 A schematic diagram of another method for image cropping is provided. According to the received input (pic, a, b), the input image (pic) is cropped into equal parts, divided into a blocks in the horizontal direction and b blocks in the vertical direction, and the input image is divided into a×b=ab image blocks; then, the segmented image is subjected to boundary fragmentation task target detection through the LNDA module, and the boundary fragmentation task target detection is eliminated by re-cropping to increase the diversity of the data and the generalization ability of the model.

[0092] It can be understood that by random cropping, glomeruli of different sizes and positions are provided for model training, which increases the diversity of samples and improves the model's adaptability to new samples.

[0093] Step A02, performing a random dithering operation on each cropped image;

[0094] It should be noted that color jittering refers to the color transformation of an image, including modification of brightness, contrast, saturation, and hue. Brightness transformation refers to increasing or decreasing the image brightness on the original basis to simulate different lighting conditions in a real environment. However, considering that too high brightness will lead to overexposure and color distortion of the image, while too low brightness will lead to loss of image details, the brightness transformation is limited to the range of 0.8 to 1.2 to obtain the complete image characteristics of the glomerulus.

[0095] Exemplarily, the brightness of each area can also be automatically adjusted based on the local features of the image, and the brightness, contrast, color and other features of each area can be extracted. Through image processing and machine learning, a suitable brightness adjustment value can be calculated for each area.

[0096] Step A03, randomly flipping each cropped image horizontally;

[0097] It should be noted that random horizontal flipping refers to flipping the image 180 degrees in the horizontal direction, and the flipping probability is random.

[0098] Step A04: rotate each cropped image at a random angle to obtain enhanced labeled samples.

[0099] It should be noted that based on the symmetry characteristics of the glomerular contour, randomly rotating the cropped image of the glomerulus can help the model learn to recognize glomeruli in different directions and improve the accuracy of diagnosis.

[0100] For example, please refer to Figure 4 , Figure 4 An operation diagram of random angle rotation is provided. First, the cropped image of the glomerulus is rotated at a random angle, and then the rotated image is completed to its original shape. The completed image is then center-cropped to obtain a glomerular image with the same size and shape as the original cropped image of the glomerulus, thereby ensuring diversity while ensuring that the glomerular features are not lost.

[0101] In this embodiment, data enhancement techniques such as random cropping, color jittering, horizontal flipping, and random angle rotation are used to increase the diversity of the training set, which helps reduce the overfitting of the model to the training data and improves the performance of the model on unknown data, thereby improving the accuracy of subsequent models in glomerulus identification.

[0102] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can be referred to the above introduction, and will not be repeated in the following. On this basis, step S20 includes:

[0103] Step S21, determining the classification result of each glomerulus according to the slice image and the preset glomerular classification model;

[0104] Exemplarily, a patient's glomerular slice image is obtained, and the glomerular slice image is converted into a low-dimensional, compact feature representation, including the shape, size, texture, etc. of the glomerulus, through an encoder of a glomerular typing model. The encoder can be constructed based on a convolutional neural network, ResNet (Residual Neural Network) and ViT (Vision Transformer), etc.; then, the feature representation is passed to a decoder of the glomerular typing model, and the feature representation is converted into a glomerular typing result through the decoder.

[0105] Step S22, determining the proportion result according to the typing result.

[0106] In this embodiment, by automatically extracting features and decoding classifications, the glomerular classification information can be obtained more quickly, thereby improving the diagnostic efficiency, while reducing the errors caused by human classification, ensuring the accuracy of the proportion results, and providing reliable data support for subsequent pathological grading.

[0107] In a feasible implementation manner, before step S20, the method further includes:

[0108] Step B01, using the acquired unlabeled renal biopsy data to train a masked autoencoder MAE, wherein the MAE includes a second encoder and a second decoder constructed based on a visual converter ViT;

[0109] In one feasible embodiment, each image in the unlabeled data of renal biopsy is randomly masked and the masked image is input into MAE; the masked image is encoded into a latent representation by the encoder of MAE, and the original image (the masked part) is reconstructed from the latent representation by the decoder of MAE; through continuous iteration and optimization, the encoder of MAE can learn the effective representation of glomerular images.

[0110] It should be noted that MAE (Masked Autoencoders) is a self-supervised learning model that learns the potential features of data through masking and reconstruction processes. MAE consists of an encoder and a decoder. The encoder is responsible for processing visible image blocks and learning useful representations of the image, while the decoder attempts to reconstruct the masked image.

[0111] Since the actual collected patient sample data are significantly different from the public datasets in terms of image quality, lesion type and sample diversity, and there are also differences in resolution and quality between samples from different centers, building encoders and decoders based on ViT can effectively reduce the MAE model's demand for a large amount of labeled data, and through global feature capture, improve the feature representation capability of the MAE encoder, thereby improving the performance of the MAE model.

[0112] For example, the relevant parameters of the model can also be adjusted according to the characteristics of the glomerular image. Considering that the classification features are discrete and may appear in local positions in the image, for example, the identification of simple mesangial hyperplasia glomeruli depends on finding continuous cell nuclei wrapped by mesangium, and the identification of capillary hyperplasia glomeruli depends on observing whether there are changes in the morphology of capillary loops. These features may be composed of only a few cells. If the patch ratio is high, the problem of obscuring the lesion may occur. Please refer to Figure 5 , many detailed features of the glomerulus are obscured, and it is difficult for the decoder to reconstruct the glomerular image based on the remaining features, resulting in poor training results. After repeated experiments, the patch ratio was modified to 1 / 32, which is exactly the ratio of the size of a cell nucleus to the size of the glomerulus, effectively avoiding the cover of continuous lesions and improving the accuracy of the model.

[0113] Step B02, determining the parameters of the trained second encoder of the MAE as the initial parameters of the first encoder of the glomerular typing model, wherein the glomerular typing model is trained based on the initial parameters of the first encoder.

[0114] By migrating the parameters of the already trained MAE model, the glomerular classification model can obtain a better starting point at the beginning of training, thereby accelerating the training process; at the same time, due to the powerful feature extraction capability of the MAE model, migrating the encoder parameters of the MAE helps to improve the accuracy of the glomerular classification model.

[0115] In this embodiment, the training process of the glomerular classification model is accelerated and the model performance is improved through reasonable parameter migration.

[0116] In a feasible implementation manner, before step B01, the method further includes:

[0117] Step B03, constructing a data-efficient image converter DEiT architecture, wherein the DEiT architecture includes a teacher model and a student model, the teacher model is obtained by training the YOLO classification model, and the student model is constructed based on ViT;

[0118] It should be noted that the DEiT (Data-Efficient Image Transformer) architecture can use limited training data to quickly and accurately learn the feature representation of images through the knowledge distillation process of the teacher-student model.

[0119] Step B04, providing soft labels to the student model through the teacher model, so that the student model can learn the image representation of various types of glomeruli;

[0120] It should be noted that the soft label refers to the output of the teacher model in the form of a probability distribution of the input data, rather than a single category label.

[0121] Exemplarily, a teacher model is obtained by training a YOLO classification model, which can accurately identify the category of glomeruli in an image. The input glomerular image is classified by the teacher model, and a predicted probability distribution (soft label) is output. Then, a student model is constructed based on ViT, and the image features and representations of glomeruli are learned by imitating the output of the teacher model.

[0122] It is understandable that through the teacher-student framework, the student model can learn from limited data more effectively, improving data utilization efficiency; at the same time, the student model not only learns the knowledge of the teacher model, prompting the Transformer model to learn the unique features of the convolutional neural network, but also can adapt to new data distributions, enhancing the generalization ability of the model.

[0123] Step B05, determining the parameters of the student model as the initial parameters of the second encoder, wherein the MAE is trained based on the initial parameters of the second encoder.

[0124] The parameters are further transferred to the MAE model for self-supervised learning to further improve the ability of feature extraction. Experiments show that based on the DEiT architecture, the introduction of MAE significantly improves the accuracy of the glomerular classification model to 88.5%.

[0125] In this embodiment, data distillation can help the model learn faster, reduce the risk of model overfitting, and enhance the generalization ability of the model, so that it can perform well in real lupus nephritis detection and improve the accuracy of pathological grading.

[0126] For example, to help understand the implementation process of the pathological grading method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 6 , Figure 6 A general framework diagram of a lupus nephritis grading model is provided, specifically:

[0127] Step S101, obtain the light microscopy image of the patient's renal biopsy, identify and mark the glomeruli in the light microscopy image through the glomerular recognition and segmentation model built based on YOLO11, crop the glomerular image according to the annotation, obtain the slice image of the glomerulus, and pass the slice image to the glomerular classification model for classification.

[0128] The glomerular classification model is obtained by steps S102 and S103. First, step S102 is to construct the DEiT architecture, and use the pre-trained YOLO-cls model (YOLO classification model) as the teacher model to teach the student model to be trained. The encoder and decoder of the student model are constructed through ViT. After each round of training, the student model will output category labels and distillation labels. The classification loss can be obtained by calculating the difference between the category label and the true label, and the distillation loss can be obtained by calculating the difference between the distillation label and the prediction result of the teacher model. The weighted sum of the distillation loss and the classification loss can be calculated to obtain the loss of DEiT, and by minimizing the loss function, the student model can learn better feature representation. The classification loss ensures that the model can accurately predict the category of the image, while the distillation loss forces the student model to imitate the behavior of the teacher model, thereby achieving a balance between data efficiency and model performance to prevent overfitting. Next is step S103. After the DEiT model training is completed, the parameters of the ViT encoder of the student model are passed into the ViT encoder of the MAE model. Then, based on the mask self-building capability of MAE, the input image is restored to the target image to further improve the feature extraction capability of the model. After the MAE model training is completed, the parameters of the ViT encoder of MAE are passed into the ViT encoder of the glomerular typing model. After the glomerular typing model is trained, it is used for glomerular typing.

[0129] It should be noted that Figure 5 The YOLO11* and YOLO-cls* in the figure indicate that they are models obtained after fine-tuning on the glomerulus dataset after data enhancement. The data enhancement includes random color jitter (including brightness, contrast, saturation and hue), horizontal flipping and random angle rotation of the glomerulus images.

[0130] Step S104, the slice image is passed into the glomerular classification model for classification. First, the slice image is subjected to feature extraction through the ViT encoder, and then the extracted feature representation is converted into a glomerular category through the ViT decoder.

[0131] Step S105, after the classification is completed, the category of each glomerulus in the slice image is passed to the data processing model for statistical analysis to obtain the proportion of each type of glomerulus. Combined with the pathological grading rules, the pathological grading result of the patient can be obtained.

[0132] In step S106, clinical data may also be combined to perform a three-level prognosis through an MLP model, and the prognosis result is represented in the form of a probability vector.

[0133] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the pathological grading method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0134] The present application also provides a pathology grading device, please refer to Figure 7 , the pathological grading device includes:

[0135] The glomerulus recognition module 10 is used to obtain the optical microscope image of the patient's renal biopsy and detect the optical microscope image using the fine-tuned glomerulus recognition model to obtain the slice image of each glomerulus;

[0136] A proportion determination module 20 is used to determine the proportion of each type of glomerulus in the light microscope image according to the slice image;

[0137] The pathology grading module 30 is used to determine the pathology grading result according to the proportion result and the preset pathology grading rule.

[0138] The pathological grading device provided in the embodiment of the present application adopts the pathological grading method in the above embodiment to improve the accuracy of the pathological grading of lupus nephritis. Compared with the prior art, the beneficial effects of the pathological grading device provided in the present application are the same as the beneficial effects of the pathological grading method provided in the above embodiment, and other technical features in the pathological grading device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0139] An embodiment of the present application provides a pathology grading device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the pathology grading method in the above-mentioned embodiment 1.

[0140] Reference below Figure 8, which shows a schematic diagram of the structure of a pathological grading device suitable for implementing the embodiment of the present application. The pathological grading device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The pathological grading device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 8 As shown, the pathological grading device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the pathological grading device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the pathology grading device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a pathology grading device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0142] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0143] The pathological grading device provided in the embodiment of the present application adopts the pathological grading method in the above embodiment to improve the accuracy of the pathological grading of lupus nephritis. Compared with the prior art, the beneficial effects of the pathological grading device provided in the present application are the same as the beneficial effects of the pathological grading method provided in the above embodiment, and other technical features in the pathological grading device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0144] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0146] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the pathology grading method in the above embodiment.

[0147] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0148] The computer-readable storage medium may be included in the pathology grading device; or may exist independently without being assembled into the pathology grading device.

[0149] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the pathology grading device, the pathology grading device: obtains a light microscopy image of a renal biopsy, and uses a preset glomerulus recognition model to detect the light microscopy image to obtain a slice image of each glomerulus; determines the proportion of each type of glomerulus in the light microscopy image based on the slice image; and determines the pathology grading result based on the proportion result and a preset pathology grading rule.

[0150] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0153] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned pathological grading method, and can improve the accuracy of the pathological grading of lupus nephritis. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the pathological grading method provided in the above-mentioned embodiment, and will not be repeated here.

[0154] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of the pathology grading method as described above when executed by a processor.

[0155] The computer program product provided in the embodiment of the present application can improve the accuracy of pathological grading of lupus nephritis. Compared with the prior art, the beneficial effects of the computer program product provided in the present application are the same as the beneficial effects of the pathological grading method provided in the above embodiment, which will not be described in detail here.

[0156] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A pathological grading method, characterized in that: The pathological grading method includes: Acquire a light microscopic image of a renal biopsy, and detect the light microscopic image using a preset glomerulus recognition model to obtain a slice image of each glomerulus; According to the slice image, determining the proportion of each type of glomerulus in the light microscope image; The pathology grading result is determined according to the proportion result and the preset pathology grading rules.

2. The pathological grading method according to claim 1, characterized in that: Before the step of detecting the light microscope image using a preset glomerulus recognition model, the method further includes: Perform data enhancement on the acquired labeled training samples to obtain enhanced labeled samples; Using a preset target detection model to learn the enhanced labeled samples to obtain a glomerulus recognition model; The glomerulus recognition model is fine-tuned using renal biopsy annotation data.

3. The pathological grading method according to claim 2, characterized in that: The step of performing data enhancement on the acquired labeled training samples to obtain enhanced labeled samples includes: Randomly cropping the labeled training samples, and migrating the labels of the labeled training samples to the cropped images; Performing a color dithering operation on each of the cropped images; Randomly flipping each of the cropped images horizontally; Each of the cropped images is rotated at a random angle to obtain an enhanced labeled sample.

4. The pathological grading method according to claim 1, characterized in that: Before the step of detecting the light microscope image using a preset glomerulus recognition model, the method further includes: According to the recall rate of glomerulus recognition in historical training data, the confidence threshold of the glomerulus recognition model is adjusted, wherein the confidence threshold is positively correlated with the recall rate of the glomerulus recognition.

5. The pathological grading method according to claim 1, characterized in that: The step of determining the proportion of each type of glomerulus in the light microscope image according to the slice image comprises: Determining the classification result of each glomerulus according to the slice image and a preset glomerular classification model; According to the typing result, the proportion result is determined.

6. The pathological grading method according to claim 5, characterized in that: Before the step of determining the classification result of each glomerulus according to the slice image and the preset glomerular classification model, the method further includes: Using the acquired unlabeled renal biopsy data, a masked autoencoder MAE is trained, wherein the MAE includes a second encoder and a second decoder constructed based on a visual converter ViT; The parameters of the second encoder of the trained MAE are determined as the initial parameters of the first encoder of the glomerular classification model, wherein the glomerular classification model is trained based on the initial parameters of the first encoder.

7. The pathological grading method according to claim 6, characterized in that: Before the step of training the masked autoencoder MAE, the method further includes: Construct a data-efficient image converter DEiT architecture, wherein the DEiT architecture includes a teacher model and a student model, wherein the teacher model is obtained by training a YOLO classification model, and the student model is constructed based on ViT; Providing soft labels to the student model through the teacher model, so that the student model can learn image representations of various types of glomeruli; The parameters of the student model are determined as initial parameters of the second encoder, wherein the MAE is trained based on the initial parameters of the second encoder.

8. A pathological grading device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the pathology grading method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the pathology grading method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the pathology grading method according to any one of claims 1 to 7 are implemented.

Images