A method and device for predicting the efficacy of LN-induced therapy based on pathological images

Abstract

This application belongs to the field of medical image processing technology and discloses a method and device for predicting the therapeutic effect of LN induction based on pathological images. The method involves acquiring LN pathological images using four different staining methods: HE staining, PAS staining, PASM staining, and Masson staining. The LN pathological images are labeled with therapeutic efficacy tags. These labeled LN pathological images are then divided into image patches, and a dataset is constructed based on these patches. Six different neural network models are trained using this dataset to obtain four single-staining prediction models for predicting LN pathological images using different staining methods. These four single-staining prediction models are then jointly constructed to create a multi-staining prediction model. The LN pathological image to be evaluated is input into the multi-staining prediction model to obtain the prediction result for the LN pathological image to be evaluated. This method can improve the prediction accuracy of LN induction therapeutic effects.

Description

Technical Field

[0001] This application relates to the field of medical image recognition technology, and in particular to a method and apparatus for predicting the therapeutic effect of LN induction based on pathological images. Background Technology

[0002] Lupus nephritis (LN) is a serious complication of systemic lupus erythematosus (SLE), affecting approximately 50% to 80% of SLE patients. Current treatment options for LN primarily include standard induction therapy, which combines glucocorticoids and advanced immunosuppressants. The efficacy of induction therapy is a key factor determining the prognosis of LN; the 10-year kidney survival rate is only 45% for those achieving partial remission and only 19% for those not achieving remission. The proportion of patients responding to first-line treatments such as cyclophosphamide (CTX) or mycophenolate mofetil (MMF) is small, with only 20%–30% of LN patients achieving complete remission after 6–12 months of induction therapy. Accurate prediction of the efficacy of induction therapy is crucial for early identification of high-risk LN patients who do not respond to standard treatments. This facilitates personalized and intensive treatment decisions, thereby improving long-term renal outcomes. Renal biopsy is the gold standard for determining the effectiveness of LN induction therapy, providing information such as glomerular lesions, tubulointerstitial damage, and vascular lesions, which helps predict patient prognosis. However, the evaluation results of kidney biopsies are significantly influenced by the pathologist's experience level. With the development of microscopic photography and scanning technology, kidney pathology sections can be digitized into high-resolution panoramic slice images (WSI). These images contain multi-scale, feature-rich information beyond human visual recognition capabilities, possessing the potential to predict the efficacy of LN induction therapy. Although some studies have attempted to use machine learning models to predict the efficacy of LN induction therapy through pathology images, current methods do not fully utilize the information in pathology images, exhibiting problems such as poor model robustness and low accuracy of evaluation results. Summary of the Invention

[0003] Therefore, embodiments of this application provide a method and apparatus for predicting the effect of LN-induced therapy based on pathological images, which can effectively improve the prediction accuracy of LN-induced therapy effect.

[0004] In one aspect, this application provides a method for predicting the efficacy of LN-induced therapy based on pathological images.

[0005] This application is achieved through the following technical solution: a method for predicting the therapeutic effect of LN induction based on pathological images, the method comprising:

[0006] LN pathological images were obtained using four different staining methods, including HE staining, PAS staining, PASM staining, and Masson staining.

[0007] The pathological images of LN were labeled with therapeutic efficacy tags;

[0008] The LN pathological images with therapeutic labels are divided into image blocks of a preset size, and a dataset is constructed based on the image blocks;

[0009] Using the dataset, six different neural network models were trained to determine four single-stain prediction models for LN pathological images with different staining methods. The single-stain prediction models were used to analyze LN pathological images to obtain prediction results of LN induction treatment effects.

[0010] The four different single-stain prediction models are combined to construct a multi-stain prediction model.

[0011] The LN pathological image to be evaluated is input into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated.

[0012] In a preferred embodiment of this application, the method may be further configured such that, before acquiring LN pathological images using four different staining methods, the method includes:

[0013] Electronic medical records of several LN patients were obtained, and the electronic medical records of LN patients were screened according to the preset inclusion and exclusion criteria to determine candidate LN patients who meet the requirements for model training.

[0014] Full images of renal biopsy sections from candidate LN patients, processed using four different staining methods, were obtained.

[0015] In a preferred embodiment of this application, the LN pathological images may be further configured to include labeling with therapeutic efficacy tags, including:

[0016] The efficacy labels are added to the LN pathological images using annotation software. The efficacy labels include CR and nCR, where CR is used to indicate complete remission after 12 months of LN induction therapy, and nCR is used to indicate incomplete remission after 12 months of LN induction therapy.

[0017] In a preferred example of this application, the LN pathological image with therapeutic labels can be further configured to divide the image into image blocks of a preset size, and a dataset can be constructed based on the image blocks, including:

[0018] LN pathological images with therapeutic labels were converted into 10x magnification, 20x magnification, and 40x magnification images;

[0019] The original size LN pathological image, the 10x magnified image, the 20x magnified image, and the 40x magnified image were cut into image blocks of preset size respectively;

[0020] Image blocks that exclude non-organic regions exceeding a preset threshold are used to obtain image blocks containing valid organized regions.

[0021] Image patches containing effective organized regions are divided into training and testing datasets according to a preset ratio.

[0022] In a preferred example of this application, the dataset can be further configured to train six different neural network models to determine four single-stain prediction models for predicting LN pathological images using different staining methods, including:

[0023] Input the dataset into the neural network model DL m In the process, the neural network model DL is obtained. m The prediction results of LN pathological images, the neural network model DL m It can be any one of the following models: AlexNet, DenseNet-121, Inception-V3, ResNet-50, VGG-11, and Vision Transformer.

[0024] Analyzing neural network models DL m Predictive accuracy of LN pathological images using four different staining methods;

[0025] For each coloring method, a neural network model DL with the highest prediction accuracy is determined. m As a single-stain prediction model.

[0026] In a preferred example of this application, it can be further configured that a warm-up strategy is introduced to adjust the learning rate when training the single-color prediction model.

[0027] In a preferred example of this application, the training of the single-color prediction model may be further configured to include:

[0028] Batch sizes were set to 8, 16, 32, and 64, and the batch size that best performed the single-stain prediction model for each staining method was selected.

[0029] In a preferred example of this application, the training of the single-color prediction model may further include:

[0030] The prediction accuracy of the single staining prediction model for LN pathological images corresponding to each staining method is determined based on the area under the working characteristic curve.

[0031] In a preferred example of this application, the structure of the single-stain prediction model may be further configured as follows:

[0032] Feature extraction layer, multi-instance learning pooling layer, and fully connected layer.

[0033] In a preferred example of this application, the structure of the multi-chromosome prediction model may be further configured as follows:

[0034] Feature extraction layer, multi-instance learning pooling layer, and fully connected layer.

[0035] In a preferred example of this application, the LN pathological image to be evaluated can be further configured to be input into a multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated, including:

[0036] The pathological image of LN to be evaluated is input into the feature extraction layer of the multi-stain prediction model to extract features and obtain feature information;

[0037] The feature information is input into a multi-instance learning pooling layer. An attention mechanism is introduced into the multi-instance learning pooling layer to aggregate the feature information through attention scores and output aggregated features.

[0038] The aggregated features are input into a fully connected layer to obtain the prediction results of the LN pathological image to be evaluated.

[0039] In a second aspect, this application provides a device for predicting the effect of LN-induced therapy based on pathological images, for performing the steps of the prediction method as described in the first aspect.

[0040] This application is achieved through the following technical solution: a device for predicting the therapeutic effect of LN induction based on pathological images, comprising:

[0041] The data acquisition module is configured to acquire LN pathological images corresponding to four different staining methods, including HE staining, PAS staining, PASM staining, and Masson staining.

[0042] The data annotation module is configured to annotate the LN pathological images with therapeutic efficacy labels;

[0043] The data preprocessing module is configured to divide the LN pathological images with therapeutic efficacy labels into image blocks of a preset size, and construct a dataset based on the image blocks;

[0044] The single staining prediction model training module is configured to train six different neural network models using the dataset to determine four single staining prediction models for LN pathological images with different staining methods.

[0045] The multi-chromosome prediction model construction module is configured to jointly construct a multi-chromosome prediction model by combining the four different single-chromosome prediction models.

[0046] The efficacy prediction module is used to input the LN pathological image to be evaluated into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated.

[0047] In a preferred embodiment of this application, the data acquisition module may further include: a data filtering unit, which is configured to acquire electronic medical records of several LN patients, filter the electronic medical records of LN patients according to preset inclusion and exclusion criteria, determine candidate LN patients that meet the requirements for model training, and acquire full images of renal biopsy sections of candidate LN patients processed by four different staining methods.

[0048] In a preferred example of this application, the data preprocessing module can be further configured to:

[0049] LN pathological images with therapeutic labels were converted into 10x magnification, 20x magnification, and 40x magnification images;

[0050] The original size LN pathological image, the 10x magnified image, the 20x magnified image, and the 40x magnified image were cut into image blocks of preset size respectively;

[0051] Image blocks that exclude non-organic regions exceeding a preset threshold are used to obtain image blocks containing valid organized regions.

[0052] Image patches containing effective organized regions are divided into training and testing datasets according to a preset ratio.

[0053] In a preferred example of this application, the single-coloring prediction model training module can be further configured to: input the dataset into the neural network model DL m In the process, the neural network model DL is obtained. m The prediction results of LN pathological images, the neural network model DL m It can be any one of the following models: AlexNet, DenseNet-121, Inception-V3, ResNet-50, VGG-11, and Vision Transformer.

[0054] Analyzing neural network models DL m Predictive accuracy of LN pathological images using four different staining methods;

[0055] For each coloring method, a neural network model DL with the highest prediction accuracy is determined. m As a single-stain prediction model.

[0056] In a preferred example of this application, the single-color prediction model training module can be further configured as follows:

[0057] In the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate.

[0058] In a preferred example of this application, the single-color prediction model training module can be further configured as follows:

[0059] The single staining prediction model was trained with batch sizes of 8, 16, 32, and 64, and the batch size that optimizes the performance of the single staining prediction model for each staining method was selected.

[0060] In a preferred example of this application, the structure of the single-stain prediction model may be further configured as follows:

[0061] Feature extraction layer, multi-instance learning pooling layer, and fully connected layer.

[0062] Thirdly, this application achieves its goal through the following technical solutions:

[0063] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described methods for predicting the effect of LN-induced therapy based on pathological images.

[0064] Fourthly, this application provides a computer-readable storage medium.

[0065] This application is achieved through the following technical solution:

[0066] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for predicting the effect of LN-induced therapy based on pathological images.

[0067] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0068] This application acquires LN pathological images using four different staining methods; labels the LN pathological images with therapeutic efficacy tags; divides the LN pathological images with therapeutic efficacy tags into image patches of a preset size, and constructs a dataset based on the image patches; trains six different neural network models using the dataset to determine four single-staining prediction models for LN pathological images using different staining methods. These single-staining prediction models are used to analyze LN pathological images to predict the efficacy of LN induction therapy; the four different single-staining prediction models are then combined to construct a multi-staining prediction model; the LN pathological image to be evaluated is input into the multi-staining prediction model to obtain the predicted result for the LN pathological image to be evaluated. This application, by integrating features provided by LN pathological images using different staining methods and LN pathological images at different magnification sizes, trains single-staining prediction models that can capture the unique features of LN pathological images using each staining method. Combining multiple single-staining prediction models to form a multi-staining prediction model enables automated analysis of LN pathological images, extracting rich pathological features that are not visible to the naked eye from LN pathological images, resulting in higher robustness and prediction accuracy. Attached Figure Description

[0069] Figure 1 A flowchart illustrating a method for predicting the efficacy of LN-induced therapy based on pathological images, provided in an embodiment of this application.

[0070] Figure 2 This is a schematic diagram of the process for preprocessing LN pathological images according to an embodiment of this application;

[0071] Figure 3 This is a schematic diagram of the processing flow of the multi-stain prediction model for LN pathological images to be evaluated, provided in an embodiment of this application.

[0072] Figure 4 A schematic diagram of the structure of a device for predicting the effect of LN-induced therapy based on pathological images, provided in an embodiment of this application;

[0073] Figure 5 A schematic diagram of the structure of a device for predicting the effect of LN-induced therapy based on pathological images, provided in another embodiment of this application;

[0074] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0075] Explanation of reference numerals in the attached figures:

[0076] Data acquisition module 1, data filtering unit 11, data annotation module 2, data preprocessing module 3, single-stain prediction model training module 4, multi-stain prediction model construction module 5, and efficacy prediction module 6. Detailed Implementation

[0077] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0078] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this application, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0080] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0081] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0082] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0083] like Figure 1 As shown, a first exemplary embodiment of this application provides a method for predicting the therapeutic effect of LN induction based on pathological images, comprising:

[0084] S10: Obtain LN pathological images corresponding to four different staining methods, including HE staining, PAS staining, PASM staining, and Masson staining.

[0085] Because each staining method can highlight different tissue structures and features, this application employs four different staining methods for analyzing and processing LN pathological images in order to fully utilize the feature information in the LN pathological images. Specifically, the renal biopsy sections of LN patients processed by HE staining, PAS staining, PASM staining, and Masson staining methods were scanned using a KF-PRO-120-HI scanner or a KFBIO scanner to generate digital WSI (also known as full-volume image), and the digital WSI was converted into TIFF format LN pathological images.

[0086] After HE staining (hematoxylin and eosin staining), the cell nucleus appears blue and the cytoplasm appears red. HE-stained LN pathological images clearly reflect various cellular components, including the thickness of the glomerular basement membrane (GBM), glomerular proliferation, necrosis, and exudative lesions, renal tubular epithelial cell damage, interstitial edema, interstitial hemorrhage, inflammatory cell infiltration, and vascular inflammation. After PAS staining (periodic acid-Schiff staining), both the mesangial matrix and membrane structures appear purplish-red, and the cell nucleus appears blue. PAS-stained LN pathological images can reflect changes in membrane structures and whether there is an increase in mesangial matrix. They are advantageous for observing basement membrane lesions, increased extracellular matrix, proliferation and exudation of various cells, and identifying whether deposited substances are rich in glycans. LN pathological images obtained by PASM staining (hexamine silver staining) can clearly show basement membrane lesions and extracellular matrix, facilitating the observation of basement membrane spikes, chains, and double-track signs. It can also reveal the vascular elastic lamina and argyrophilic protein deposition in arteries. Masson trichrome staining is a connective tissue staining method. Masson staining (trichrome staining) shows the basement membrane, mesangial matrix, and collagen as green (or blue), immune complexes, fibrinoid necrosis, and thrombi as red, and cell nuclei as blue-black. LN pathological images obtained by Masson trichrome staining can effectively distinguish collagen fibers, elastic fibers, and reticular fibers.

[0087] S20: Label the pathological images of LN with therapeutic efficacy tags.

[0088] Treatment efficacy labels are applied to each LN pathology image based on the corresponding electronic medical record. Specifically, this involves first obtaining the electronic medical record for each patient corresponding to each LN pathology image. The electronic medical record must include at least clinical data, pathological data, treatment data, follow-up data, and biopsy results confirmed by at least two renal pathologists with more than five years of clinical experience.

[0089] S30: Divide LN pathological images with therapeutic labels into image blocks of a preset size, and construct a dataset based on the image blocks.

[0090] To adapt to the neural network model, image processing tools are used to segment LN pathological images with therapeutic labels into multiple non-overlapping image blocks of the same size. Each image block is assigned a corresponding therapeutic label based on the original LN pathological image's therapeutic label. All image blocks corresponding to each LN pathological image are packaged into data packets for storage. A randomization method is used to divide all data packets into training and validation datasets according to a preset ratio, such as 7:3 or 8:2. The training dataset is used to train the subsequent model, while the validation dataset is used to verify the performance of the trained model and adjust its parameters. The above ratio is only an example; the training and validation datasets can be set according to actual needs and are not limited in this application.

[0091] S40: Using the dataset, six different neural network models were trained to determine four single-stain prediction models for LN pathological images with different staining methods. The single-stain prediction models were used to analyze LN pathological images to obtain prediction results of LN induction treatment effects.

[0092] In the actual implementation, six deep learning model architectures (neural network models) were initially trained, including AlexNet, DenseNet-121, Inception-V3, ResNet-50, VGG-11, and Vision Transformer, to determine the optimal network model for each histological staining method. For each of the four staining methods, a single-staining prediction model with the best predictive performance was determined: a first single-staining prediction model for HE staining, a second for PAS staining, a third for PASM staining, and a fourth for Masson staining.

[0093] S50: Combine four different single-chromosome prediction models to construct a multi-chromosome prediction model.

[0094] Specifically, the first, second, third, and fourth single-stain prediction models are integrated to construct a multi-stain prediction model, referred to as the LNCR-DL model. For example, different weights can be assigned to the single-stain prediction models based on their prediction accuracy for LN pathological images corresponding to each staining method, and then a weighted average is used to obtain the multi-stain prediction model.

[0095] Multi-stain prediction models can integrate the predictive advantages of various single-stain prediction models for different staining methods, and can provide more accurate and comprehensive prediction results.

[0096] S60: Input the LN pathological image to be evaluated into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated.

[0097] This application trains a single-stain prediction model by integrating features provided by LN pathological images from different staining methods. The model can capture the unique features of LN pathological images from each staining method. Then, multiple single-stain prediction models are combined to form a multi-stain prediction model, which enables automated analysis of LN pathological images and extracts rich pathological features that cannot be recognized by the naked eye from LN pathological images. This results in higher robustness and prediction accuracy.

[0098] In one exemplary embodiment, before obtaining LN pathological images using four different staining methods in step S10, the method further includes:

[0099] Electronic medical records (EMRs) of several LN patients were obtained and screened according to pre-defined inclusion and exclusion criteria to identify candidate LN patients who met the model training requirements. Full images of renal biopsy sections from candidate LN patients were obtained, processed using four different staining methods. The inclusion criteria for LN patients' EMRs were: 1) age ≥ 1 year; 2) confirmed LN by renal biopsy within 6 months prior to induction therapy; 3) receiving induction therapy with glucocorticoids combined with cyclophosphamide (CTX) or glucocorticoids combined with mycophenolate mofetil (MMF); 4) at least 12 months of follow-up after the start of induction therapy. The exclusion criteria were: LN patients who received other immunosuppressant therapy within 6 months prior to induction therapy, or patients with incomplete clinical and follow-up data for treatment response evaluation.

[0100] In an exemplary embodiment, when labeling the LN pathological image with therapeutic efficacy tags in step S20, the specific steps include:

[0101] Efficacy labels were added to the LN pathological images using annotation software. These labels included CR and nCR, where CR indicated complete remission after 12 months of LN induction therapy, and nCR indicated incomplete remission after 12 months of LN induction therapy. The criteria for complete remission were: 24-hour urinary protein <0.5g and normal eGFR or within 10% of the normal range; the criteria for partial remission were: a 24-hour urinary protein decrease of ≥50% and a level <3g / day, and improved or stable eGFR (±25%); no response meant neither complete nor partial remission.

[0102] In one exemplary embodiment, such as Figure 2 As shown, in step S30, LN pathological images with therapeutic efficacy labels are divided into image blocks of a preset size, and a dataset is constructed based on the image blocks, including:

[0103] S301: Convert LN pathology images with therapeutic labels into 10x, 20x, and 40x magnified images. Image processing software or programming libraries, such as the Pillow library in Python, can be used to magnify each LN pathology image by 10x, 20x, and 40x, and the magnified images can be saved as new files. For example, convert LN pathology images P... i After magnifying by 10x, 20x, and 40x respectively, save as P i-10 P i-20 and P i-40 i represents the pathological image of LN P. i Its unique number.

[0104] S302: Cut the original size LN pathological image, 10x magnified image, 20x magnified image, and 40x magnified image into image blocks of preset size respectively.

[0105] Image processing tools (such as Python's PIL library) are used to segment the images, and a combined numbering format is used to record the position of each image patch in the original LN pathological image. This combined numbering format can be image ID_row number_column number. For example, P... i-10,2,3 P represents the pathological image of LN. i The image block in the 2nd row and 3rd column of the 10x magnified image can have its preset size set as needed, for example, a 512×512 pixel block. Based on the image size and the image block size, the number of rows and columns to be cut can be calculated. This will transform the original size LN pathological image P... i All image blocks cut out from the image are packaged into a data packet. i Save the LN pathological image P i All image patches cut out from the 10x magnified image are packaged into a data packet. i-10 Save the LN pathological image P i All image patches cut out from the 20x magnified image are packaged into a data packet. i-20 Save the LN pathological image P i All image patches cut out from the 40x magnified image are packaged into a data packet. i-40 Save it.

[0106] S303: Exclude image blocks where the proportion of non-organic regions exceeds a preset threshold, and obtain image blocks containing valid organized regions.

[0107] The pixel values ​​of image patches are extracted, and the patches are classified as either non-organic or organized regions based on these values. Specifically, pixels with values ​​greater than 225 are considered non-organic regions. The percentage of pixels with values ​​greater than 225 in each patch is calculated. If this percentage exceeds a preset threshold, the patch is considered to have an excessive proportion of non-organic regions and is deleted. The preset threshold can be set according to actual needs, for example, to 50%. By excluding most image patches that are non-organic regions, the impact of useless data on model training can be reduced, thereby improving training efficiency. Simultaneously, it can reduce overfitting of the model to noisy or irrelevant features during training, improving model stability and prediction accuracy.

[0108] S304: Divide the image patch containing the effective organized region into a training dataset and a test dataset according to a preset ratio.

[0109] Before building the model, StainTools is needed to normalize the colors of the image patches corresponding to each histological staining method in order to minimize the variability of staining.

[0110] In an exemplary embodiment, step S40 involves training six different neural network models using a training dataset to obtain four single-stain prediction models for predicting LN pathological images using different staining methods, specifically including:

[0111] S401: Input the dataset into the neural network model DL m In the process, the neural network model DL is obtained. m P on LN pathological images i j The prediction results of the neural network model DL m It can be any one of the following models: AlexNet, DenseNet-121, Inception-V3, ResNet-50, VGG-11, and Vision Transformer.

[0112] S402: Analyzing Neural Network Model DL m Predictive accuracy of LN pathological images using four different staining methods;

[0113] S403: For each coloring method, determine the neural network model DL with the highest prediction accuracy. m As a single-color prediction model, specifically, the dataset is input into the neural network model DL. m The neural network model DL is obtained through training. mFor each LN pathological image in the dataset, the prediction accuracy of the LN pathological image is determined based on the predicted value and the therapeutic effect label, thus obtaining a neural network model DL. m The prediction accuracy is calculated as follows: first prediction accuracy for the original size image of a LN pathological image, second prediction accuracy for a 10x magnified image, third prediction accuracy for a 20x magnified image, and fourth prediction accuracy for a 40x magnified image. The mean of the first, second, third, and fourth prediction accuracies is used as the neural network model DL. i Prediction accuracy for LN pathological images. This is achieved after obtaining the neural network model DL. m After assessing the prediction accuracy of LN pathological images using four different staining methods, a neural network model DL with the highest prediction accuracy was determined for each staining method. m As a single-stain prediction model.

[0114] For example, the dataset is input into the AlexNet model for training, yielding the AlexNet model's predicted value for each LN pathology image. The prediction accuracy of the AlexNet model for each LN pathology image is determined based on the predicted value and the actual therapeutic labels of the LN pathology images. The prediction accuracy for each LN pathology image is the average prediction accuracy of the AlexNet model for four image sizes (original size, 10x magnification, 20x magnification, and 40x magnification). Classifying the LN pathology images according to four staining methods allows for the statistical analysis of the AlexNet model's prediction accuracy for LN pathology images using the four staining methods. Similarly, the dataset is input into the DenseNet-121 model for training, yielding the DenseNet-121 model's predicted value for each LN pathology image. Following the same steps as with the AlexNet model, the statistical analysis of the DenseNet-121 model's prediction accuracy for LN pathology images using the four staining methods is then performed. The prediction accuracy of the Inception-V3, ResNet-50, VGG-11, and Vision Transformer models for LN pathological images using four staining methods was then obtained. Based on the prediction accuracy of the six neural network models for LN pathological images using the four staining methods, the neural network model with the highest prediction accuracy for each staining method was determined as the single-stain prediction model for that method. Analysis showed that Inception-V3 performed best for HE, PASM, and TRI staining, while Vision Transformer had the highest prediction ability for PAS staining among all neural networks.

[0115] The structure of the single-color prediction model includes: a feature extraction layer, a multi-instance learning pooling layer, and a fully connected layer. The input to the feature extraction layer is the input data, and its output is connected to the input of the multi-instance learning pooling layer. The output of the multi-instance learning pooling layer is connected to the input of the fully connected layer, and the output of the fully connected layer is the prediction result.

[0116] In one exemplary embodiment, in the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate. Specifically, the preset number of training iterations is 5, starting with a lower initial learning rate and further adjusting it using cosine annealing.

[0117] In one exemplary embodiment, training a single staining prediction model includes: training the single staining prediction model with batch sizes of 8, 16, 32, and 64, and selecting the batch size that optimizes the performance of the single staining prediction model for each staining method.

[0118] In one exemplary embodiment, training the single staining prediction model includes using the area under the working feature curve (ROC) of the LN pathological images corresponding to each staining method as an evaluation metric.

[0119] It should also be noted that during the training of the single-color prediction model, the log loss function is used to calculate the loss value, and the SGD optimizer is used to update the model parameters, with momentum set to 0.9 and weight decay set to 5e. -4 Repeat the iteration until the model accuracy converges.

[0120] The structure of the multi-stain prediction model includes: a feature extraction layer, a multi-instance learning pooling layer, and a fully connected layer. The input to the feature extraction layer is the input data, and its output is connected to the input of the multi-instance learning pooling layer. The output of the multi-instance learning pooling layer is connected to the input of the fully connected layer, and the output of the fully connected layer is the prediction result.

[0121] In one exemplary embodiment, such as Figure 3 As shown, in step S60, the LN pathological image to be evaluated is input into the multi-stain prediction model to obtain the prediction result of the LN pathological image to be evaluated, specifically including:

[0122] S601: Input the LN pathological image to be evaluated into the feature extraction layer of the multi-stain prediction model to extract features and obtain feature information;

[0123] S602: Input the feature information into the multi-instance learning pooling layer. The multi-instance learning pooling layer introduces an attention mechanism to aggregate the feature information through attention scores and output aggregated features.

[0124] S603: Input the aggregated features into the fully connected layer to obtain the prediction results of the LN pathological image to be evaluated.

[0125] The predicted result is the probability value of complete remission after 12 months of treatment based on LN pathological images.

[0126] Another embodiment of this application also provides a device for predicting the effect of LN-induced therapy based on pathological images, used to perform the above-described method, such as... Figure 4 As shown, the device includes:

[0127] Data acquisition module 1 is configured to acquire LN pathological images using four different staining methods, including HE staining, PAS staining, PASM staining, and Masson staining.

[0128] Data annotation module 2 is configured to annotate LN pathological images with therapeutic efficacy labels;

[0129] Data preprocessing module 3 is configured to divide LN pathological images with therapeutic labels into image blocks of preset sizes and construct a dataset based on the image blocks;

[0130] The single staining prediction model training module 4 is configured to train six different neural network models using a dataset to determine four single staining prediction models for LN pathological images with different staining methods.

[0131] The multi-chromosome prediction model building module 5 is configured to jointly construct a multi-chromosome prediction model by combining four different single-chromosome prediction models.

[0132] The efficacy prediction module 6 is used to input the LN pathological image to be evaluated into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated.

[0133] In one exemplary embodiment, such as Figure 5 As shown, the data acquisition module 1 also includes a data screening unit 11, which is configured to acquire electronic medical records of several LN patients, screen the electronic medical records of LN patients according to preset inclusion and exclusion criteria, determine candidate LN patients that meet the requirements for model training, and acquire full images of renal biopsy sections of candidate LN patients processed by four different staining methods.

[0134] In an exemplary embodiment, the data preprocessing module 3 is used to: convert LN pathological images with therapeutic efficacy labels into 10x magnified images, 20x magnified images, and 40x magnified images; cut the original-size LN pathological images, 10x magnified images, 20x magnified images, and 40x magnified images into image blocks of preset sizes; exclude image blocks whose proportion of non-tissue regions exceeds a preset threshold to obtain image blocks containing effective tissue regions; and divide the image blocks containing effective tissue regions into training datasets and test datasets according to a preset ratio.

[0135] In one exemplary embodiment, the single-stain prediction model training module 4 is used for:

[0136] Input the dataset into the neural network model DL m In the process, the neural network model DL is obtained. m The prediction results of LN pathological images by the neural network model DL m It can be any one of the following models: AlexNet, DenseNet-121, Inception-V3, ResNet-50, VGG-11, and Vision Transformer.

[0137] Analyzing neural network models DL m Predictive accuracy of LN pathological images using four different staining methods;

[0138] For each coloring method, a neural network model DL with the highest prediction accuracy is determined. m As a single-stain prediction model.

[0139] In one exemplary embodiment, the single-stain prediction model training module 4 is configured as follows:

[0140] In the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate.

[0141] In one exemplary embodiment, the single-stain prediction model training module 4 is configured as follows:

[0142] The single staining prediction model was trained with batch sizes of 8, 16, 32, and 64, and the batch size that optimizes the performance of the single staining prediction model for each staining method was selected.

[0143] Specifically, the structure of the single-color prediction model includes: a feature extraction layer, a multi-instance learning pooling layer, and a fully connected layer.

[0144] This device is the first to employ a multi-stain, cross-scale LN pathological image analysis strategy. It can comprehensively predict LN pathological images based on different staining and magnification levels, using the model's weakly supervised multi-instance learning for automated analysis. This model can extract features that are not visually identifiable or easily overlooked from pathological images. Compared to traditional manual semi-quantitative pathological classification methods, it provides richer pathological feature information and boasts advantages such as strong robustness and high prediction efficiency. The model exhibits strong scalability and adaptability, with no limitations on the number and types of stained pathological images, and can accommodate diverse datasets with high heterogeneity. That is, as long as an LN patient has more than one type of staining data, the model can perform inference and output prediction results. Furthermore, the model is used to predict the 12-month treatment effect in LN patients, which has positive significance for assisting in the diagnosis of LN induction therapy efficacy.

[0145] The specific limitations of the LN-induced therapy effect prediction device based on pathological images provided in this embodiment can be found in the embodiments of the LN-induced therapy effect prediction method based on pathological images described above, and will not be repeated here. Each module in the above-described LN-induced therapy effect prediction device based on pathological images can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0146] This application provides a computer device, such as... Figure 6 As shown, the computer device may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it causes the processor to perform the steps of the method for predicting the effect of LN-induced therapy based on pathological images, as described in any of the above embodiments.

[0147] The working process, working details, and technical effects of the computer device provided in this embodiment can be found in the embodiment of the prediction method for LN induction therapy effect based on pathological images described above, and will not be repeated here.

[0148] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the method for predicting the effect of LN-induced therapy based on pathological images as described in any of the above embodiments. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0149] The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiment of the method for predicting the effect of LN induction therapy based on pathological images described above, and will not be repeated here.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system described in this application can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for predicting the therapeutic effect of LN induction based on pathological images, characterized in that, The method includes: Pathological images of lupus nephritis (LN) were obtained using four different staining methods, including HE staining, PAS staining, PASM staining, and Masson staining; the LN was lupus nephritis. The pathological images of LN were labeled with therapeutic efficacy tags; LN pathological images with therapeutic labels are divided into image blocks of a preset size, and a dataset is constructed based on the image blocks; Using the dataset, six different neural network models were trained to determine four single-stain prediction models for LN pathological images with different staining methods. The single-stain prediction models were used to analyze LN pathological images to obtain prediction results of LN induction treatment effects. The four different single-stain prediction models are combined to construct a multi-stain prediction model. The LN pathological image to be evaluated is input into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated.

2. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 1, characterized in that, Before obtaining LN pathological images using four different staining methods, the following steps are also included: Electronic medical records of several LN patients were obtained, and the electronic medical records of LN patients were screened according to the preset inclusion and exclusion criteria to determine candidate LN patients who meet the requirements for model training. Full images of renal biopsy sections from candidate LN patients, processed using four different staining methods, were obtained.

3. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 1, characterized in that, Labeling the LN pathological images with therapeutic efficacy tags includes: The efficacy labels are added to the LN pathological images using annotation software. The efficacy labels include CR and nCR, where CR is used to indicate complete remission after 12 months of LN induction therapy, and nCR is used to indicate incomplete remission after 12 months of LN induction therapy.

4. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 1, characterized in that, The LN pathological images with therapeutic labels are divided into image blocks of a preset size, and a dataset is constructed based on the image blocks, including: LN pathological images with therapeutic labels were converted into 10x magnification, 20x magnification, and 40x magnification images; The original size LN pathological image, the 10x magnified image, the 20x magnified image, and the 40x magnified image were cut into image blocks of preset size respectively; Image blocks that exclude non-organic regions exceeding a preset threshold are used to obtain image blocks containing valid organized regions. Image patches containing effective organized regions are divided into training and testing datasets according to a preset ratio.

5. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 1, characterized in that, Using the dataset, six different neural network models were trained to determine four single-stain prediction models for LN pathological images using different staining methods, including: The dataset is input into the neural network model DLm to obtain the prediction results of the neural network model DLm on LN pathological images. The neural network model DLm can be any one of the following: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model and Vision Transformer model. The prediction accuracy of the neural network model DLm for LN pathological images with four different staining methods was analyzed. For each coloring method, a neural network model DLm with the highest prediction accuracy is selected as the single-coloring prediction model.

6. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 5, characterized in that, When training a single-color prediction model, a warm-up strategy is introduced to adjust the learning rate.

7. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 5, characterized in that, Training a single-color prediction model includes: Batch sizes were set to 8, 16, 32, and 64, and the batch size that best performed the single-stain prediction model for each staining method was selected.

8. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 7, characterized in that, Training a single-stain prediction model includes: The prediction accuracy of the single staining prediction model for LN pathological images corresponding to each staining method is determined based on the area under the working characteristic curve.

9. The method for predicting the therapeutic effect of LN induction based on pathological images according to any one of claims 5 to 8, characterized in that, The structure of a single-stain prediction model includes: Feature extraction layer, multi-instance learning pooling layer, and fully connected layer.

10. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 9, characterized in that, The structure of the multi-chromosome prediction model includes: Feature extraction layer, multi-instance learning pooling layer, and fully connected layer.

11. The method for predicting the therapeutic effect of LN induction based on pathological images according to claim 10, characterized in that, The LN pathological image to be evaluated is input into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated, including: The pathological image of LN to be evaluated is input into the feature extraction layer of the multi-stain prediction model to extract features and obtain feature information; The feature information is input into a multi-instance learning pooling layer. An attention mechanism is introduced into the multi-instance learning pooling layer to aggregate the feature information through attention scores and output aggregated features. The aggregated features are input into a fully connected layer to obtain the prediction results of the LN pathological image to be evaluated.

12. A device for predicting the therapeutic effect of LN induction based on pathological images, characterized in that, For performing the method as described in any one of claims 1 to 11, comprising: The data acquisition module is configured to acquire LN pathological images corresponding to four different staining methods, including HE staining, PAS staining, PASM staining, and Masson staining; the LN refers to lupus nephritis. The data annotation module is configured to annotate the LN pathological images with therapeutic efficacy labels; The data preprocessing module is configured to divide the LN pathological images with therapeutic efficacy labels into image blocks of a preset size, and construct a dataset based on the image blocks; The single staining prediction model training module is configured to train six different neural network models using the dataset to determine four single staining prediction models for LN pathological images with different staining methods. The multi-chromosome prediction model construction module is configured to jointly construct a multi-chromosome prediction model by combining the four different single-chromosome prediction models. The efficacy prediction module is used to input the LN pathological image to be evaluated into the multi-stain prediction model to obtain the prediction results of the LN pathological image to be evaluated.

13. The device for predicting the therapeutic effect of LN induction based on pathological images according to claim 12, characterized in that, The data acquisition module also includes a data filtering unit, which is configured to acquire electronic medical records of several LN patients, filter the electronic medical records of LN patients according to preset inclusion and exclusion criteria, and determine candidate LN patients that meet the requirements for model training. Full images of renal biopsy sections from candidate LN patients, processed using four different staining methods, were obtained.

14. The device for predicting the therapeutic effect of LN induction based on pathological images according to claim 12, characterized in that, The data preprocessing module is used for: LN pathological images with therapeutic labels were converted into 10x magnification, 20x magnification, and 40x magnification images; The original size LN pathological image, the 10x magnified image, the 20x magnified image, and the 40x magnified image were cut into image blocks of preset size respectively; Image blocks that exclude non-organic regions exceeding a preset threshold are used to obtain image blocks containing valid organized regions. Image patches containing effective organized regions are divided into training and testing datasets according to a preset ratio.

15. The device for predicting the therapeutic effect of LN induction based on pathological images according to claim 12, characterized in that, The single-stain prediction model training module is used for: The dataset is input into the neural network model DLm to obtain the prediction results of the neural network model DLm on LN pathological images. The neural network model DLm can be any one of the following: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model and Vision Transformer model. The prediction accuracy of the neural network model DLm for LN pathological images with four different staining methods was analyzed. For each coloring method, a neural network model DLm with the highest prediction accuracy is selected as the single-coloring prediction model.

16. The device for predicting the therapeutic effect of LN induction based on pathological images according to claim 15, characterized in that, The single-stain prediction model training module is configured as follows: When training the neural network model DLm, a warm-up strategy is introduced to adjust the learning rate.

17. The device for predicting the therapeutic effect of LN induction based on pathological images according to claim 15, characterized in that, The single-stain prediction model training module is configured as follows: The single staining prediction model was trained with batch sizes of 8, 16, 32, and 64, and the batch size that best performed the single staining prediction model for each staining method was selected.

18. The device for predicting the therapeutic effect of LN induction based on pathological images according to claim 15, characterized in that, The structure of the single-stain prediction model includes: Feature extraction layer, multi-instance learning pooling layer, and fully connected layer.

19. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images