Prediction method and device for LN induction treatment effect based on pathological image

Abstract

The invention belongs to the technical field of medical image processing, and discloses a method and a device for predicting an LN induction treatment effect based on a pathological image. The method comprises the following steps: acquiring LN pathological images of four different dyeing methods, wherein the dyeing methods comprise an HE dyeing method, a PAS dyeing method, a PASM dyeing method and a Masson dyeing method; labeling a curative effect label on the LN pathological image; dividing the LN pathological image with the curative effect label into image blocks, and constructing a data set based on the image blocks; training the six different neural network models by using the data set to obtain four single-staining prediction models for predicting LN pathological images of different staining methods; combining the four different single-staining prediction models to construct a multi-staining prediction model; and inputting a to-be-evaluated LN pathological image into the multi-staining prediction model to obtain a prediction result of the to-be-evaluated LN pathological image. The prediction precision of the LN induction treatment effect can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image recognition, and in particular to a method and device for predicting the effect of LN induction treatment based on pathological images. Background Art

[0002] Lupus nephritis (LN) is a serious complication of systemic lupus erythematosus, affecting approximately 50% to 80% of SLE patients. Currently, the main treatments for lupus nephritis include standard induction therapy with glucocorticoids and advanced immunosuppressants. The efficacy of induction remission therapy is one of the key factors determining the prognosis of lupus nephritis (LN). The 10-year renal survival rate of partial remission patients is only 45%, and the 10-year renal survival rate of non-remission patients is only 19%. The proportion of patients who are effective in the first-line treatment of cyclophosphamide (CTX) or mycophenolate mofetil (MMF) is small, and only 20% to 30% of LN patients can achieve complete remission within 6-12 months of induction therapy. Accurate prediction of the efficacy of induction remission therapy is crucial for early identification of high-risk LN patients who do not respond to standard treatment regimens, which helps to make personalized and enhanced treatment decisions, thereby improving patients' long-term renal prognosis. Renal puncture biopsy is the gold standard for determining the efficacy of LN induction therapy, and can provide information including glomerular lesions, tubulointerstitial damage, and vascular lesions that are helpful in predicting patient prognosis. However, the evaluation results of renal biopsy are greatly affected by the experience level of the pathologist. With the development of microphotography and scanning technology, renal pathological sections can be digitized into high-resolution panoramic slice images (WSIs), which contain multi-scale feature information that exceeds human visual recognition capabilities and have the potential to predict the effect of LN induction therapy. Although some studies have attempted to use machine learning models to predict the effect of LN induction therapy through pathological images, current methods have not fully utilized the information in pathological images, and there are problems with poor model robustness and low accuracy of evaluation results. Summary of the invention

[0003] To this end, the embodiments of the present application provide a method and device for predicting the effect of LN induction therapy based on pathological images, which can effectively improve the prediction accuracy of the effect of LN induction therapy.

[0004] In a first aspect, the present application provides a method for predicting the effect of LN induction therapy based on pathological images.

[0005] The present application is realized by the following technical solution: a method for predicting the effect of LN induction treatment based on pathological images, the method comprising:

[0006] Acquire LN pathological images corresponding to four different staining methods, wherein the staining methods include HE staining method, PAS staining method, PASM staining method and Masson staining method;

[0007] Annotating the LN pathological image with a therapeutic effect label;

[0008] Dividing the LN pathological image with the therapeutic effect label into image blocks of a preset size, and constructing a data set based on the image blocks;

[0009] Six different neural network models are trained using the data set to determine four single staining prediction models for predicting LN pathological images of different staining methods, wherein the single staining prediction models are used to analyze LN pathological images to obtain prediction results of LN induction treatment effects;

[0010] The four different single staining prediction models are combined to construct a multi-staining prediction model;

[0011] The LN pathological image to be evaluated is input into the multi-staining prediction model to obtain the prediction result of the LN pathological image to be evaluated.

[0012] In a preferred example of the present application, it can be further configured that before acquiring LN pathological images of four different staining methods, the method further includes:

[0013] Obtain the electronic medical records of several LN patients, screen the electronic medical records of LN patients according to the preset inclusion and exclusion criteria, and determine the candidate LN patients who meet the model training requirements;

[0014] Full-length images of renal biopsy sections from candidate LN patients were obtained using four different staining methods.

[0015] In a preferred example of the present application, it can be further configured to label the LN pathological image with a therapeutic effect label, including:

[0016] The efficacy labels are added to the LN pathological image by annotation software, and the efficacy labels include CR and nCR, wherein CR is an efficacy label used to indicate complete remission after 12 months of LN induction treatment, and nCR is an efficacy label used to indicate incomplete remission after 12 months of LN induction treatment.

[0017] In a preferred example of the present application, it can be further configured that the LN pathological image with the efficacy label is divided into image blocks of a preset size, and a data set is constructed based on the image blocks, including:

[0018] Convert LN pathological images with therapeutic effect labels into 10x magnified images, 20x magnified images, and 40x magnified images;

[0019] Cutting the LN pathological image of the original size, the image of 10 times magnification, the image of 20 times magnification, and the image of 40 times magnification into image blocks of preset sizes respectively;

[0020] Excluding image blocks where the proportion of non-tissue areas exceeds a preset threshold, and obtaining image blocks containing valid tissue areas;

[0021] The image blocks containing the effective tissue area are divided into training data sets and test data sets according to a preset ratio.

[0022] In a preferred example of the present application, it can be further configured that six different neural network models are trained using the data set to determine four single staining prediction models for predicting LN pathological images of different staining methods, including:

[0023] Input the dataset to the neural network model DL m In the neural network model DL m For the prediction results of LN pathological images, the neural network model DL m Any of the following models: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model, and Vision Transformer model;

[0024] Analyzing Neural Network Models DL m Prediction accuracy of LN pathology images using four different staining methods;

[0025] For each dyeing method, a neural network model DL with the highest prediction accuracy is determined. m As a single staining prediction model.

[0026] In a preferred example of the present application, it can be further configured that, when training the single-coloring prediction model, a warm-up strategy is introduced to adjust the learning rate.

[0027] In a preferred example of the present application, it can be further configured that when training a single staining prediction model, the following steps are included:

[0028] The batch size was set to 8, 16, 32, and 64, from which the batch size that optimizes the performance of the single staining prediction model corresponding to each staining method was selected.

[0029] In a preferred example of the present application, it can be further configured that when training the single staining prediction model, it includes:

[0030] The prediction accuracy of the single staining prediction model for LN pathological images corresponding to each staining method was determined based on the area under the operating characteristic curve.

[0031] In a preferred example of the present application, it can be further configured that the structure of the single staining prediction model includes:

[0032] Feature extraction layer, multi-instance learning pooling layer and fully connected layer.

[0033] In a preferred example of the present application, it can be further configured that the structure of the multi-staining prediction model includes:

[0034] Feature extraction layer, multi-instance learning pooling layer and fully connected layer.

[0035] In a preferred example of the present application, it can be further configured that the LN pathological image to be evaluated is input into the multi-staining prediction model to obtain the prediction result of the LN pathological image to be evaluated, including:

[0036] The LN pathological image to be evaluated is input into the feature extraction layer of the multi-staining prediction model to extract features and obtain feature information;

[0037] The feature information is input into a multi-instance learning pooling layer, an attention mechanism is introduced into the multi-instance learning pooling layer, the feature information is aggregated through attention scores, and the aggregated features are output;

[0038] The aggregated features are input into the fully connected layer to obtain the prediction result of the LN pathological image to be evaluated.

[0039] In a second aspect, the present application provides a device for predicting the effect of LN induced treatment based on pathological images, which is used to execute the steps of the prediction method as described in the first aspect.

[0040] The present application is realized by the following technical solution: a device for predicting the effect of LN induction treatment based on pathological images, comprising:

[0041] A data acquisition module is configured to acquire LN pathological images corresponding to four different staining methods, wherein the staining methods include HE staining method, PAS staining method, PASM staining method and Masson staining method;

[0042] A data annotation module, configured to annotate the LN pathology image with a therapeutic effect label;

[0043] A data preprocessing module, configured to divide the LN pathology image with the efficacy label into image blocks of a preset size, and construct a data set based on the image blocks;

[0044] a single staining prediction model training module, configured to train six different neural network models using the data set to determine four single staining prediction models for predicting LN pathological images of different staining methods;

[0045] A multi-staining prediction model construction module is configured to jointly construct a multi-staining prediction model by combining the four different single-staining prediction models;

[0046] The efficacy prediction module is used to input the LN pathology image to be evaluated into the multi-staining prediction model to obtain the prediction result of the LN pathology image to be evaluated.

[0047] In a preferred example of the present application, it can be further configured that the data acquisition module also includes: a data screening unit, wherein the data screening unit is configured to obtain the electronic medical records of several LN patients, screen the electronic medical records of LN patients according to preset inclusion criteria and exclusion criteria, and determine candidate LN patients who meet the model training requirements; obtain full-volume images of renal biopsy sections of candidate LN patients processed by four different staining methods.

[0048] In a preferred example of the present application, it can be further configured that the data preprocessing module is used to:

[0049] Convert LN pathological images with therapeutic effect labels into 10x magnified images, 20x magnified images, and 40x magnified images;

[0050] Cutting the LN pathological image of the original size, the image of 10 times magnification, the image of 20 times magnification, and the image of 40 times magnification into image blocks of preset sizes respectively;

[0051] Excluding image blocks where the proportion of non-tissue areas exceeds a preset threshold, and obtaining image blocks containing valid tissue areas;

[0052] The image blocks containing the effective tissue area are divided into training data sets and test data sets according to a preset ratio.

[0053] In a preferred example of the present application, it can be further configured that the single staining prediction model training module is used to: input the data set into the neural network model DL m In the neural network model DL m For the prediction results of LN pathological images, the neural network model DL m Any of the following models: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model, and Vision Transformer model;

[0054] Analyzing Neural Network Models DL m Prediction accuracy of LN pathology images using four different staining methods;

[0055] For each dyeing method, a neural network model DL with the highest prediction accuracy is determined. m As a single staining prediction model.

[0056] In a preferred example of the present application, it can be further configured that the single staining prediction model training module is configured as follows:

[0057] In the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate.

[0058] In a preferred example of the present application, it can be further configured that the single staining prediction model training module is configured as follows:

[0059] The single staining prediction model was trained with batch sizes set to 8, 16, 32, and 64, and the batch size that optimizes the performance of the single staining prediction model corresponding to each staining method was selected.

[0060] In a preferred example of the present application, it can be further configured that the structure of the single staining prediction model includes:

[0061] Feature extraction layer, multi-instance learning pooling layer and fully connected layer.

[0062] On the third aspect, the present application is realized through the following technical solutions:

[0063] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned methods for predicting the effect of LN induction therapy based on pathological images are implemented.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium.

[0065] This application is achieved through the following technical solutions:

[0066] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for predicting the effect of LN induced treatment based on pathological images.

[0067] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0068] The present application obtains LN pathological images of four different staining methods; annotates the LN pathological images with efficacy labels; divides the LN pathological images with efficacy labels into image blocks of preset sizes, and constructs a data set based on the image blocks; uses the data set to train six different neural network models, and determines four single staining prediction models for predicting LN pathological images of different staining methods, and the single staining prediction models are used to analyze the LN pathological images to obtain the prediction results of the LN induction treatment effect; four different single staining prediction models are jointly constructed into a multi-staining prediction model; the LN pathological images to be evaluated are input into the multi-staining prediction model to obtain the prediction results of the LN pathological images to be evaluated. The present application integrates the features provided by LN pathological images of different staining methods and LN pathological images of different magnified sizes, and the trained single staining prediction model can capture the unique features of the LN pathological images of each staining method, and then combines multiple single staining prediction models to form a multi-staining prediction model, so as to realize the automated analysis of LN pathological images, extract rich pathological features that cannot be recognized by the naked eye from the LN pathological images, and have higher robustness and prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of a process of predicting the effect of LN induction therapy based on pathological images provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of a process for preprocessing LN pathological images provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of a processing flow of a LN pathology image to be evaluated by a multi-staining prediction model provided in an embodiment of the present application;

[0072] Figure 4 A schematic diagram of the structure of a device for predicting the effect of LN induction therapy based on pathological images provided in one embodiment of the present application;

[0073] Figure 5 A schematic diagram of the structure of a device for predicting the effect of LN induction therapy based on pathological images provided in another embodiment of the present application;

[0074] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application;

[0075] Description of reference numerals:

[0076] Data acquisition module 1, data screening unit 11, data annotation module 2, data preprocessing module 3, single staining prediction model training module 4, multi-staining prediction model construction module 5, efficacy prediction module 6. DETAILED DESCRIPTION

[0077] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0079] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0080] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0081] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0082] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0083] like Figure 1 As shown, the first exemplary embodiment of the present application provides a method for predicting the effect of LN induction treatment based on pathological images, which includes:

[0084] S10: Obtain LN pathological images corresponding to four different staining methods, including HE staining method, PAS staining method, PASM staining method and Masson staining method.

[0085] Because each staining method can highlight different tissue structures and features, in order to make full use of the characteristic information in LN pathological images, this application uses four different staining methods for LN pathological images for analysis and processing. Specifically, the renal biopsy sections of LN patients treated with HE staining, PAS staining, PASM staining and Masson staining are scanned using the scanner KF-PRO-120-HI or the scanner KFBIO to generate digital WSI (also called full-volume image), and the digital WSI is converted into LN pathological images in TIFF format.

[0086] Among them, after HE staining (hematoxylin and eosin staining), the cell nucleus is blue and the cytoplasm is red. The LN pathological images corresponding to the HE staining method can clearly reflect various cell components, including the thickness of the glomerular basement membrane (GBM), glomerular proliferation, necrosis and exudative lesions, tubular epithelial cell damage, interstitial edema, interstitial hemorrhage, inflammatory cell infiltration, vascular inflammation and other information. After PAS staining (periodic acid Schiff staining), the mesangial matrix and membranous structure are purple-red, and the cell nucleus is blue. The LN pathological images corresponding to the PAS staining method can reflect the changes in the membranous structure and whether the mesangial matrix increases. It has advantages in observing basement membrane lesions, the increase of extracellular matrix, the proliferation and exudation of various cells, and identifying whether the deposited substances are rich in sugar chains. The LN pathological images obtained by PASM staining (hexamine silver staining) can clearly show the basement membrane lesions and extracellular matrix, which is convenient for the observation of the spikes, chain rings and double track signs of the basement membrane, and can also observe the deposition of vascular elastic membrane and argyrophilic protein in the arteries. Masson trichrome staining is a connective tissue staining method. Masson staining (trichrome staining) shows that the basement membrane, mesangial matrix and collagen are green (or blue), immune complexes, fibrinoid necrosis, and thrombus are all red, and the cell nucleus is blue-black. The LN pathological images obtained by Masson trichrome staining can well distinguish collagen fibers, elastic fibers and reticular fibers.

[0087] S20: Label the LN pathology image with therapeutic efficacy labels.

[0088] The LN pathology images are labeled with efficacy labels according to the electronic medical records that match each LN pathology image. Specifically, the electronic medical records of the patients corresponding to each LN pathology image need to be obtained first, and the electronic medical records include at least clinical data, pathology data, treatment data, follow-up data, and biopsy results determined by at least two renal pathology experts with more than five years of clinical experience.

[0089] S30: Divide the LN pathological image with the efficacy label into image blocks of a preset size, and construct a data set based on the image blocks.

[0090] In order to adapt to the neural network model, an image processing tool is used to cut the LN pathology image with the efficacy label into multiple image blocks of the same size and non-overlapping, and a corresponding efficacy label is assigned to each image block according to the efficacy label of the original LN pathology image. All image blocks corresponding to each LN pathology image are packaged into data packets for storage, and all data packets are divided into training data sets and verification data sets according to a preset ratio using a randomization method. For example, the training data set and the verification data set can be divided according to a ratio of 7:3 or 8:2. The training data set is used to train subsequent models, and the verification data set is used to verify the performance of the trained model and adjust the parameters of the model. The above ratios are only used for examples. The training data set and the verification data set can be set according to actual needs and are not limited in this application.

[0091] S40: Six different neural network models are trained using the data set to determine four single staining prediction models for predicting LN pathological images using different staining methods. The single staining prediction models are used to analyze LN pathological images to obtain prediction results of LN induction therapy effects.

[0092] In the actual implementation process, six deep learning model architectures (neural network models) were initially trained, including the AlexNet model, the DenseNet-121 model, the Inception-V3 model, the ResNet-50 model, the VGG-11 model, and the Vision Transformer model, to determine the best network model for each histological staining method. A single staining prediction model with the best prediction performance was determined for each of the four staining methods, and the first single staining prediction model was constructed for the HE staining method, the second single staining prediction model was constructed for the PAS staining method, the third single staining prediction model was constructed for the PASM staining method, and the fourth single staining prediction model was constructed for the Masson staining method.

[0093] S50: Four different single staining prediction models are combined to construct a multi-staining prediction model.

[0094] Specifically, the first single staining prediction model, the second single staining prediction model, the third single staining prediction model and the fourth single staining prediction model are integrated to construct a multi-staining prediction model, which is called the LNCR-DL model. Exemplarily, different weights can be assigned to the single staining prediction models according to the prediction accuracy of the LN pathological images corresponding to each staining method by the first single staining prediction model, the second single staining prediction model, the third single staining prediction model and the fourth single staining prediction model, and then the weighted average is used to obtain the multi-staining prediction model.

[0095] The multi-staining prediction model can integrate the prediction advantages of each single-staining prediction model for different staining methods and can provide more accurate and comprehensive prediction results.

[0096] S60: Inputting the LN pathological image to be evaluated into the multi-staining prediction model to obtain a prediction result of the LN pathological image to be evaluated.

[0097] This application integrates the features provided by LN pathology images of different staining methods, and the trained single staining prediction model can capture the unique features of LN pathology images of each staining method. Multiple single staining prediction models are then combined to form a multi-staining prediction model to achieve automated analysis of LN pathology images, extract rich pathological features that cannot be recognized by the naked eye from LN pathology images, and have higher robustness and prediction accuracy.

[0098] In an exemplary embodiment, before acquiring LN pathological images of four different staining methods in step S10, the method further includes:

[0099] The electronic medical records of several LN patients were obtained, and the electronic medical records of LN patients were screened according to the preset inclusion and exclusion criteria to determine the candidate LN patients who met the model training requirements; the full-volume images of the renal biopsy sections of the candidate LN patients processed by four different staining methods were obtained. Among them, the inclusion criteria of the electronic medical records of LN patients were: 1) age ≥ 1 year old; 2) LN was confirmed by renal biopsy and biopsy within 6 months before induction therapy; 3) received induction therapy with glucocorticoids combined with cyclophosphamide (CTX) or glucocorticoids combined with mycophenolate mofetil (MMF); 4) followed up for at least 12 months after the start of induction therapy. The exclusion criteria of electronic medical records were: LN patients who received other immunosuppressive treatment within 6 months before induction therapy, or patients with incomplete clinical data and follow-up data for treatment response evaluation.

[0100] In an exemplary embodiment, when the LN pathology image is annotated with a therapeutic effect label in step S20, the specific steps include:

[0101] The efficacy labels are added to the LN pathological images by annotation software. The efficacy labels include CR and nCR, where CR is an efficacy label indicating complete remission after 12 months of LN induction treatment, and nCR is an efficacy label indicating incomplete remission after 12 months of LN induction treatment. Among them, the standard for complete remission is 24-hour urine protein <0.5g, eGFR is normal or within 10% of the normal range; the standard for partial remission is 24-hour urine protein decrease of ≥50%, and the level is <3g / d, and eGFR improves or stabilizes (±25%); no response means neither complete remission nor partial remission.

[0102] In an exemplary embodiment, Figure 2 As shown, in step S30, the LN pathological image with the therapeutic effect label is divided into image blocks of a preset size, and a data set is constructed based on the image blocks, including:

[0103] S301: Convert the LN pathology image with the efficacy label into a 10-fold magnified image, a 20-fold magnified image, and a 40-fold magnified image. Each LN pathology image can be magnified 10 times, 20 times, and 40 times by image processing software or programming libraries, such as the Pillow library in Python, and the magnified images are saved as new files. For example, convert the LN pathology image P i After zooming in 10 times, 20 times and 40 times respectively, save it as P i-10 , P i-20 and P i-40 , i is the LN pathological image P i The unique number of the

[0104] S302: Cutting the LN pathological image of the original size, the image magnified 10 times, the image magnified 20 times, and the image magnified 40 times into image blocks of preset sizes respectively.

[0105] Use image processing tools (such as Python's PIL library) to cut the image, and use a combined number format to record the position of each image block in the original LN pathology image. The combined number format can be image ID_row number_column number. For example, P i-10,2,3 Represents LN pathological image P i The image block of the 2nd row and 3rd column in the 10-fold magnified image can be set according to the requirements, for example, it can be a 512×512 pixel block. According to the size of the image and the size of the image block, the number of rows and columns cut out can be calculated. i All the image blocks cut out are packaged into data packets Packet i Save and save the LN pathological image P i All image blocks cut out from the 10 times magnified image are packaged into packets. i-10 Save and save the LN pathological image P i All image blocks cut out from the 20 times magnified image are packaged into packets i-20 Save and save the LN pathological image P i All image blocks cut out from the 40 times magnified image are packaged into packets i-40 to save.

[0106] S303: Exclude image blocks in which the proportion of non-tissue areas exceeds a preset threshold, and obtain image blocks containing valid tissue areas.

[0107] Extract the pixel value of the image block, and judge whether it belongs to the non-tissue area or the tissue area according to the pixel value of the image block. Specifically, the pixels with a pixel value greater than 225 in the image block are considered to be non-tissue areas, and the proportion of pixels with a pixel value greater than 225 in each image block is counted. If the proportion exceeds the preset threshold, it is considered that the proportion of the non-tissue area of ​​the image block is too high, and the image block is deleted. The preset threshold can be set according to actual needs, for example, set to 50%. By excluding image blocks that are mostly non-tissue areas, the impact of useless data on model training can be reduced, thereby improving training efficiency. At the same time, the overfitting of the model to noise or irrelevant features during the training process can be reduced, and the stability and prediction accuracy of the model can be improved.

[0108] S304: Divide the image blocks containing the effective tissue area into a training data set and a test data set according to a preset ratio.

[0109] Before model building, the image blocks corresponding to each histological staining method were color normalized using StainTools to minimize staining variability.

[0110] In an exemplary embodiment, in step S40, six different neural network models are trained using the training data set to obtain four single staining prediction models for predicting LN pathological images of different staining methods, specifically including:

[0111] S401: Input the data set into the neural network model DL m In the neural network model DL m For LN pathological images P i j The prediction results of the neural network model DL m Any of the following models: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model, and Vision Transformer model;

[0112] S402: Analyze the neural network model DL m Prediction accuracy of LN pathology images using four different staining methods;

[0113] S403: Determine a neural network model DL with the highest prediction accuracy for each dyeing method m As a single staining prediction model. Specifically, the data set is input into the neural network model DL m Train in and get the neural network model DL mFor the predicted value of each LN pathological image in the data set, the prediction accuracy of the LN pathological image is determined based on the predicted value and the efficacy label, and a neural network model DL m The first prediction accuracy for the original size image of a LN pathology image, the second prediction accuracy for the 10-fold magnified image, the third prediction accuracy for the 20-fold magnified image, and the fourth prediction accuracy for the 40-fold magnified image are calculated as the average of the first prediction accuracy, the second prediction accuracy, the third prediction accuracy, and the fourth prediction accuracy as the neural network model DL i The prediction accuracy of LN pathological images. m After the prediction accuracy of LN pathological images of four different staining methods, a neural network model DL with the highest prediction accuracy was determined for each staining method. m As a single staining prediction model.

[0114] Exemplarily, the data set is input into the AlexNet model for training to obtain the predicted value of the AlexNet model for each LN pathological image, and the prediction accuracy of the AlexNet model for each LN pathological image is determined based on the predicted value and the real efficacy label of the LN pathological image. Among them, the prediction accuracy of each LN pathological image is the average prediction accuracy of the AlexNet model for four sizes of LN pathological images (original size, 10 times magnification, 20 times magnification and 40 times magnification). The LN pathological images are classified according to the four staining methods, and the prediction accuracy of the AlexNet model for the LN pathological images of the four staining methods can be counted. Similarly, the data set is input into the DenseNet-121 model for training to obtain the predicted value of the DenseNet-121 model for each LN pathological image, and the prediction accuracy of the DenseNet-121 model for the LN pathological images of the four staining methods is obtained by statistics according to the same steps as the AlexNet model. Then, the prediction accuracy of the Inception-V3 model, ResNet-50 model, VGG-11 model and Vision Transformer model for LN pathological images of four staining methods was obtained. Based on the prediction accuracy of the six neural network models for LN pathological images of four staining methods, the neural network model with the highest prediction accuracy corresponding to each staining method was determined as the single staining prediction model of the staining method. Among them, it was determined through analysis that Inception-V3 had the best prediction effect for HE, PASM and TRI staining, while Vision Transformer had the highest prediction ability for PAS staining among all neural networks.

[0115] The structure of the single color prediction model includes: feature extraction layer, multi-instance learning pooling layer and fully connected layer. The input end of the feature extraction layer is the input data, the output end of the feature extraction layer is connected to the input end of the multi-instance learning pooling layer, the output end of the multi-instance learning pooling layer is connected to the input end of the fully connected layer, and the output end of the fully connected layer outputs the prediction result.

[0116] In an exemplary embodiment, when the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate. Specifically, the number of learning times is preset to 5, starting from a lower initial learning rate, and further adjusting the initial learning rate through cosine annealing.

[0117] In an exemplary embodiment, training the single staining prediction model includes: setting the batch size to 8, 16, 32 and 64 to train the single staining prediction model, and selecting the batch size that optimizes the performance of the single staining prediction model corresponding to each staining method.

[0118] In an exemplary embodiment, training the single staining prediction model includes: using the single staining prediction model to use the area under the operating characteristic curve (ROC) of the LN pathology image corresponding to each staining method as an evaluation indicator.

[0119] It should also be noted that in the training process of the single-color prediction model, the logarithmic loss function is used to calculate the loss value, and the SGD optimizer is used to update the model parameters, with the momentum set to 0.9 and the weight decay set to 5e -4 , repeat the iteration until the model accuracy converges.

[0120] The structure of the multi-coloring prediction model includes: feature extraction layer, multi-instance learning pooling layer and fully connected layer. The input end of the feature extraction layer is the input data, the output end of the feature extraction layer is connected to the input end of the multi-instance learning pooling layer, the output end of the multi-instance learning pooling layer is connected to the input end of the fully connected layer, and the output end of the fully connected layer outputs the prediction result.

[0121] In an exemplary embodiment, Figure 3 As shown, step S60, inputting the LN pathological image to be evaluated into the multi-staining prediction model to obtain the prediction result of the LN pathological image to be evaluated, specifically includes:

[0122] S601: inputting the LN pathological image to be evaluated into the feature extraction layer of the multi-staining prediction model to extract features and obtain feature information;

[0123] S602: inputting feature information into a multi-instance learning pooling layer, introducing an attention mechanism into the multi-instance learning pooling layer, aggregating feature information through attention scores, and outputting aggregated features;

[0124] S603: Input the aggregated features into the fully connected layer to obtain the prediction result of the LN pathological image to be evaluated.

[0125] The prediction result is the probability that the LN pathological image is considered to be a complete remission after 12 months of treatment.

[0126] Another embodiment of the present application also provides a prediction device for LN induction treatment effect based on pathological images, which is used to perform the above method, such as Figure 4 As shown, the device comprises:

[0127] A data acquisition module 1 is configured to acquire LN pathological images using four different staining methods, the staining methods including HE staining method, PAS staining method, PASM staining method and Masson staining method;

[0128] A data annotation module 2 is configured to annotate the LN pathology image with a therapeutic effect label;

[0129] A data preprocessing module 3 is configured to divide the LN pathology image with the efficacy label into image blocks of a preset size and construct a data set based on the image blocks;

[0130] The single staining prediction model training module 4 is configured to train six different neural network models using the data set to determine four single staining prediction models for predicting LN pathological images of different staining methods;

[0131] A multi-staining prediction model construction module 5 is configured to jointly construct a multi-staining prediction model by combining four different single-staining prediction models;

[0132] The efficacy prediction module 6 is used to input the LN pathological image to be evaluated into the multi-staining prediction model to obtain the prediction result of the LN pathological image to be evaluated.

[0133] In an exemplary embodiment, Figure 5 As shown, the data acquisition module 1 also includes: a data screening unit 11, which is configured to obtain electronic medical records of several LN patients, screen the electronic medical records of LN patients according to preset inclusion criteria and exclusion criteria, and determine candidate LN patients that meet the model training requirements; obtain full-volume images of renal biopsy sections of candidate LN patients processed by four different staining methods.

[0134] In an exemplary embodiment, the data preprocessing module 3 is used to: convert the LN pathology image with the efficacy label into a 10-fold magnified image, a 20-fold magnified image, and a 40-fold magnified image; cut the original-size LN pathology image, the 10-fold magnified image, the 20-fold magnified image, and the 40-fold magnified image into image blocks of preset sizes, respectively; exclude image blocks in which the proportion of non-tissue areas exceeds a preset threshold, and obtain image blocks containing effective tissue areas; divide the image blocks containing effective tissue areas into a training data set and a test data set according to a preset ratio.

[0135] In an exemplary embodiment, the single staining prediction model training module 4 is used to:

[0136] Input the dataset to the neural network model DL m In the neural network model DL m Prediction results of LN pathological images, neural network model DL m Any of the following models: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model, and Vision Transformer model;

[0137] Analyzing Neural Network Models DL m Prediction accuracy of LN pathology images using four different staining methods;

[0138] For each dyeing method, a neural network model DL with the highest prediction accuracy is determined. m As a single staining prediction model.

[0139] In an exemplary embodiment, the single staining prediction model training module 4 is configured to:

[0140] In the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate.

[0141] In an exemplary embodiment, the single staining prediction model training module 4 is configured to:

[0142] The single staining prediction model was trained with batch sizes set to 8, 16, 32, and 64, and the batch size that optimizes the performance of the single staining prediction model corresponding to each staining method was selected.

[0143] Specifically, the structure of the single staining prediction model includes: feature extraction layer, multi-instance learning pooling layer and fully connected layer.

[0144] The device adopts a multi-staining-cross-scale LN pathological image analysis strategy for the first time, which can make predictions based on the features provided by LN pathological images with different staining and different magnifications. The model automatically analyzes LN pathological images based on weakly supervised multi-instance learning, which can extract features in pathological images that cannot be recognized by the naked eye or are easily ignored. Compared with the traditional manual semi-quantitative pathological classification method, it can provide richer pathological feature information, and has the advantages of strong robustness and high prediction efficiency. The model has strong scalability and adaptability. It has no restrictions on the number and type of each stained pathological image, and can be compatible with a variety of data with large heterogeneity, that is, as long as the LN patient has more than one type of staining data, the model can perform inference and output prediction results. And the model is used to predict the 12-month treatment effect of LN patients, which has positive significance for assisting the diagnosis of the effect of LN induction therapy.

[0145] The specific definition of the prediction device of the LN induced treatment effect based on pathological images provided in this embodiment can be referred to the embodiment of the prediction method of the LN induced treatment effect based on pathological images above, and will not be repeated here. Each module in the above-mentioned prediction device of the LN induced treatment effect based on pathological images can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0146] The present application embodiment provides a computer device, such as Figure 6 As shown, the computer device may include a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of the method for predicting the effect of LN induction treatment based on pathological images as in any of the above embodiments.

[0147] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the above embodiment of the method for predicting the effect of LN induction treatment based on pathological images, and will not be described in detail here.

[0148] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for predicting the effect of LN induction therapy based on pathological images in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0149] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above-mentioned embodiment of the method for predicting the effect of LN induction therapy based on pathological images, and will not be described in detail here.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system described in the present application is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for predicting the effect of LN induction therapy based on pathological images, characterized in that: The method comprises: Acquire LN pathological images corresponding to four different staining methods, wherein the staining methods include HE staining method, PAS staining method, PASM staining method and Masson staining method; Annotating the LN pathological image with a therapeutic effect label; Dividing the LN pathological image with the therapeutic effect label into image blocks of a preset size, and constructing a data set based on the image blocks; Six different neural network models are trained using the data set to determine four single staining prediction models for predicting LN pathological images of different staining methods, wherein the single staining prediction models are used to analyze LN pathological images to obtain prediction results of LN induction treatment effects; The four different single staining prediction models are combined to construct a multi-staining prediction model; The LN pathological image to be evaluated is input into the multi-staining prediction model to obtain the prediction result of the LN pathological image to be evaluated.

2. The method for predicting the effect of LN induction therapy based on pathological images according to claim 1, characterized in that: Before acquiring LN pathology images using four different staining methods, the following were also included: Obtain the electronic medical records of several LN patients, screen the electronic medical records of LN patients according to the preset inclusion and exclusion criteria, and determine the candidate LN patients who meet the model training requirements; Full-length images of renal biopsy sections from candidate LN patients were obtained using four different staining methods.

3. The method for predicting the effect of LN induction therapy based on pathological images according to claim 1, characterized in that: The LN pathological image is labeled with a therapeutic effect label, including: An efficacy label is added to the LN pathological image by annotation software, wherein the efficacy label includes CR and nCR, wherein CR is an efficacy label used to indicate complete remission after 12 months of LN induction treatment, and nCR is an efficacy label used to indicate incomplete remission after 12 months of LN induction treatment.

4. The method for predicting the effect of LN induction therapy based on pathological images according to claim 1, characterized in that: Dividing the LN pathological image with the therapeutic effect label into image blocks of a preset size, and constructing a data set based on the image blocks, including: Convert LN pathological images with therapeutic effect labels into 10x magnified images, 20x magnified images, and 40x magnified images; Cutting the original-size LN pathological image, the 10-fold magnified image, the 20-fold magnified image, and the 40-fold magnified image into image blocks of preset sizes respectively; Excluding image blocks where the proportion of non-tissue areas exceeds a preset threshold, and obtaining image blocks containing valid tissue areas; The image blocks containing the effective tissue area are divided into training data sets and test data sets according to a preset ratio.

5. The method for predicting the effect of LN induction therapy based on pathological images according to claim 1, characterized in that: Six different neural network models were trained using the data set, and four single staining prediction models for predicting LN pathological images with different staining methods were determined, including: Input the dataset to the neural network model DL m In the neural network model DL m For the prediction results of LN pathological images, the neural network model DL m Any of the following models: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model, and Vision Transformer model; Analyzing Neural Network Models DL m Prediction accuracy of LN pathology images using four different staining methods; For each dyeing method, a neural network model DL with the highest prediction accuracy is determined. m As a single staining prediction model.

6. The method for predicting the effect of LN induction therapy based on pathological images according to claim 5, characterized in that: When training the single-color prediction model, a warm-up strategy is introduced to adjust the learning rate.

7. The method for predicting the effect of LN induction therapy based on pathological images according to claim 5, characterized in that: When training a single staining prediction model, include: The batch size was set to 8, 16, 32, and 64, from which the batch size that optimizes the performance of the single staining prediction model corresponding to each staining method was selected.

8. The method for predicting the effect of LN induction therapy based on pathological images according to claim 7, characterized in that: When training a single-stain prediction model, include: The prediction accuracy of the single staining prediction model for LN pathological images corresponding to each staining method was determined based on the area under the operating characteristic curve.

9. The method for predicting the effect of LN induction therapy based on pathological images according to any one of claims 5 to 8, characterized in that: The structure of the single-staining prediction model includes: Feature extraction layer, multi-instance learning pooling layer and fully connected layer.

10. The method for predicting the effect of LN induction therapy based on pathological images according to claim 9, characterized in that: The structure of the multi-staining prediction model includes: Feature extraction layer, multi-instance learning pooling layer and fully connected layer.

11. The method for predicting the effect of LN induction therapy based on pathological images according to claim 10, characterized in that: The LN pathological image to be evaluated is input into the multi-staining prediction model to obtain the prediction results of the LN pathological image to be evaluated, including: The LN pathological image to be evaluated is input into the feature extraction layer of the multi-staining prediction model to extract features and obtain feature information; The feature information is input into a multi-instance learning pooling layer, an attention mechanism is introduced into the multi-instance learning pooling layer, the feature information is aggregated through attention scores, and the aggregated features are output; The aggregated features are input into the fully connected layer to obtain the prediction result of the LN pathological image to be evaluated.

12. A device for predicting the effect of LN induction treatment based on pathological images, characterized in that: Used to perform the method according to any one of claims 1 to 11, comprising: A data acquisition module is configured to acquire LN pathological images corresponding to four different staining methods, wherein the staining methods include HE staining method, PAS staining method, PASM staining method and Masson staining method; A data annotation module, configured to annotate the LN pathology image with a therapeutic effect label; A data preprocessing module, configured to divide the LN pathology image with the efficacy label into image blocks of a preset size, and construct a data set based on the image blocks; a single staining prediction model training module, configured to train six different neural network models using the data set to determine four single staining prediction models for predicting LN pathological images of different staining methods; A multi-staining prediction model construction module is configured to jointly construct a multi-staining prediction model by combining the four different single-staining prediction models; The efficacy prediction module is used to input the LN pathology image to be evaluated into the multi-staining prediction model to obtain the prediction result of the LN pathology image to be evaluated.

13. The device for predicting the effect of LN induction therapy based on pathological images according to claim 12, characterized in that: The data acquisition module further includes: a data screening unit, which is configured to obtain electronic medical records of a number of LN patients, screen the electronic medical records of LN patients according to preset inclusion criteria and exclusion criteria, and determine candidate LN patients that meet the model training requirements; Full-length images of renal biopsy sections from candidate LN patients were obtained using four different staining methods.

14. The device for predicting the effect of LN induction therapy based on pathological images according to claim 12, characterized in that: The data preprocessing module is used to: Convert LN pathological images with therapeutic effect labels into 10x magnified images, 20x magnified images, and 40x magnified images; Cutting the original-size LN pathological image, the 10-fold magnified image, the 20-fold magnified image, and the 40-fold magnified image into image blocks of preset sizes respectively; Excluding image blocks where the proportion of non-tissue areas exceeds a preset threshold, and obtaining image blocks containing valid tissue areas; The image blocks containing the effective tissue area are divided into training data sets and test data sets according to a preset ratio.

15. The device for predicting the effect of LN induction therapy based on pathological images according to claim 12, characterized in that: The single staining prediction model training module is used to: Input the dataset to the neural network model DL m In the neural network model DL m For the prediction results of LN pathological images, the neural network model DL m Any of the following models: AlexNet model, DenseNet-121 model, Inception-V3 model, ResNet-50 model, VGG-11 model, and Vision Transformer model; Analyzing Neural Network Models DL m Prediction accuracy of LN pathology images using four different staining methods; For each dyeing method, a neural network model DL with the highest prediction accuracy is determined. m As a single staining prediction model.

16. The device for predicting the effect of LN induction therapy based on pathological images according to claim 15, characterized in that: The single staining prediction model training module is configured as follows: In the neural network model DL m During training, a warm-up strategy is introduced to adjust the learning rate.

17. The device for predicting the effect of LN induction therapy based on pathological images according to claim 15, characterized in that: The single staining prediction model training module is configured as follows: The single staining prediction model was trained with batch sizes set to 8, 16, 32, and 64, and the batch size that optimizes the performance of the single staining prediction model corresponding to each staining method was selected.

18. The device for predicting the effect of LN induction therapy based on pathological images according to claim 15, characterized in that: The structure of the single staining prediction model includes: Feature extraction layer, multi-instance learning pooling layer and fully connected layer.

19. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images