A method for determining the degree of glomerular segmental sclerosis based on deep learning, a computer device, and a computer-readable storage medium

Abstract

The present invention discloses a method for determining the degree of glomerular segmental sclerosis based on deep learning, a computer device, and a computer-readable storage medium. The method of the present invention can be used to determine glomerular segmental sclerosis and its degree, providing an accurate basis for subsequent diagnosis and treatment. In the present invention, the average processing time per image is 4.768*10-7 ms, and the accuracy rate of the method of the present invention is above 97%. The method proposed in this application can be used to solve the problems of poor accuracy and poor effectiveness of current traditional image processing.

Description

Technical Field

[0001] The present invention specifically relates to a method for determining the degree of glomerular segmental sclerosis based on deep learning, a computer device, and a computer-readable storage medium. Background Art

[0002] The glomerulus is a mass of spherical capillary networks. The afferent arteriole enters the renal capsule from the vascular pole, divides into 5 - 8 branches, and then divides into many loop-shaped capillaries. These capillaries are coiled into 5 - 8 capillary lobules or segments. There is mesangial tissue connecting the capillaries within the lobule, and there are very few anastomotic branches between the capillaries. The capillaries of each lobule are successively concentrated into larger blood vessels, and then converge with the small blood vessels of other lobules to form the efferent arteriole, which leaves the glomerulus from the vascular pole.

[0003] Segmental glomerulosclerosis is a glomerular disease, and its histopathological feature is segmental scarring of the glomerulus, with or without the formation of foam cells and adhesions in the glomerular capillaries. Segmental means that some lobules of the glomerulus are involved; global sclerosis means the segmental hyaline change or scar formation of the entire glomerulus.

[0004] In the prior art, it is mainly determined whether there is glomerular segmental sclerosis through the clinical experience of doctors, usually using light microscopy, and the diagnostic efficiency of this method is slow. In addition, in the existing technology, some traditional image detection methods are mainly applied, such as using the APIs provided in opencv for segmentation or detection. Although this method can process the current pictures, it cannot provide a general method for processing a large number of pictures.

[0005] Although the traditional image processing method can process the current picture samples, it does not have good generalization ability for large-scale data sets. It is necessary to manually set parameters for individual picture data, which increases the time cost of sample processing. In addition, manually setting parameters cannot obtain the optimal parameter values, which also results in low accuracy. Summary of the Invention

[0006] In view of the above situation, to overcome the defects of the prior art, the present invention provides a method for determining the degree of glomerular segmental sclerosis based on deep learning, a computer device, and a computer-readable storage medium.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for determining the degree of glomerular segmental sclerosis based on deep learning includes the following steps:

[0009] (1) Input the pictures into the glomerulus segmentation and glomerular sclerosis classification model to obtain the classification result of each picture;

[0010] (2) Segment the glomerular capillary loop contour to obtain a list of glomerular capillary loop contours. If the length of the contour list is 0, it indicates that there are no capillary loops in the glomerulus; otherwise, it is determined that there are capillary loops and the corresponding capillary loop contour list is saved.

[0011] (3) Using the glomerular capillary loop contours obtained in step (2), calculate the sum of the contour areas of all capillary loops in each glomerulus, and calculate the ratio with the outer contour area of the segmentally sclerosed glomerulus obtained in step (1) to obtain an area ratio.

[0012] (4) Use a batch of image datasets to repeat the calculation in step (3) to obtain the area ratio, and then use the Kmeans clustering algorithm to obtain the threshold, which is the discrimination threshold for the degree of glomerular segmental sclerosis. Divide the degree of glomerular segmental sclerosis according to the threshold.

[0013] Further, in step (1), in the glomerular segmentation and glomerular sclerosis classification stage, the segmentation task and the classification task are combined into one task, and the segmented glomeruli are directly judged for their categories.

[0014] Further, in step (1), the unet model is used to perform both classification and segmentation tasks simultaneously. The output result of the downsampling is used for classification, and the result after the upsampling is used for the segmentation task; the efficientnet_b3 network model is added to the unet model.

[0015] Further, in step (1), the training process of the glomerular segmentation and glomerular sclerosis classification model includes dataset preprocessing, model construction, and training.

[0016] Further, dataset preprocessing: Eliminate some interfering pictures, and make picture masks and classification labels.

[0017] Further, model construction and training: The head module connects the segmentation and classification tasks, and the head module is connected to the neural network unet; the features extracted by the neural network unet are passed to the head module; then the result output by the head module is used to calculate the error between the predicted value and the true value using the cross-entropy loss function; the error is passed, and finally the weight and bias parameters are updated for iterative training until the error between the predicted value and the true value is close and the iteration stops.

[0018] Further, the production of the picture mask: According to the glomerular contour, create a blank picture with the same length and width as the picture using numpy.zeros, and then use the function drawContours in opencv to draw the glomerular contour on the blank picture, with the contour part filled with 1 and the remaining area being 0, thus completing the production of a picture mask.

[0019] Furthermore, during the training process of the glomerular segmentation and glomerulosclerosis classification model, the parameter settings are as follows: the picture size is 512×512, the number of training rounds is 200, the batch size is 16 pictures, and the learning rate is 0.01.

[0020] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it can implement the steps in the above-mentioned method for determining the degree of glomerular segmental sclerosis based on deep learning.

[0021] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it can implement the steps in the above-mentioned method for determining the degree of glomerular segmental sclerosis based on deep learning.

[0022] The beneficial effects of the present invention are as follows:

[0023] The method of the present invention can be used to determine glomerular segmental sclerosis and its degree, providing an accurate basis for subsequent diagnosis and treatment. In the present invention, the average processing time per picture is 4.768*10 -7 ms, and the accuracy rate of the method of the present invention is above 97%. The method proposed in this application can be used to solve the problems of poor accuracy and poor efficiency in current traditional image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of the method for determining the degree of glomerular segmental sclerosis based on deep learning of the present invention.

[0025] Figure 2 is a schematic diagram of a computer device.

[0026] Figure 3 is a picture containing the contour of the glomerular capillary loop and the glomerular contour. Among them, the inner contour of the outer circle is the contour b of the glomerular capillary loop, and the outer contour of the outer circle is the glomerular contour a.

[0027] Figure 4 is the initial pathological picture of Zhang in Example 1. DETAILED DESCRIPTION OF THE INVENTION

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that the specific implementation manners are only detailed descriptions of the present invention and should not be regarded as limitations of the present invention.

[0029] Example 1

[0030] A method for determining the degree of glomerular segmental sclerosis based on deep learning includes the following steps:

[0031] (1) Input the pictures into the segmentation and classification model to obtain the classification results of each picture; here, the pictures refer to the pictures collected by a pathological section scanner, which are pictures in tif format.

[0032] Specifically, in the glomerulus segmentation and glomerulosclerosis classification stage, the segmentation task and the classification task are combined into one task, and the segmented glomeruli are directly judged for their categories; the picture data set to be processed is batch-input into the glomerulus segmentation and glomerulosclerosis classification model. After inputting the batch of pictures, the classification results of each picture are obtained, and the output classification results and the corresponding picture information are saved.

[0033] In the present invention, an appropriate batch processing data set can be divided according to the video memory capacity. For example, the input pictures account for about 70% of the video memory. Of course, the input pictures can also account for about 60% of the video memory. The present invention does not make specific limitations on this.

[0034] In conventional operations, a segmentation network model is needed to first segment the glomeruli, and then a classification network model is used to classify glomerulosclerosis; in this way, two network models need to be trained, and the diagnostic efficiency is low. The present invention uses one network model to process two tasks simultaneously. The unet segmentation network model is used. The present invention improves the existing unet network model, and uses the unet model to perform classification and segmentation tasks simultaneously. The output result of the downsampling is used for classification, and the result after the upsampling is used for the segmentation task; the efficientnet_b3 network model is added to the model: specifically, the efficientnet_b3 network model is used in the encoding stage, and other parts remain unchanged; the encoding stage is directly replaced, and the encoding information output by the efficientnet_b3 and the unet decoding information are spliced using torch.cat.

[0035] The features extracted from the output of the backbone network model (shallow edge features and deep semantic features) are upsampled in the processing of the segmentation task to generate a two-dimensional score picture that is the same size as the original picture in width and height, and then the contours of the segmented glomeruli are found according to the score map using the findContours function in opencv. The backbone network model is the unet network.

[0036] In the classification task, the features extracted by the unet are connected to the fully connected layer to output the classification features, and then the softmax function is used to calculate the scores of each category. The category with the highest score is the final classification result. The above-mentioned categories refer to the categories of glomerulosclerosis, including global sclerosis, segmental sclerosis, no sclerosis, and others.

[0037] The score calculation formula for each category can be expressed as: S i =ei / Σ j e j ,

[0038] where S i represents the score of the current category i to be calculated, i represents the current category to be calculated, j represents all categories, and e i represents the probability value of one of the categories i, and Σ j e j represents the sum of the probability values corresponding to all categories.

[0039] The training process of the glomerular segmentation and glomerulosclerosis classification model is mainly divided into two steps:

[0040] (1) Dataset preprocessing. In the preprocessing stage, some interfering pictures need to be removed, and picture masks and classification labels are made;

[0041] The interfering pictures refer to the pictures that affect the algorithm performance, such as pictures with wrong labels, or very blurred pictures, etc.

[0042] Picture mask making: According to the glomerular contour, a blank picture with the same length and width as the picture is created using numpy.zeros, and then the glomerular contour is drawn on the blank picture using the function drawContours in opencv. The contour part is filled with 1, and the remaining area is 0. In this way, a picture mask is made. The classification label can be manually marked.

[0043] (2) Model construction and training. The construction of the model requires designing the neural network for training: the model segmentation head and the model classification head. The head module connects the two tasks (segmentation and classification), and the head module is connected to the neural network unet; the features extracted by the neural network are passed to the head module; then the error between the predicted value and the true value is calculated using the cross-entropy loss function for the result output by the head module. This error is passed through the backpropagation algorithm, and finally the weight bias parameters are updated for iterative training until the error between the predicted value and the true value is very close and the iteration stops. The cross-entropy loss function and the backpropagation algorithm used in the model are both conventional functions and algorithms in the prior art, and the present invention does not improve them.

[0044] Parameter settings in the training process of the glomerular segmentation and glomerulosclerosis classification model: picture size 512×512, number of training rounds 200 times, batch size 16 pictures, learning rate 0.01.

[0045] (2) Segment the glomerular capillary loop contour. Use a deep learning segmentation model to obtain the contour of the glomerular capillary loop. If the length of the contour list is 0, it means that there is no capillary loop in this glomerulus. Otherwise, it is determined that there is a capillary loop and the corresponding capillary loop contour list is saved;

[0046] The picture is input into the trained segmentation network model to obtain the contour of the capillary loop. Here, the contour refers to a list set of contours; because there may be multiple capillary loops, there will be multiple contours, and its contour is an array. All the contours are placed in a list, representing all the capillary loops obtained through the segmentation model. A capillary loop contour can basically be represented as [np.array([[],[]],‘i4’),np.array([[],[]],‘i4’)...np.array([[],[]],‘i4’)]; such a contour drawn on the picture is as Figure 3 shown. Among them, the inner contour of the outer circle is the glomerular capillary loop contour b, and the outer contour of the outer circle is the glomerular contour a.

[0047] In the present invention, the glomerular capillary loop contour is obtained by segmenting with a U-Net network segmentation model.

[0048] Model training process: The training of the model includes two parts: data preprocessing and model construction.

[0049] (1) In the data preprocessing stage, some abnormal data sets (i.e., data sets that will affect model training) are removed, and then segmentation mask pictures are made. The process of making segmentation mask pictures is as follows: First, use the np.zeros() function to generate a picture with the same length and width as the training picture; then, according to the labeled json file, draw the contour of the capillary loop in the above-generated picture, so as to obtain the mask picture of the training picture.

[0050] (2) Build the model network. The present invention uses a U-Net network model.

[0051] In the training stage, the data is input into the built network model. The input value is upsampled to obtain a predicted mask picture. Then, the predicted mask and the label mask picture made above are used to calculate the error with the cross-entropy loss function. The obtained error is propagated through the backpropagation algorithm and the weight and bias parameters are updated. Stop after iterating the preset number of rounds in this way. The cross-entropy loss function and backpropagation algorithm used in the model are both conventional functions and algorithms in the prior art, and the present invention does not improve them.

[0052] Parameter settings during the training of the glomerular capillary loop contour segmentation model: batch size = 32, learning rate lr = 0.01, number of iteration rounds epoch = 200.

[0053] (3) Using the glomerular capillary loop contour obtained in step (2), sum up the contour areas of all capillary loops of each glomerulus, and take the ratio of the sum to the outer contour area of the segmentally sclerosed glomerulus obtained in step (1) to obtain an area ratio.

[0054] (4) Repeat the calculation in step (3) using a reasonable number of picture datasets. Then, use the machine learning Kmeans clustering algorithm for the obtained area ratios. The resulting threshold is the discrimination threshold for the degree of glomerular segmental sclerosis. In the present invention, the conventional Kmeans algorithm in the prior art is used to find the threshold. As a tool for finding the threshold, the present invention does not improve it.

[0055] In this embodiment, the calculation in step (3) is repeated using approximately 1000 pictures. To make the obtained threshold more generalizable, a larger picture dataset can be selected for processing; the specific number of pictures can be determined according to specific task requirements.

[0056] Specifically, the method for determining the degree of glomerular segmental sclerosis based on deep learning includes the following steps:

[0057] As Figure 4 shown, obtain the pathological picture of Zhang, and then cut the pathological picture into small pictures. The specific process of cutting small pictures: obtain the glomerular contour (obtain the contours of all glomeruli in Zhang's pathological picture according to the pre - algorithm, or obtain the glomerular contour according to other conventional methods in the prior art, or the glomerular contour is known in advance).

[0058] Then, according to the glomerular contour, obtain the position coordinates of the center point of the contour. The position of the center can be obtained by getting the coordinates of the upper - left corner (l, t) and the lower - right corner (r, b) of the contour, and using (l + r) / / 2, (t + b) / / 2. Then take the largest side in (r – l, b - t) and denote it as hw, and expand this side by 1.5 times. Then, taking the center point as the standard, use the formula: (l + r) / / 2 - hw / / 2, (t + b) / / 2 – hw / / 2, (l + r) / / 2 + hw / / 2, (t + b) / / 2 to obtain the cutting positions of the picture, that is, the upper - left and lower - right positions of the cut picture. The cutting window is a rectangular frame, and thus the cut small picture is obtained. Then input the small picture into the trained glomerular segmentation and glomerular sclerosis classification model to obtain the contour of the glomerular sclerosis area and the glomerular sclerosis classification for each picture (the categories of glomerular sclerosis include: global sclerosis, segmental sclerosis, no sclerosis, and others).

[0059] The input image of the segmentation model of the glomerular capillary loop contour is an image with a classification result of glomerular segmental sclerosis. If the image category obtained in the above steps is glomerular global sclerosis or other categories, then this image is discarded and the following steps are no longer performed.

[0060] The image obtained in the above steps (the classification result of this image is glomerular segmental sclerosis) is input into the trained segmentation model of the glomerular capillary loop contour to obtain the glomerular capillary loop contour. Then, the area of segmental sclerosis is compared with the area of the glomerular capillary loop to obtain an area ratio T. After calculating a batch of data volumes, a batch of area ratios can be obtained. These batch of area ratios are clustered using the kmeans algorithm to obtain the threshold for discriminating the degree of glomerular sclerosis. Thus, the degree of glomerular segmental sclerosis is classified according to the threshold.

[0061] Classification of the degree of glomerular segmental sclerosis: mild, moderate, severe;

[0062] In this embodiment, if the area ratio T is greater than 0 and less than 0.0653, then the degree of glomerular segmental sclerosis is mild.

[0063] If the area ratio T is greater than or equal to 0.0653 and less than 0.187, then the degree of glomerular segmental sclerosis is moderate.

[0064] If the area ratio T is greater than or equal to 0.187 and less than or equal to 1, then the degree of glomerular segmental sclerosis is severe.

[0065] In this embodiment, the threshold for discriminating the degree of glomerular sclerosis is obtained by calculating a batch of test sample data (about 3000 images), and then clustering the area ratio of segmental sclerosis and the glomerular capillary loop using the kmeans algorithm. The kmeans algorithm adopted in the present invention is conventional in the prior art and has not been improved.

[0066] In this embodiment, when the method of the present invention is used to process specific images, the average processing time per image is 4.768*10 - 7 ms, and the accuracy rate is 97.84%.

[0067] Example 2, referring to the appendix Figure 2 。

[0068] In this embodiment, a computer device 100 is provided, including a memory 102, a processor 101, and a computer program 103 stored on the memory 102 and executable on the processor 101. When the processor 101 executes the computer program 103, it can implement the steps in the method for determining the degree of glomerular segmental sclerosis based on deep learning provided in the above Embodiment 1.

[0069] Embodiment 3

[0070] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can implement the steps in the method for determining the degree of glomerular segmental sclerosis based on deep learning provided in each of the above embodiments.

[0071] In this embodiment, the computer program may be the computer program in Embodiment 2.

[0072] In this embodiment, the computer-readable storage medium may be run by the computer device in Embodiment 2.

[0073] Those of ordinary skill in the art can understand that implementing all or part of the processes in the above embodiments can be accomplished by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention may include non-volatile and / or volatile memories. Non-volatile memories may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0074] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0075] The above embodiments only represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the protection scope. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A method for determining the degree of glomerular segmental sclerosis based on deep learning, characterized in that, It includes the following steps: (1) Input the pictures into the glomerular segmentation and glomerulosclerosis classification model to obtain the classification results of each picture. The classification results are the categories of glomerulosclerosis, and the categories of glomerulosclerosis include global sclerosis, segmental sclerosis, and no sclerosis; (2) Input the pictures of glomerular segmental sclerosis into the segmentation model of the glomerular capillary loop contour to segment the glomerular capillary loop contour and obtain a list of glomerular capillary loop contours. If the length of the contour list is 0, it means that there is no capillary loop in this glomerulus. Otherwise, it is determined that there is a capillary loop and the corresponding capillary loop contour list is saved; (3) Use the glomerular capillary loop contours obtained in step (2) to calculate the sum of the contour areas of all capillary loops of each glomerulus, and divide it by the outer contour area of the segmentally sclerosed glomerulus obtained in step (1) to obtain an area ratio; (4) Repeat step (3) using a batch of picture datasets to calculate the area ratios, cluster the batch of area ratios using the kmeans algorithm to obtain the threshold for determining the degree of glomerulosclerosis, and divide the degree of glomerular segmental sclerosis according to the threshold.

2. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 1, characterized in that, In step (1), in the glomerular segmentation and glomerulosclerosis classification stage, the segmentation task and the classification task are combined into one task, and the segmented glomeruli are directly judged for their categories.

3. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 1, characterized in that, In step (1), the unet model is used to perform both classification and segmentation tasks simultaneously. The output result of the downsampling is used for classification, and the result after the upsampling is used for the segmentation task; the efficientnet_b3 network model is added to the unet model.

4. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 1, characterized in that, In step (1), the training process of the glomerular segmentation and glomerulosclerosis classification model includes dataset preprocessing, model construction, and training.

5. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 4, characterized in that, Dataset preprocessing: Eliminate interfering pictures and make picture masks and classification labels.

6. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 4, characterized in that, Model construction and training: The head module connects the segmentation and classification tasks, and the head module is connected to the neural network unet; the features extracted by the neural network unet are passed to the head module; then the error between the predicted value and the true value is calculated using the cross-entropy loss function for the result output by the head module; the error is passed, and finally the weight and bias parameters are updated for iterative training until the error between the predicted value and the true value is close and the iteration stops.

7. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 5, characterized in that, Making of the picture mask: According to the glomerular contour, create a blank picture with the same length and width as the picture using numpy.zeros, and then use the function drawContours in opencv to draw the glomerular contour on the blank picture. The contour part is filled with 1, and the remaining area is 0, so that a picture mask is made.

8. The method for determining the degree of glomerular segmental sclerosis based on deep learning according to claim 4, characterized in that, Parameter settings in the training process of the glomerular segmentation and glomerulosclerosis classification model: Picture size 512×512, number of training rounds 200 times, batch size 16 pictures, learning rate 0.

01.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can implement the steps in any one of claims 1-8 of a method for determining the degree of glomerular segmental sclerosis based on deep learning.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, it can implement the steps in a method for determining the degree of glomerular segmental sclerosis based on deep learning according to any one of claims 1-8.

Images