Deep learning-based kidney picture processing method, system, device and medium

Abstract

The application discloses a kidney picture processing method, system, device and medium based on deep learning. The method comprises the following steps: preprocessing a to-be-processed kidney picture to obtain a processed kidney picture; processing the processed kidney picture by using a deep learning segmentation model to obtain a kidney picture with glomerular contours; wherein the kidney picture with glomerular contours comprises a plurality of glomeruli; positioning capillary balls in each glomerulus from the kidney picture with glomerular contours to obtain a kidney picture with capillary ball contours; and positioning blood cells, cell nuclei and cell matrix in the capillary balls from the kidney picture with capillary ball contours respectively. The method disclosed by the application has high processing efficiency for the to-be-processed kidney picture, can accurately position the glomerular contours in the kidney, and accurately position the capillary balls, blood cells, cell nuclei and cell matrix in the glomeruli from the glomeruli, so that the acquisition of various index information in the glomeruli is realized.

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to the technical field of medical picture processing, and particularly relates to a kidney picture processing method, system, device and medium based on deep learning. BACKGROUND

[0002] Normal glomerulus contains different morphological and functional cells, including capillary endothelial cells, podocytes, mesangial cells, epithelial cells, blood cells, etc. Considering the high blood flow of the kidney and the special function of urine concentration, the kidney is very sensitive to the toxic effects of drugs and environmental chemicals. Drug toxicity (some heterologous proteins, ADC drugs and monoclonal antibodies, etc. Biological preparations), inflammation and immune mechanisms, etc. can cause the proliferation of cells in the glomerulus, thereby forming characteristic morphological changes and affecting the function of the glomerulus. Therefore, in the field of pathology, the diagnosis of glomerulonephritis is mainly undertaken by pathological analysis experts, and the accuracy and efficiency of the diagnosis results mainly depend on the knowledge reserve and reading experience of the pathological experts. Therefore, the existing kidney pathological picture processing efficiency is low, and the details of the glomerulonephritis in the kidney picture cannot be accurately located, and the index is single. SUMMARY

[0003] In view of the above defects or deficiencies in the prior art, it is desirable to provide a kidney picture processing method, system, device and medium based on deep learning.

[0004] In a first aspect, a kidney picture processing method based on deep learning is provided, comprising:

[0005] preprocessing a kidney picture to be processed to obtain a processed kidney picture;

[0006] processing the processed kidney picture using a deep learning segmentation model to obtain a kidney picture with glomerular contours; wherein the kidney picture with glomerular contours includes a plurality of glomeruli;

[0007] locating the capillary glomerulus in each glomerulus from the kidney picture with glomerular contours to obtain a kidney picture with capillary glomerulus contours;

[0008] Further locating blood cells, cell nuclei and cell matrix inside the capillary glomerulus from the kidney picture with capillary glomerulus contours, respectively.

[0009] In a second aspect, a kidney picture processing system based on deep learning is provided, comprising:

[0010] a processing module configured to preprocess a kidney picture to be processed to obtain a processed kidney picture;

[0011] a deep learning module, configured to process the processed kidney image by using a deep learning segmentation model to obtain a kidney image with glomerular contours, wherein the kidney image with glomerular contours includes a plurality of glomeruli;

[0012] a capillary positioning module, configured to locate capillary balls in each glomerulus from the kidney image with glomerular contours to obtain a kidney image with capillary ball contours;

[0013] a blood cell positioning module, configured to locate blood cells inside the capillary balls from the kidney image with capillary ball contours;

[0014] a nucleus positioning module, configured to locate cell nuclei inside the capillary balls from the kidney image with capillary ball contours;

[0015] a cell matrix positioning module, configured to locate cell matrices inside the capillary balls from the kidney image with capillary ball contours.

[0016] In a third aspect, an electronic device is provided, including:

[0017] one or more processors;

[0018] a memory for storing one or more programs,

[0019] when the one or more programs are executed by the one or more processors, the one or more processors perform the deep learning-based kidney image processing method provided in the embodiments of the present application.

[0020] In a fourth aspect, a computer-readable storage medium storing a computer program is provided, and the program is executed by a processor to implement the deep learning-based kidney image processing method provided in the embodiments of the present application.

[0021] According to the technical solutions provided in the embodiments of the present application, the glomeruli in the processed kidney image are segmented based on a deep learning segmentation model first, wherein the deep learning segmentation model is trained to have the ability to recognize specified targets, such as kidneys, blood cells, cell nuclei, etc.; then the capillary balls are segmented from the glomeruli; finally, various blood cells, cell nuclei and cell matrices are segmented from the capillary balls. Therefore, the deep learning-based kidney image processing method provided in the present application has high processing efficiency for the kidney image to be processed, can accurately locate the glomerular contours in the kidney, and accurately locate the capillary balls, blood cells, cell nuclei and cell matrices in the glomeruli from the glomeruli, so as to realize the acquisition of various index information in the glomeruli. BRIEF DESCRIPTION OF DRAWINGS

[0022] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in connection with the following drawings:

[0023] Figure 1 An exemplary flow chart of a deep learning-based kidney picture processing method provided by an embodiment of the present application;

[0024] Figure 2 Another exemplary flow chart of a deep learning-based kidney picture processing method provided by an embodiment of the present application;

[0025] Figure 3 A pre-processing flow chart of a kidney HE slice picture provided by an embodiment of the present application;

[0026] Figure 4 A comparison chart of a kidney HE slice picture before and after processing provided by an embodiment of the present application; wherein, Fig. (a) is an untreated kidney HE slice picture; Fig. (b) is a kidney slice picture with the outer contour of the kidney removed;

[0027] Figure 5 A U-Net deep learning segmentation model provided by an embodiment of the present application; 2

[0028] Figure 6 A data set chart of kidney sub-block pictures provided by an embodiment of the present application;

[0029] Figure 7 A comparison chart of locating glomerular contour from a kidney slice picture with the outer contour of the kidney removed provided by an embodiment of the present application; wherein, Fig. (a) is a kidney slice picture with the outer contour of the kidney removed; Fig. (b) is a kidney picture with glomerular contour;

[0030] Figure 8 An exemplary structural chart of a glomerular slice picture provided by an embodiment of the present application;

[0031] Figure 9 An exemplary flow chart of locating capillary ball contour from a glomerular slice picture provided by an embodiment of the present application;

[0032] Figure 10 A comparison chart of locating capillary ball contour from a glomerular slice picture provided by an embodiment of the present application; wherein, Fig. (a) is an untreated glomerular slice picture; Fig. (b) is a glomerular picture with capillary ball contour;

[0033] Figure 11 An exemplary flow chart of locating blood cell contour from a glomerular slice picture provided by an embodiment of the present application;

[0034] Figure 12 ​A glomerulus slice map with blood cell contours provided by an embodiment of the present application;

[0035] Figure 13 An exemplary flow chart for locating cell nucleus contours from a glomerulus slice map provided by an embodiment of the present application;

[0036] Figure 14 A glomerulus map with cell nucleus contours provided by an embodiment of the present application;

[0037] Figure 15 An exemplary flow chart for locating cell matrix contours from a glomerulus slice map provided by an embodiment of the present application;

[0038] Figure 16 A flow chart for establishing a standard control space of kidney glomerulus provided by an embodiment of the present application;

[0039] Figure 17 A diagnostic flow chart for a kidney slice map to be processed provided by an embodiment of the present application;

[0040] Figure 18 A multi-dimensional space visualization display of kidney glomerulus provided by an embodiment of the present application; wherein, Fig. (a) is a standard control space of kidney glomerulus, Fig. (b) is a space distribution point set of all glomerulus in the kidney slice map to be processed, Fig. (c) is a space distribution point set of normal glomerulus in the standard control space of kidney glomerulus, and Fig. (d) is a space distribution point set of abnormal glomerulus in the kidney slice map to be processed;

[0041] Figure 19 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, but not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0043] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0044] Please refer to Figures 1-2 , which shows an exemplary flow chart of a kidney picture processing method based on deep learning provided by an embodiment of the present application.

[0045] As Figures 1-2 shown, in the present embodiment, the kidney picture processing method based on deep learning provided by the present application comprises:

[0046] S110: preprocessing the kidney image to be processed to obtain a processed kidney image;

[0047] S120: processing the processed kidney image by using a deep learning segmentation model to obtain a kidney image with glomerular contours; wherein the kidney image with glomerular contours includes a plurality of glomeruli;

[0048] S130: locating the capillary ball in each glomerulus from the kidney image with glomerular contours to obtain a kidney image with capillary ball contours;

[0049] S140: locating the blood cells, cell nuclei and cell matrix inside the capillary ball from the kidney image with capillary ball contours, respectively.

[0050] Specifically, in this application, the deep learning segmentation model is first used to segment the glomerulus in the processed kidney image, wherein the deep learning segmentation model is trained to have the ability to identify the specified target, such as identifying the kidney, glomerulus, blood cells, cell nucleus, etc. Then the capillary ball is segmented from the glomerulus. Finally, various blood cells, cell nuclei and cell matrix are segmented from the capillary ball. Therefore, the kidney image processing method based on deep learning in this application has high processing efficiency for the kidney image to be processed, can accurately locate the glomerulus in the kidney, and accurately locate the capillary ball, blood cells, cell nuclei and cell matrix in the glomerulus from the glomerulus, so as to realize the acquisition of various index information in the glomerulus.

[0051] In some embodiments, as shown in Figures 3-4 The step S110 of preprocessing the kidney image to be processed to obtain a processed kidney image includes the following steps:

[0052] S111: hematoxylin-eosin staining of the kidney section, scanning to obtain a kidney HE section image, which is the kidney image to be processed;

[0053] S112: first converting the kidney HE section image into a kidney image in RGBA format, then separating the kidney image in RGBA format into R, G, B and A channels, then merging the B, G and R channel pictures and converting them into YUV format pictures, and finally filtering the YUV and A channels by threshold to obtain a kidney contour image;

[0054] S113: mapping the coordinates of the kidney in the kidney contour image to the corresponding level of the kidney HE section image to obtain a processed kidney image.

[0055] Specifically, preprocessing the kidney image to be processed can accurately locate the kidney contour. The specific kidney contour positioning flow chart is shown in Figure 3 .

[0056] In step S111, hematoxylin-eosin (HE) staining is first performed on the kidney section, so that the blood cells in the glomerulus of the kidney section are dyed red and the cell nucleus is dyed purple, thereby facilitating subsequent image segmentation processing of the kidney section. In the digital acquisition stage, a scanner is used to scan the HE-stained kidney cross-section and longitudinal section samples into a kidney HE section with a large resolution (e.g., 30000x40000), and a kidney HE section image with high clarity is obtained through scanning.

[0057] In step S112, the kidney HE section image obtained in step S111 is processed to remove the areas unrelated to the kidney, and the outlines of the left and right kidneys are located. The specific outline locating method includes: first reading the kidney HE section image and converting it into a kidney image in RGBA format, then separating the kidney image in RGBA format into R (red), G (green), B (blue), and A (transparency) channels, then merging the B, G, and R channels and converting them into a YUV format picture, and finally performing color threshold filtering on the Y (brightness), U (color), V (saturation), and A (total) channels to locate the areas unrelated to the kidney, thereby obtaining a kidney outline image. The color threshold filtering is divided into two parts running in parallel, which are the YUV format picture and the A channel picture.

[0058] The color threshold filtering is a kidney target locating method based on a multi-scale color space, and the calculation formula is as follows:

[0059]

[0060] wherein, Countor kidney represents the pixel value of the kidney outline image, Filter represents filtering, and if represents if.

[0061] That is, the pixel points with Y, U, and V pixel values in the YUV channel within the threshold interval [min, max] and the A pixel value in the A channel greater than the preset transparency threshold (e.g., 10) are retained, otherwise, they are uniformly set to 0, thereby achieving filtering of the image color features. The pixel threshold interval of the Y channel is [25, 180], the pixel threshold interval of the U channel is [100, 200], the pixel threshold interval of the V channel is [18, 230], and the transparency threshold 10 is only illustrative, and a person skilled in the art can set it to other values according to actual needs.

[0062] In step S113, a kidney outline mapping method is used to map the coordinates of the kidney in the kidney outline image to the corresponding level of the kidney HE section image, thereby obtaining a kidney section image with the outer contour of the kidney removed, i.e., a processed kidney image, as shown in Figure 4 wherein Figure 4 (a) is an unprocessed kidney HE section image;Figure 4 (b) A slice of the kidney with the outer contour removed. Kidney contour mapping methods refer to smoothly transitioning the kidney contour from a lower level to a higher level, preventing jagged edges. Generally, kidney contour mapping methods employ linear interpolation algorithms. This algorithm uses a straight line connecting two known quantities to determine the value of an unknown quantity between them, such as unilinear interpolation or bilinear interpolation, as detailed below:

[0063] Formula for single linear interpolation:

[0064]

[0065] Here, (x0, y0) and (x1, y1) are known coordinates. To obtain the value of y of a certain position x on the line within the interval [x0, x1], since x is known, the value of y can be obtained using the above formula. The process of finding x given y is the same as above, except that x and y are interchanged.

[0066] Bilinear interpolation formula:

[0067]

[0068] Bilinear interpolation is a cubic one-dimensional linear interpolation that uses four nearest neighbor points f(0,0), f(0,1), f(1,0), and f(1,1) to estimate the gray level of a given point f(x,y).

[0069] In some implementations, in step S120, the deep learning segmentation model includes U-Net, U-Net++, and U... 2 -Net.

[0070] Specifically, the deep learning segmentation model uses the U-Net series of models, such as: U-Net, U-Net++, U... 2 -Net and other neural network models with relatively deep network layers. Among them, such as Figure 5 As shown, U 2 -Net employs multi-level deep feature integration and multi-scale feature extraction, enabling the model to better focus on overall and local information of the slice, thereby improving the accuracy of glomerular target detection. U 2 The -Net model consists of 6 encoding modules and 5 decoding modules. The encoding modules are mainly responsible for downsampling encoding to obtain rich contextual information and classify cells and background. The decoders are mainly responsible for upsampling decoding to achieve accurate cell localization.

[0071] In some implementations, such as Figure 5 , Figure 6 and Figure 7As shown, in step S120, the processed kidney image is processed by using the deep learning segmentation model to obtain a kidney image with glomerular contours, including:

[0072] S121: The processed kidney image is cut to obtain multiple kidney patch images, and the coordinates of each kidney patch image are recorded; wherein the step length of cutting is a preset multiple of the size of the kidney patch image;

[0073] S122: The multiple kidney patch images are input into the deep learning segmentation model for prediction to obtain multiple binary images; wherein the coordinates of the multiple binary images are consistent with the coordinates of the corresponding kidney patch images;

[0074] S123: The glomeruli in the multiple binary images are segmented and connected, and the multiple binary images are spliced into an image with a size consistent with that of the processed kidney image according to the recorded coordinates to obtain a kidney image with glomerular contours.

[0075] Specifically, the deep learning segmentation model is used to process the processed kidney image, which can accurately locate the glomerular contours in the kidney image.

[0076] In the training stage of the deep learning segmentation model, the size of the processed kidney image is very large, and its resolution may be 30000x40000, which cannot be directly sent into the deep learning segmentation model for training and prediction. Therefore, the processed kidney image needs to be cut into multiple kidney patch images with a certain size (such as Figure 6 As shown), the step length of cutting is a preset multiple of the size of the kidney patch image, for example, the step length of cutting is 1 / 4 or 1 / 5 of the size of the kidney patch image, etc., which can ensure the integrity of the glomerular contours obtained finally.

[0077] It should be noted that in the training stage of the deep learning segmentation model, the annotation data comes from the annotation of pathologists, which can ensure the accuracy of the data and help improve the training indicators of the deep learning segmentation model. In the training stage of the deep learning segmentation model, in order to fully train the deep learning segmentation model, it is necessary to ensure that each kidney patch image contains glomeruli, i.e., the glomerular contours exist in the annotation results.

[0078] In the embodiments of the present application, the coordinates of each kidney sub-block picture need to be recorded, such as the coordinates of the upper left corner of each kidney sub-block picture, so as to facilitate subsequent splicing of the pictures predicted by the deep learning segmentation model into a high-resolution picture with the same size as the processed kidney picture (or kidney HE slice picture) according to the recorded coordinates. It can be understood that those skilled in the art can set the coordinate positions of each kidney sub-block picture according to actual needs, such as the upper right corner, the lower left corner, the lower right corner, etc.

[0079] In step S122, the plurality of kidney sub-block pictures are input into the deep learning segmentation model for prediction, and a plurality of O, 255 pixel value binary pictures are obtained correspondingly; the coordinates of the plurality of binary pictures are retained with the coordinates of the corresponding kidney sub-block pictures, so as to facilitate subsequent splicing according to the coordinates.

[0080] In step S123, the watershed algorithm or the concave point detection algorithm is used to perform adhesion segmentation on the glomeruli in the plurality of binary pictures. The glomerular adhesion segmentation is a segmentation processing of a plurality of glomerular contour images from a morphological perspective, so as to ensure that the minimum analysis unit is the glomerulus. Then, the plurality of predicted binary pictures are spliced into a high-resolution picture with the same size as the processed kidney picture (or kidney HE slice picture) according to the recorded coordinates, and a kidney picture with glomerular contours is obtained, such as shown in FIG. 2. Figure 7 Figure 7 (a) is a kidney slice picture without kidney outer contour; Figure 7 (b) is a kidney picture with glomerular contours.

[0081] The watershed algorithm is a morphological segmentation method based on topological theory. In general morphological segmentation-based methods, each gray level of a picture is considered to correspond to an elevation contour. Thus, each local minimum value of the picture has an influence area, and the boundary of these influence areas is called a "watershed". The watershed algorithm includes the following steps:

[0082] 1) Classify all pixels in the gradient image according to the gray value, and set a geodesic distance threshold;

[0083] 2) Find the pixel point with the minimum gray value (h min , the default label is the lowest gray value point), and let the geodesic distance threshold start to grow from the minimum value (h min +1,..., h max ), and calculate h min ​If the geodesic distance to a pixel is less than a set geodesic distance threshold, the pixel is submerged; otherwise, a dam is set on the pixel to classify the neighboring pixels.

[0084] Cell images generally exhibit concave features. This application can use a concave point detection algorithm to segment glomerular adhesions. The basic principle of this algorithm is: first, search for all local concave points on the cell edge; then, classify the local concave points according to the category of the concave area. Usually, one concave area corresponds to one local concave point class; finally, determine the middle point from the local concave point class as the main concave point of this concave area.

[0085] In some implementations, such as Figures 9-10 As shown, in step S130, locating the capillary glomeruli within each glomerulus from the kidney map with glomerular outlines to obtain a kidney map with capillary glomerular outlines includes:

[0086] S131: Obtain each glomerular slice image from the kidney image with glomerular outlines; convert each glomerular slice image into an RGB format glomerular image, and perform color threshold filtering on the RGB format glomerular image for at least two channels to obtain an initial mask binary image;

[0087] S132: Determine the externally connected region within the glomerulus, remove the externally connected region from the initial binary mask map to obtain a capillary glomerulus mask binary map; wherein, the externally connected region within the glomerulus refers to the white cavity inside the glomerulus connected to the Bowman's capsule cavity;

[0088] S133: Repeat the color threshold filtering and external connectivity region determination of the glomerulus in sequence, and then perform a bitwise XOR operation on the capillary glomerulus mask binary image and the glomerular slice image to obtain a glomerular image with capillary glomerulus outline.

[0089] S134: Map the coordinates of the capillary glomeruli in the glomerular image with capillary glomeruli outline to the corresponding level of the HE slice image of the kidney to obtain a kidney image with capillary glomeruli outline.

[0090] Specifically, the structure of a glomerular slice is as follows: Figure 8 As shown, the glomerulus includes the cellular matrix, nucleus, blood cells, Bowman's capsule, and non-glomerular tissues located within the glomerular outline. In this application, as... Figure 9 As shown, areas within the glomerulus unrelated to the capillary glomerulus (non-glomerular tissues, Bowman's capsule, etc.) are removed to complete the localization of the capillary glomerulus within the glomerulus.

[0091] In step S131, each glomerular slice image is obtained from the kidney image with glomerular profile obtained in step S120, and each glomerular slice image is converted into a glomerular picture in RGB format. Color threshold filtering is performed on the glomerular picture in RGB format in at least two channels. Since HE staining is mainly concentrated in purple and red, i.e., blood cells are dyed red and cell nuclei are dyed purple. Therefore, in the present application, the R channel and the B channel are preferentially selected to perform color threshold filtering on the glomerular picture in RGB format to locate the capillary ball in the glomerulus, and an initial mask binary image of the capillary ball is obtained. Of course, those skilled in the art can also select other channels to perform color threshold filtering according to actual needs, such as the R channel and the G channel, or the R channel, the G channel and the B channel, etc.

[0092] In the formula, color threshold filtering is performed on the glomerular picture in RGB format in the R channel and the B channel, and the filtering formula is as follows:

[0093]

[0094] In the formula, Mask capillary_tuft represents the pixel value of the initial mask binary image of the capillary ball. If the pixel value of the R channel of the glomerular picture in RGB format is greater than R 阈值 , and the pixel value of the B channel is greater than B 阈值 , the pixel point is retained and set to 1; otherwise, the pixel point is uniformly set to 0. In the formula, R 阈值 and B 阈值 may be set arbitrarily according to actual needs, such as R 阈值 = 10 and B 阈值 = 65.

[0095] In step S132, the outer connected region of the glomerulus refers to the white cavity inside the glomerulus which is highly connected with the Bowman capsule cavity. This part of the region is not included in the capillary ball, and the outer connected region determination mainly uses pixel-level erosion operation to separate the Bowman capsule cavity and the capillary ball.

[0096] In step S133, steps S131 and S132 are repeatedly executed in sequence and cross. The number of cross-repeated execution can be 2, 3, 5, etc. After the R channel and the B channel are filtered and then superimposed in the R channel and the B channel, the outer connected region is removed. After color threshold filtering and outer connected region determination, a capillary ball mask binary image is obtained. The capillary ball mask binary image is subjected to a bit XOR pixel-level operation with the glomerular slice image, and a glomerular image with a capillary ball profile is obtained, as shown in Figure 10 Figure 10 (a) is a glomerular slice image; Figure 10 (b) is a glomerular image with a capillary ball profile. In the formula, the bit XOR pixel-level operation formula is as follows: ​

[0097]

[0098] wherein, Image capillary_tuft represents the pixel value of the glomerulus profile image, Compose represents a combination processing function, Image kidney represents the pixel value of the kidney image with glomerulus profile.

[0099] In step S124, the profile coordinates of the capillary glomerulus in the glomerulus image with capillary glomerulus profile are mapped to the corresponding level of the kidney HE slice image (or the kidney image with glomerulus profile obtained in step S120), the capillary glomerulus profile in the glomerulus of the kidney HE slice image is located, and a kidney image with capillary glomerulus profile is obtained. The specific profile coordinate mapping method is as described above, and single linear difference method, bilinear difference method, etc. can be used.

[0100] In some embodiments, as shown in Figures 11-12 the method for locating blood cells inside the capillary glomerulus from the kidney image with capillary glomerulus profile in step S140 includes:

[0101] S141: performing blur judgment on the kidney image with glomerulus profile, removing the glomerulus that does not meet the condition, and obtaining a kidney image containing qualified glomerulus;

[0102] S142: obtaining each glomerulus slice image from the kidney image containing qualified glomerulus, and adjusting the size of the glomerulus slice image;

[0103] S143: locating the capillary glomerulus in the glomerulus slice image, and obtaining a glomerulus image with capillary glomerulus profile;

[0104] S144: inputting the glomerulus image with capillary glomerulus profile into a deep learning segmentation model for processing, and obtaining a glomerulus image with blood cell profile;

[0105] S145: mapping the coordinates of the blood cells in the glomerulus image with blood cell profile to the corresponding level of the kidney HE slice image, and obtaining a kidney image with blood cell profile in the glomerulus.

[0106] Specifically, processing the kidney image with capillary glomerulus profile can accurately locate the blood cells in the capillary glomerulus in the glomerulus.

[0107] In step S141, glomerular unsharpness determination is performed on the kidney image with glomerular contours obtained in step S120 to remove the glomeruli that do not meet the conditions, and a kidney image containing qualified glomeruli is obtained. The glomerular unsharpness determination is a key link to ensure the positioning of the intraglomerular components. The glomerular unsharpness determination is to evaluate the blurring degree of the unit glomerulus at the pixel level. The glomerular unsharpness determination method includes at least one of the following: Gaussian blurring method, box blurring method, Kawase blurring method, double blurring method, Brenner detection method, Tenengrad gradient function, Laplacian blurring method, and HE staining analysis method.

[0108] For example, the Laplacian blurring method has the following calculation formula:

[0109] Gray kidnev = Image kidney > 60

[0110] Unsharpness kidney = Laplacian(Gray kidney ) = ∑ x，y | Gray kidney |

[0111]

[0112] wherein Gray kidney represents the gray value of the kidney image with glomerular contours, Image kidney represents the pixel value of the kidney image with glomerular contours, Unsharpness kianey represents the unsharpness of the kidney image with glomerular contours, Laplacian represents the Laplacian operator, and ∑ x，y | Gray kidney | represents the sum of the absolute values of the gray values of all glomeruli in the kidney image with glomerular contours. Unsharpness kidney represents the unsharpness of the glomeruli in the kidney image with glomerular contours. If the value of Unsharpness kidney is greater than a preset unsharpness threshold (for example, 7), the value is retained, otherwise, the value is filtered out.

[0113] It should be noted that the unsharpness threshold is given by a pathologist through a large number of reading results of glomerulonephritis sections, to ensure the accuracy of the unsharpness screening.

[0114] For example, the HE staining analysis method is to make an estimate according to the proportion of the nucleus and the blood cell. val_H is the HE staining red cell staining, corresponding to the blood cell, which is represented as the R channel value of the glomerular picture at the pixel level, and val_E is the HE staining purple cell staining, corresponding to the nucleus, which is represented as the sum of the B channel value and the R channel value of the glomerular picture at the pixel level. The specific calculation formula is as follows:

[0115]

[0116] wherein val_E represents the pixel value of the nucleus, and val_H represents the pixel value of the blood cell; if the ratio of the nucleus and the blood cell is greater than 9.7, the pixel point is retained, otherwise, the pixel point is removed. The threshold value 9.7 can also be set to other values according to actual needs.

[0117] As shown in Figure 11 , in step S142, each glomerular slice picture is obtained from the kidney picture containing qualified glomeruli, and the size of the glomerular slice picture is adjusted to the size specified by the deep learning segmentation model. Since the sizes of the glomeruli are different, in order to save the calculation cost and improve the calculation efficiency, the contour set of the blood cells in the glomeruli is divided into two part sets according to the size, i.e. 1024 and 512, and is respectively pixel-padded and scaled to the specified size (1024*1024, 512*512). The pixel padding and scaling principle is to pad the short side to the same size as the long side by 0 pixels and then scale it to the same size. The specific calculation method is as follows:

[0118]

[0119] wherein Resized represents the scaling operation, and All kidney represent the two size glomerular picture sets.

[0120] In step S143, the capillary ball positioning is performed on each glomerular slice picture to obtain a glomerular picture with a capillary ball contour, wherein the positioning method of the capillary ball is as described in step S130, and the present application will not be repeated here.

[0121] In step S144, the glomerular picture with the capillary ball contour is input into the deep learning segmentation model for processing according to the adjusted size, to obtain a glomerular picture with a blood cell contour as shown in Figure 12 ; wherein the black circled part is the blood cell. The deep learning segmentation model is as described above, and the deep learning segmentation model is trained to have the ability to identify blood cells, and the annotation information is also derived from the pathological expert annotation. The blood cell contour calculation process supports concurrency, and the concurrency entries depend on the number of size sets.

[0122] Step S145: Map the coordinates of blood cells in the glomerular image with blood cell outlines to the corresponding level of the kidney HE slice image (or the kidney image with glomerular outlines obtained in step S120) to obtain a kidney image with blood cell outlines within the glomerulus. The specific outline coordinate mapping method is as described above, and can be a single linear difference method, bilinear difference method, etc.

[0123] In some implementations, such as Figures 13-14 As shown, in step S140, the method for locating the cell nuclei inside the capillary glomerulus from the kidney image with the outline of the capillary glomerulus includes:

[0124] S151: Obtain a glomerular section image from the kidney image with glomerular outline, and perform hematoxylin-eosin enhancement staining on the glomerular section image to obtain an enhanced glomerular section image;

[0125] S152: The enhanced glomerular slice image is subjected to capillary glomerulus localization to obtain a glomerular image with capillary glomerulus outline;

[0126] S153: Input the glomerular image with capillary glomerulus outline into a deep learning segmentation model for processing to obtain a glomerular image with cell nucleus outline; progressively traverse each cell nucleus in the glomerular image with cell nucleus outline, segment the adhered cell nuclei to obtain a glomerular image with independent cell nucleus outline;

[0127] S154: Map the coordinates of the cell nuclei in the glomerular image with independent cell nucleus outlines to the corresponding level of the kidney HE slice image to obtain a kidney image with the cell nucleus outlines within the glomerulus.

[0128] Specifically, processing images of kidneys with capillary glomeruli outlines can precisely locate cell nuclei within the capillary glomeruli.

[0129] like Figure 13 As shown, in step S151, each glomerular slice is obtained from the kidney image with glomerular outlines obtained in step S120, and the glomerular slice is stained with hematoxylin and eosin to obtain an enhanced glomerular slice to enhance the contrast effect of cell nuclei.

[0130] In step S152, the enhanced glomerular slice obtained in step S151 is used to locate the capillary glomerulus to obtain a glomerular image with the outline of the capillary glomerulus; wherein, the method for locating the capillary glomerulus is as described in step S130, and will not be repeated in this application.

[0131] In step S153, the glomerulus graph with capillary ball contours obtained in step S152 is input into a deep learning segmentation model for processing, where the deep learning segmentation model has the ability to identify cell nuclei, and a glomerulus graph with cell nucleus contours is obtained. Each cell nucleus in the glomerulus graph with cell nucleus contours is iterated step by step, and the adhered cell nuclei are segmented to obtain a glomerulus graph with independent cell nucleus contours as shown in FIG. 13B, where the gray circled part is the cell nucleus. The adhered cell nucleus segmentation method can use a watershed algorithm or a concave point detection algorithm, as described above. The cell nucleus adhered segmentation can ensure the accuracy of subsequent cell nucleus quantity index statistics. Figure 14

[0132] In step S154, the coordinates of the cell nuclei in the glomerulus graph with independent cell nucleus contours are mapped to the corresponding level of the kidney HE slice graph (or the kidney graph with glomerulus contours obtained in step S120) to obtain a kidney graph with glomerulus cell nucleus contours. The specific contour coordinate mapping method is as described above, and can use a single linear difference method, a bilinear difference method, etc.

[0133] In some embodiments, as shown in FIG. 13A, the method for positioning the cell matrix inside the capillary ball from the kidney graph with capillary ball contours in step S140 includes: Figure 15

[0134] S161: superimpose the initial mask binary graph, the glomerulus graph with blood cell contours, and the glomerulus graph with cell nucleus contours to obtain a binary mask graph irrelevant to the cell matrix;

[0135] S162: perform a bitwise AND operation on the binary mask graph irrelevant to the cell matrix and the glomerulus slice graph to obtain a glomerulus graph with cell matrix contours;

[0136] S163: map the coordinates of the cell matrix in the glomerulus graph with cell matrix contours to the corresponding level of the kidney HE slice graph to obtain a kidney graph with glomerulus cell matrix contours.

[0137] Specifically, the components irrelevant to the cell matrix in the kidney graph with capillary ball contours obtained in step S130 are removed one by one to complete the positioning of the cell matrix in the capillary ball inside the glomerulus. The specific positioning process of the cell matrix is as follows:

[0138] In step S161, the initial mask binary graph obtained in step S131 and the glomerulus graph with blood cell contours obtained in step S143, and the glomerulus graph with cell nucleus contours obtained in step S154 are superimposed to obtain a binary mask graph irrelevant to the cell matrix.

[0139] ​​S162: Perform a bit-and-pixel set operation between the binary mask image irrelevant to the cell matrix and the kidney HE slice image (or the kidney image with glomerular contour obtained in step S120), to obtain a glomerular image with cell matrix contour. The bit-and-pixel set operation formula is as follows:

[0140]

[0141] wherein, Image matrix represents the pixel value of the cell matrix image, Mask no_matrix represents the pixel value of the binary mask image irrelevant to the cell matrix.

[0142] In step S163, the coordinates of the cell matrix in the glomerular image with cell matrix contour are mapped to the corresponding level of the kidney HE slice image (or the kidney image with glomerular contour obtained in step S120), to obtain a kidney image with cell matrix contour in the glomerulus. The specific mapping method is as described above, and can be a single linear interpolation method, a bilinear interpolation method, etc.

[0143] In some embodiments, step S140 further comprises:

[0144] S150: Determine the capillary ball area, blood cell area, cell nucleus area and number, cell matrix area, blood cell proportion, cell nucleus proportion and cell matrix proportion in each glomerulus according to the glomerular image with capillary ball contour, the glomerular image with blood cell contour, the glomerular image with cell nucleus contour and the glomerular image with cell matrix contour, respectively.

[0145] Specifically, the capillary ball area Area glo in each glomerulus is determined according to the glomerular image with capillary ball contour; the blood cell area Area blood in each glomerulus is determined according to the glomerular image with blood cell contour; the cell nucleus area Area nucleus and the number Num nucleus of cell nuclei in each glomerulus are determined according to the glomerular image with cell nucleus contour; and the cell matrix area Area cyt in each glomerulus is determined according to the glomerular image with cell matrix contour.

[0146] Then, the following indexes are calculated according to the above indexes:

[0147] Cell nucleus proportion:

[0148] Blood cell proportion:

[0149] Cell matrix proportion:

[0150] By adopting the method provided in the embodiments of the present application, the area of capillary glomerulus, the area of blood cells, the area and quantity of cell nucleus, the area of cell matrix, the proportion of blood cells, the proportion of cell nucleus and the proportion of cell matrix in the glomerulus in the kidney slice image can be quantitatively analyzed.

[0151] In some embodiments, referring to Figures 16-17 , step S150 further comprises step S160:

[0152] S161: Establishing a kidney glomerulus standard control space

[0153] Collecting a control group kidney slice image, determining index information of each glomerulus in the control group kidney slice image, the index information of the glomerulus at least including one of the following: capillary glomerulus area, cell nucleus quantity, blood cell proportion, cell nucleus proportion and cell matrix proportion;

[0154] According to the index information of each glomerulus, an initial glomerulus multi-dimensional feature space is constructed, a Gaussian distribution analysis is performed on each index of the glomerulus in the initial glomerulus multi-dimensional feature space, discrete points are removed, and then the integrity and correlation of each index in the glomerulus are analyzed, irrelevant indexes are removed, and a multi-dimensional glomerulus standard control point set is obtained;

[0155] The multi-dimensional glomerulus standard control point set is optimized to obtain a kidney glomerulus standard control space;

[0156] S162: Determining abnormal indexes in a to-be-processed kidney slice image

[0157] Obtaining a to-be-processed kidney slice image, determining index information of each glomerulus in the to-be-processed kidney slice image, mapping the index information of each glomerulus in the to-be-processed kidney slice image to the kidney glomerulus standard control space to perform glomerulus individual state analysis, and determining the overall distribution state of all glomeruli in the to-be-processed kidney slice image.

[0158] Specifically, firstly, the indicator information of all glomeruli in the kidney slices of the control group (which have no pathological features) is collected. The indicator information includes capillary glomerular area, number of nuclei, proportion of blood cells, proportion of nuclei, and proportion of cellular matrix, etc., to establish a standard control space for all glomeruli in the kidney. Then, the indicator information of each glomerulus in the kidney slice to be processed is obtained. Multiple indicators of each glomerulus in the kidney slice to be processed are quantitatively analyzed. Then, the multiple indicator information of each glomerulus in the kidney slice to be processed is mapped to the standard control space of kidney glomeruli. The individual status of each glomerulus in the kidney slice to be processed is output one by one. Finally, the overall status distribution of all glomeruli in the kidney slice to be processed (i.e., the ratio of the number of glomeruli in the kidney slice to the number of glomeruli inside and outside the standard control space of kidney glomeruli) is integrated to complete the intelligent diagnosis of glomerulonephritis.

[0159] like Figure 16 The diagram illustrates the process of establishing a standard control space for kidney glomeruli. Kidney slices from a control group (healthy kidneys without pathological features) are collected. The parameters of all glomeruli within these control group slices are determined, including five parameters for each glomerulus: capillary glomerular area, number of nuclei, percentage of blood cells, percentage of nuclei, and percentage of cytosol. Based on these five parameters, an initial multidimensional feature space for glomeruli is constructed. Gaussian distribution analysis is then performed on each parameter for each glomerulus to remove discrete points. The discrete point set is filtered with 85% as the standard retention size (multidimensional Gaussian distribution filtering). Those skilled in the art can set other discrete point set filtering ratios, such as 90%, according to actual needs. Finally, the completeness and correlation of each parameter within the glomerulus are analyzed, and irrelevant parameters are removed to obtain a multidimensional standard control set for glomeruli (multi-parameter stability filtering). Finally, the multidimensional glomerular standard control point set was optimized to represent the kidney glomerular standard control space with the fewest possible multidimensional spatial point sets. The selection and comparison of indicators need to be cross-checked three times to ensure the integrity of feature information. The number of indicator selections and comparisons can also be set to other values ​​according to actual needs, such as 4, 5, etc.

[0160] like Figure 17As shown, the index information of each glomerulus in the to-be-processed kidney slice image is obtained by performing steps S110-S150 on the to-be-processed kidney slice image, the index information of each glomerulus in the to-be-processed kidney slice image is mapped to the kidney glomerulus standard control space one by one for glomerular individual state analysis, and the glomerular individual state is expressed as the external distribution in the kidney glomerulus standard control space; and the overall distribution state of all glomeruli in the to-be-processed kidney slice image is determined, and the overall state distribution of the glomeruli is expressed as the proportion of the number of glomeruli in the to-be-processed kidney slice image in the kidney glomerulus standard control space.

[0161] As shown in Figure 18 , Figure 18 The kidney glomerulus multi-dimensional space visualization display diagram provided by the embodiment of the present application; wherein, Figure 18 (a) a kidney glomerulus standard control space, Figure 18 (b) a spatial distribution point set of all glomeruli in the to-be-processed kidney slice image; Figure 18 (c) a spatial distribution point set of normal glomeruli in the kidney glomerulus standard control space; by comparing Figure 18 (b) and Figure 18 (c), the spatial distribution point set of abnormal glomeruli in the to-be-processed kidney slice image shown in Figure 19 (d) can be obtained, which can be used for auxiliary diagnosis of glomerular nephritis.

[0162] By establishing the kidney glomerulus standard control space, the embodiment of the present application can determine the individual state of the glomeruli and the overall state distribution of all glomeruli in the to-be-processed kidney slice image, can realize automatic diagnosis of glomerular nephritis lesions, can accurately display normal and lesion glomeruli, can display the degree of glomerular nephritis lesions in the to-be-diagnosed kidney slice in a multi-dimensional visualization manner, and can provide a basis for the treatment of clinical glomerular nephritis. Compared with manual reading, the kidney picture processing method provided by the present application can significantly improve the picture processing efficiency, has high intelligence, and can ensure the uniformity and standardization of reading quality.

[0163] It should be noted that although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired results. On the contrary, the steps depicted in the flowchart can change the order of execution.

[0164] In a second aspect, the embodiment of the present application provides a kidney picture processing system based on deep learning, comprising:

[0165] The processing module is configured to pre-process the to-be-processed kidney picture to obtain a processed kidney picture.

[0166] a deep learning module configured to process the processed kidney image by using a deep learning segmentation model to obtain a kidney image with glomerular contours, wherein the kidney image with glomerular contours comprises a plurality of glomeruli;

[0167] a capillary positioning module configured to locate capillary balls in each glomerulus from the kidney image with glomerular contours to obtain a kidney image with capillary ball contours;

[0168] a blood cell positioning module configured to locate blood cells inside the capillary balls from the kidney image with capillary ball contours;

[0169] a nucleus positioning module configured to locate nuclei inside the capillary balls from the kidney image with capillary ball contours;

[0170] a cell matrix positioning module configured to locate cell matrices inside the capillary balls from the kidney image with capillary ball contours.

[0171] In some embodiments, the processing module comprises:

[0172] an HE staining unit configured to perform hematoxylin-eosin staining on a kidney section, and scan to obtain a kidney HE section image, which is the kidney image to be processed;

[0173] a threshold filtering unit configured to first convert the kidney HE section image into a kidney image in RGBA format, then separate the kidney image in RGBA format into four channels of R, G, B and A, then combine the three channels of B, G and R and convert into a YUV format picture, and finally perform threshold filtering on the four channels of YUV and A to obtain a kidney contour image;

[0174] a kidney mapping unit configured to map the coordinates of the kidney in the kidney contour image to a corresponding level of the kidney HE section image to obtain a processed kidney image.

[0175] In some embodiments, the deep learning module comprises:

[0176] a cutting unit configured to cut the processed kidney image to obtain a plurality of kidney patch pictures, and record the coordinates of each kidney patch picture, wherein the step length of the cutting is a preset multiple of the size of the kidney patch picture, and each kidney patch picture comprises at least one glomerulus;

[0177] a glomerular segmentation unit configured to input the plurality of kidney patch pictures into the deep learning segmentation model for prediction to obtain a plurality of binary images, wherein the coordinates of the plurality of binary images are consistent with the coordinates of the corresponding kidney patch pictures;

[0178] A glomerulus splicing unit is configured to perform adhesion segmentation on multiple glomerulus binary images, and splices the multiple glomerulus binary images according to recorded coordinates to obtain a picture consistent with the size of the processed kidney image, thereby obtaining a kidney image with a glomerulus contour.

[0179] In some embodiments, the capillary positioning module comprises:

[0180] A threshold filtering unit is configured to obtain each glomerulus slice image from the kidney image with a glomerulus contour, convert each glomerulus slice image into an RGB format glomerulus picture, and perform color threshold filtering on at least two channels of the RGB format glomerulus picture to obtain an initial mask binary image.

[0181] A region removal unit is configured to determine an outer connected region in the glomerulus, remove the outer connected region from the initial mask binary image to obtain a capillary ball mask binary image, wherein the outer connected region in the glomerulus refers to a white cavity inside the glomerulus connected to the Bowman capsule cavity.

[0182] A capillary ball determination unit is configured to repeatedly perform color threshold filtering and outer connected region determination on the glomerulus, and then perform a bit XOR operation on the capillary ball mask binary image and the glomerulus slice image to obtain a glomerulus image with a capillary ball contour.

[0183] A capillary ball mapping unit is configured to map the coordinates of the capillary ball in the glomerulus image with a capillary ball contour to the corresponding level of the kidney HE slice image to obtain a kidney image with a capillary ball contour.

[0184] In some embodiments, the blood cell positioning module comprises:

[0185] A blur determination unit is configured to perform blur determination on the kidney image with a glomerulus contour, remove the glomerulus that does not meet the condition, and obtain a kidney image containing qualified glomerulus.

[0186] A size adjustment unit is configured to obtain each glomerulus slice image from the kidney image containing qualified glomerulus, and adjust the size of the glomerulus slice image.

[0187] A capillary ball positioning unit is configured to perform capillary ball positioning on the glomerulus slice image to obtain a glomerulus image with a capillary ball contour.

[0188] A blood cell segmentation unit is configured to input the glomerulus image with a capillary ball contour into a deep learning segmentation model for processing to obtain a glomerulus image with a blood cell contour.

[0189] a blood cell mapping unit, configured to map coordinates of blood cells in the glomerulus graph with blood cell contours to a corresponding level of the kidney HE slice graph, to obtain a kidney graph with blood cell contours in glomeruli.

[0190] In some embodiments, the nucleus positioning module comprises:

[0191] an HE staining unit, configured to acquire a glomerulus slice graph from the kidney graph with glomerulus contours, perform hematoxylin-eosin (HE) staining on the glomerulus slice graph, and obtain an enhanced glomerulus slice graph;

[0192] a capillary ball positioning unit, configured to perform capillary ball positioning on the enhanced glomerulus slice graph, to obtain a glomerulus graph with capillary ball contours;

[0193] a nucleus segmentation unit, configured to input the glomerulus graph with capillary ball contours into a deep learning segmentation model for processing, to obtain a glomerulus graph with nucleus contours; and perform segmentation on adherent nuclei in the glomerulus graph with nucleus contours, to obtain a glomerulus graph with independent nucleus contours;

[0194] a nucleus mapping unit, configured to map coordinates of nuclei in the glomerulus graph with independent nucleus contours to a corresponding level of the kidney HE slice graph, to obtain a kidney graph with nucleus contours in glomeruli.

[0195] In some embodiments, the cell matrix positioning module comprises:

[0196] a superimposition unit, configured to superimpose the initial mask binary graph, the glomerulus graph with blood cell contours, and the glomerulus graph with nucleus contours, to obtain a binary mask graph irrelevant to cell matrix;

[0197] a cell matrix determination unit, configured to perform a bitwise AND operation on the binary mask graph irrelevant to cell matrix and the glomerulus slice graph, to obtain a glomerulus graph with cell matrix contours;

[0198] a cell matrix mapping unit, configured to map coordinates of cell matrix in the glomerulus graph with cell matrix contours to a corresponding level of the kidney HE slice graph, to obtain a kidney graph with cell matrix contours in glomeruli.

[0199] In some embodiments, the system further comprises:

[0200] The index determination module is configured to determine the capillary ball area, the blood cell area, the number of cell nuclei, the area of cell matrix, the blood cell proportion, the cell nucleus proportion and the cell matrix proportion in each glomerulus according to the glomerulus image with the capillary ball contour, the glomerulus image with the blood cell contour, the glomerulus image with the cell nucleus contour and the glomerulus image with the cell matrix contour respectively.

[0201] In some embodiments, the system further comprises:

[0202] The standard control space establishment module is configured to:

[0203] The control group kidney slice images are collected, and index information of each glomerulus in the control group kidney slice images is determined, the index information of the glomerulus including at least one of the following: capillary ball area, number of cell nuclei, blood cell proportion, cell nucleus proportion and cell matrix proportion;

[0204] An initial glomerulus multi-dimensional feature space is constructed according to the index information of each glomerulus, Gaussian distribution analysis is performed on each index of the glomerulus in the initial glomerulus multi-dimensional feature space, discrete points are removed, and the integrity and correlation of each index in the glomerulus are analyzed to remove irrelevant indexes, thereby obtaining a multi-dimensional glomerulus standard control point set;

[0205] The multi-dimensional glomerulus standard control point set is optimized to obtain a kidney glomerulus standard control space;

[0206] The abnormal index determination module is configured to:

[0207] The kidney slice images to be processed are obtained, index information of each glomerulus in the kidney slice images to be processed is determined, the index information of each glomerulus in the kidney slice images to be processed is mapped to the kidney glomerulus standard control space for glomerulus individual state analysis, and the overall distribution state of all glomeruli in the kidney slice images to be processed is determined.

[0208] Figure 19 A structural schematic diagram of an electronic device is shown.

[0209] As Figures 1-2As shown, as another aspect, the present application also provides an electronic device 200 including one or more central processing units (CPUs) 201 that can perform various appropriate actions and processes in accordance with programs stored in a read only memory (ROM) 202 or programs loaded from a storage section 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for system operation are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0210] Connected to the I / O interface 205 are an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as necessary. A removable media 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 210 as necessary, so that a computer program read therefrom is installed into the storage section 208 as necessary.

[0211] In particular, the processes described above with reference to ​ the present disclosure can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine readable medium, the computer program containing program code for executing a deep learning based kidney picture processing method. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 209, and / or installed from the removable media 211.

[0212] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0213] In another aspect, this application also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into the device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the deep learning-based kidney image processing method described in this application.

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0215] The units or modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described units or modules can also be arranged in a processor, for example, each of the units can be a software program arranged in a computer or a mobile smart device, or can be a separately configured hardware device. In some cases, the names of these units or modules do not constitute a limitation on the units or modules themselves.

[0216] The above description is merely preferred embodiments of the present application and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the inventive concept. For example, the above features are replaced with each other to form a technical solution with similar functions disclosed in the present application (but not limited to).

Claims

1. A deep learning-based method for processing kidney images, characterized in that, include: Preprocess the kidney image to be processed to obtain the processed kidney image; The processed kidney image is processed using a deep learning segmentation model to obtain a kidney image with glomerular outlines; wherein, the kidney image with glomerular outlines includes multiple glomeruli; From the kidney map with glomerular outlines, the capillary glomeruli within each glomerulus are located to obtain a kidney map with capillary glomerular outlines. Then, from the kidney diagram with the outline of capillary glomeruli, locate the blood cells, cell nuclei, and cell matrix inside the capillary glomeruli respectively; Specifically, locating the capillary glomeruli within each glomerulus from the kidney map with glomerular outlines to obtain a kidney map with capillary glomerular outlines includes: Each glomerular slice image is obtained from the kidney image with glomerular outlines. Each glomerular slice image is converted into a glomerular image in RGB format. The RGB format glomerular image is then subjected to color threshold filtering for at least two channels to obtain an initial mask binary image. The externally connected region within the glomerulus is determined, and the initial binary mask image is modified by removing the externally connected region to obtain a capillary glomerulus binary mask image; wherein, the externally connected region within the glomerulus refers to the white cavity inside the glomerulus that is connected to the cavity of Bowman's capsule; The color threshold filtering and external connectivity region determination of the glomerulus are repeated sequentially. Then, the binary image of the capillary glomerulus mask and the glomerular slice image are XORed to obtain a glomerular image with the outline of the capillary glomerulus. The coordinates of the capillaries in the glomerular image with capillary glomerulus outline are mapped to the corresponding level of the kidney HE slice image to obtain a kidney image with capillary glomerulus outline.

2. The kidney image processing method based on deep learning according to claim 1, characterized in that, The kidney images to be processed are preprocessed to obtain processed kidney images, including: The kidney sections were stained with hematoxylin and eosin and scanned to obtain HE-images of the kidneys. The HE-images of the kidneys were images of the kidneys to be processed. First, the kidney HE slice image is converted into a kidney image in RGBA format. Then, the RGBA kidney image is separated into four channels: R, G, B, and A. Next, the three channels of B, G, and R are merged and converted into a YUV image. Finally, the four channels of YUV and A are threshold filtered to obtain the kidney contour image. The coordinates of the kidney in the kidney contour map are mapped to the corresponding level of the kidney HE slice map to obtain the processed kidney map.

3. The kidney image processing method based on deep learning according to claim 2, characterized in that, The coordinates of the kidney in the kidney contour map are mapped to the corresponding level of the kidney HE slice map using either single linear interpolation or bilinear interpolation to obtain the processed kidney map.

4. The kidney image processing method based on deep learning according to any one of claims 1-3, characterized in that, The processed kidney image is processed using a deep learning segmentation model to obtain a kidney image with glomerular contours, including: The processed kidney image is segmented to obtain multiple kidney block images, and the coordinates of each kidney block image are recorded; wherein, the segmentation step size is a preset multiple of the size of the kidney block image; Multiple kidney segmentation images are input into the deep learning segmentation model for prediction, resulting in multiple binary images; wherein the coordinates of the multiple binary images are consistent with the coordinates of the corresponding kidney segmentation images. The glomeruli in multiple binary images are segmented by adhesion, and the multiple binary images are stitched together according to the recorded coordinates to form an image with the same size as the processed kidney image, thus obtaining a kidney image with glomerular outlines.

5. The kidney image processing method based on deep learning according to claim 4, characterized in that, The watershed algorithm or the concave point detection algorithm is used to segment the glomeruli in multiple binary images to achieve adhesion.

6. The kidney image processing method based on deep learning according to claim 1, characterized in that, A method for locating blood cells within capillary glomeruli from a kidney image having a capillary glomerulus outline includes: The ambiguity of the kidney image with glomerular outlines is determined, and glomeruli that do not meet the criteria are removed to obtain a kidney image containing qualified glomeruli. From the kidney image containing qualified glomeruli, obtain a slice of each glomerulus and adjust the size of the glomerular slice; The glomerular slice image is used to locate the capillary glomerulus to obtain a glomerular image with the outline of the capillary glomerulus; The glomerular image with capillary glomerulus outline is input into a deep learning segmentation model for processing to obtain a glomerular image with blood cell outline. The coordinates of the blood cells in the glomerular image with blood cell outlines are mapped to the corresponding level of the kidney HE slice image to obtain a kidney image with blood cell outlines within the glomerulus.

7. The kidney image processing method based on deep learning according to claim 6, characterized in that, A method for locating cell nuclei within capillary glomeruli from a kidney image having a capillary glomerulus outline includes: Glomerular slices were obtained from the kidney image with glomerular outlines, and the glomerular slices were subjected to hematoxylin-eosin enhanced staining to obtain enhanced glomerular slices. The enhanced glomerular slice image was localized to the capillary glomerulus to obtain a glomerular image with the outline of the capillary glomerulus; The glomerular image with capillary glomerulus outline is input into a deep learning segmentation model for processing to obtain a glomerular image with cell nucleus outline; each cell nucleus in the glomerular image with cell nucleus outline is traversed step by step, and the adhered cell nuclei are segmented to obtain a glomerular image with independent cell nucleus outline; The coordinates of the cell nuclei in the glomerular image with independent cell nucleus outlines are mapped to the corresponding level of the kidney HE slice image to obtain a kidney image with the cell nucleus outlines within the glomerulus.

8. The kidney image processing method based on deep learning according to claim 7, characterized in that, A method for locating the intracellular matrix of capillaries from a kidney image having a capillary glomerulus outline includes: The initial binary mask image, the glomerular image with blood cell outlines, and the glomerular image with cell nucleus outlines are superimposed to obtain a binary mask image that is independent of the cell matrix. By performing a bitwise AND operation between the binary mask image, which is independent of the cell matrix, and the glomerular slice image, a glomerular image with the outline of the cell matrix is ​​obtained. The coordinates of the cellular matrix in the glomerular image with the cellular matrix outline are mapped to the corresponding level of the kidney HE slice image to obtain a kidney image with the cellular matrix outline.

9. The kidney image processing method based on deep learning according to any one of claims 1-7, characterized in that, The kidney image processing method further includes: determining the area of ​​the capillary glomerulus, the area and number of blood cells, the area of ​​the cell matrix, the proportion of blood cells, the proportion of cell nuclei, and the proportion of cell matrix in each glomerulus based on the glomerular image with the outline of the capillary glomerulus, the glomerular image with the outline of blood cells, the glomerular image with the outline of cell nuclei, and the glomerular image with the outline of cell matrix, respectively.

10. The kidney image processing method based on deep learning according to claim 9, characterized in that, The kidney image processing method further includes: Collect kidney slice images from the control group and determine the index information of each glomerulus in the kidney slice images of the control group. The index information of the glomerulus includes at least one of the following: capillary glomerular area, number of nuclei, percentage of blood cells, percentage of nuclei, and percentage of cell matrix. An initial glomerular multidimensional feature space is constructed based on the indicator information of each glomerulus. Gaussian distribution analysis is performed on each indicator of the glomerulus in the initial glomerular multidimensional feature space to remove discrete points. Then, the integrity and correlation of each indicator in the glomerulus are analyzed to remove irrelevant indicators, and a multidimensional glomerular standard control point set is obtained. The multidimensional glomerular standard control point set is optimized to obtain the kidney glomerular standard control space; Obtain a kidney slice image to be processed, determine the index information of each glomerulus in the kidney slice image to be processed, map the index information of each glomerulus in the kidney slice image to the standard control space of the kidney glomerulus to perform individual glomerular state analysis, and determine the overall distribution state of all glomeruli in the kidney slice image to be processed.

11. A kidney image processing system based on deep learning, characterized in that, The processing system is used to execute the deep learning-based kidney image processing method as described in any one of claims 1-10, the processing system comprising: The processing module is used to preprocess the kidney image to be processed, so as to obtain the processed kidney image. The deep learning module is used to process the processed kidney image using a deep learning segmentation model to obtain a kidney image with glomerular outlines; wherein the kidney image with glomerular outlines includes multiple glomeruli; A capillary localization module is used to locate the capillary spheres within each glomerulus from the kidney map with glomerular outlines, thereby obtaining a kidney map with capillary sphere outlines. A blood cell localization module is used to locate blood cells inside the capillary glomerulus from the kidney image with the outline of the capillary glomerulus; A cell nucleus localization module is used to locate cell nuclei inside the capillary glomerulus from the kidney image with the outline of the capillary glomerulus; A cell matrix localization module is used to locate the cell matrix inside the capillary glomerulus from the kidney map with the outline of the capillary glomerulus.

12. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the steps of the deep learning-based kidney image processing method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the deep learning-based kidney image processing method according to any one of claims 1-10.

Images