A kidney component segmentation method and system based on deep learning

Abstract

The present invention relates to a deep learning-based kidney component segmentation method and system, belonging to the field of medical image segmentation technology, and solves the problem of low kidney component segmentation accuracy in the prior art. The method includes: obtaining a kidney pathology image and corresponding component annotation information, segmenting the kidney pathology image, screening a target image from the segmented image, and constructing a sample set based on the target image and the corresponding component annotation information; constructing a multi-channel coded fusion image segmentation model, and training the multi-channel coded fusion image segmentation model based on the sample set to obtain a kidney component segmentation model; segmenting the kidney pathology image to be segmented, screening a target image from the segmented image, and inputting the model into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and obtaining a component segmentation map of the kidney pathology image to be segmented based on the segmentation result. High-precision kidney component segmentation is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and system for kidney component segmentation based on deep learning. Background Art

[0002] Chronic kidney disease (CKD) has been recognized as a major public health problem worldwide. A nationwide cross-sectional survey revealed a CKD prevalence of 10.8%, indicating that nearly 120 million adults in my country suffer from kidney disease. Renal pathology is the gold standard for diagnosing kidney disease and serves as a bridge between pathogenesis and clinical manifestations. The acquisition and analysis of pathomorphological information in renal biopsy specimens is crucial for assessing disease severity and prognosis, as well as for selecting treatment decisions. It is also a crucial component of basic renal research.

[0003] However, traditional pathological assessment methods are based solely on the existing knowledge of pathologists, using semi-quantitative assessments of several pathological parameters in the glomeruli, tubulointerstitium, and arterioles to reflect disease severity. As kidney disease research deepens, traditional assessments are increasingly unable to meet the needs of clinical and scientific research. The next generation of renal pathology requires a more refined and comprehensive approach to replace traditional assessments.

[0004] Renal pathology images contain several major components: glomeruli, tubules, arterioles, and renal interstitium. Each component can harbor lesions of varying types and severity, and these lesions collectively contribute to the progression of renal disease. Accurate assessment of lesions in each component is crucial for overall renal disease assessment, treatment decision-making, and prognosis. Accurate assessment requires measurement and statistical analysis of lesions in each component. This is based on precise segmentation of the boundaries of each component in renal pathology images, followed by further analysis based on the segmentation results. For example, after accurately segmenting the boundaries of glomeruli and tubules, further measurements of their size, shape, and uniformity are performed to comprehensively reflect the severity of renal disease. Boundary segmentation and component identification and classification require extensive medical knowledge and experience from renal pathologists. However, manual segmentation and classification are time-consuming and labor-intensive, often requiring several hours for a single specimen, making them impractical for routine implementation. Recent advances in deep learning algorithms, GPUs, and intelligent chips have made it possible to accurately assess large sample sizes using image recognition in computer vision.

[0005] The deep learning model architectures used in existing medical image segmentation are mostly traditional UNet and its simple variants, which have insufficient feature extraction capabilities. Therefore, the segmentation accuracy of sclerotic glomeruli, arteries, etc. is low, and the segmentation precision of kidney components is low. Summary of the Invention

[0006] In view of the above analysis, the embodiments of the present invention aim to provide a kidney component segmentation method and system based on deep learning to solve the problem of low accuracy of existing kidney component segmentation.

[0007] In one aspect, an embodiment of the present invention provides a method for segmenting kidney components based on deep learning, comprising the following steps:

[0008] Obtaining kidney pathology images and corresponding component annotation information, segmenting the kidney pathology images, screening target images from the segmented images, and constructing a sample set based on the target images and corresponding component annotation information;

[0009] Constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0010] The kidney pathology image to be segmented is cut, and a target image is selected from the cut image and input into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and a component segmentation map of the kidney pathology image to be segmented is obtained based on the segmentation result.

[0011] Based on the further improvement of the above method, the multi-channel coding fusion image segmentation model includes:

[0012] The first encoding module includes a plurality of sequentially connected downsampling units, and is used to extract multi-layer semantic features from the sample image;

[0013] The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image;

[0014] The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module;

[0015] A decoding module, comprising a plurality of upsampling units connected in sequence, configured to recover a feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map;

[0016] The output layer is used to classify components based on the final feature map to obtain a component segmentation map;

[0017] The number of downsampling units in the first encoding module is the same as the number of upsampling units in the encoding module and corresponds one to one;

[0018] A spatial channel cross attention module is connected between the downsampling unit and the corresponding upsampling unit, which is used to perform attention extraction on the semantic features extracted by the downsampling unit and pass them to the corresponding upsampling unit.

[0019] Based on the further improvement of the above method, the pixel value of the interpolation point (x, y) is calculated using the following formula:

[0020] Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction:

[0021] f(x,y1)=α1f(x1,y1)+α2f(x2,y1)

[0022] f(x,y2)=α3f(x1,y2)+α4f(x2,y2)

[0023]

[0024] α1=α 1_ori / (α 1_ori +α 2_ori )

[0025] α2=α 2_ori / (α 1_ori +α 2_ori )

[0026]

[0027] α3=α 3_ori / (α 3_ori +α 4_ori )

[0028] α4=α 4_ori / (α 3_ori +α 4_ori )

[0029] The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction:

[0030]

[0031] Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α3_ori and α 4_ori are intermediate parameters, (x1,y1), (x2,y1), (x1,y2) and (x2,y2) represent the four original points closest to the interpolation point. and Represents the weights of the four original points.

[0032] Based on the further improvement of the above method, the weight of the original point is calculated using the following formula:

[0033]

[0034] Among them, θ t Represents the percentage of pixels of the tth component in the renal pathology image.

[0035] Based on a further improvement of the above method, the following loss function is used to train the multi-channel coding fusion image segmentation model based on the sample set:

[0036] Loss = L DiceLoss +L SDFL

[0037] Among them, L DiceLoss represents the Dice loss of the training sample, L SDFL represents the size density focal loss of the training samples, ω1 and ω2 represent the weights.

[0038] Based on the further improvement of the above method, the following formula is used to calculate the size density focal loss of the training sample:

[0039]

[0040] in, represents the probability that the i-th pixel of the training sample predicted by the model belongs to the t-th component, γ t Indicates the penalty factor corresponding to the t-th component, k t Represents the number of pixels of the t-th component of the sample, represents the size probability distribution function corresponding to the t-th component, M represents the number of sample pixels, and N represents the number of component types.

[0041] Based on the further improvement of the above method, the size probability distribution function of each type is obtained in the following way:

[0042] The number of pixels Cn belonging to the tth component in each sample of the statistical sample set t ;

[0043] The number of pixels Cn t is the independent variable, and the number of pixels of the tth component in the sample is Cn tThe proportion of the number of samples to the total number of samples is taken as the dependent variable, and data fitting is performed to obtain the size probability distribution function.

[0044] Based on the further improvement of the above method, the kidney pathology images are segmented and the target images are screened from the segmented images, including:

[0045] Cutting the kidney pathology image according to a first size to obtain a plurality of cut images;

[0046] The renal pathology image is converted into a grayscale image, and the average grayscale value of the annotated area is calculated according to the component annotation information corresponding to the renal pathology image;

[0047] If, in the corresponding area of ​​the grayscale image, the number of pixels whose grayscale values ​​exceed the average grayscale value accounts for a proportion exceeding a first threshold value in the total number of pixels in the cut image, the cut image is the target image.

[0048] Based on a further improvement of the above method, the kidney pathology image is cut according to the first size to obtain multiple cut images, including:

[0049] A sliding window of a first size is used to slide on the renal pathology image from left to right and from top to bottom. If the renal pathology image in the current sliding window is smaller than the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window; the renal pathology image in the current sliding window is used as the cut image.

[0050] On the other hand, an embodiment of the present invention provides a kidney component segmentation system based on deep learning, comprising the following modules:

[0051] A sample set construction module is used to obtain kidney pathology images and corresponding component annotation information, segment the kidney pathology images, screen target images from the segmented images, and construct a sample set based on the target images and corresponding component annotation information;

[0052] A model training module is used to construct a multi-channel coding fusion image segmentation model, and train the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0053] The segmentation module is used to segment the kidney pathology image to be segmented, select the target image from the segmented image and input it into the kidney component segmentation model to obtain the segmentation result corresponding to the target image; and obtain the component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

[0054] Compared with the existing technology, the present invention constructs a sample set by cutting the kidney pathology image and screening the target image from the cut image, thereby introducing less noise and making the trained segmentation model more accurate. By constructing a multi-channel coding fusion image segmentation model and using multi-channel coding to extract image features, the feature extraction capability is improved and the demand for data volume is reduced, thereby improving the segmentation accuracy of the model. The trained kidney component segmentation model can be used to quickly segment the kidney pathology image to be segmented, thereby improving the segmentation efficiency and accuracy.

[0055] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0057] Figure 1 Flowchart of a kidney component segmentation method based on deep learning according to an embodiment of the present invention;

[0058] Figure 2 Schematic diagram of interpolation points according to an embodiment of the present invention;

[0059] Figure 3 This is a block diagram of a kidney component segmentation system based on deep learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0061] A specific embodiment of the present invention discloses a kidney component segmentation method based on deep learning, such as Figure 1 As shown, the following steps are included:

[0062] S1. Obtain a renal pathology image and corresponding component annotation information, segment the renal pathology image, select a target image from the segmented image, and construct a sample set based on the target image and corresponding component annotation information;

[0063] S2. Constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0064] S3. Cut the kidney pathology image to be segmented, select a target image from the cut image, input it into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and obtain a component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

[0065] Compared with the existing technology, the deep learning-based kidney component segmentation method provided in this embodiment constructs a sample set by cutting the kidney pathology image and screening the target image from the cut image, thereby introducing less noise and making the trained segmentation model more accurate. By constructing a multi-channel coding fusion image segmentation model and using multi-channel coding to extract image features, the feature extraction capability is improved and the demand for data volume is reduced, thereby improving the segmentation accuracy of the model. The trained kidney component segmentation model can be used to quickly segment the kidney pathology image to be segmented, thereby improving the segmentation efficiency and accuracy.

[0066] During implementation, a renal pathology image of the patient is obtained. The renal pathology image is a digital scanned slice (WSI) image of the patient's renal biopsy pathology obtained by scanning the entire slice after PAS staining of the patient's renal biopsy histological slice using a slice scanner.

[0067] The corresponding component annotation information is obtained by professional renal pathologists using annotation software to annotate the contours of each kidney component. The annotation objects include:

[0068] Normal glomerular outline: This is the basement membrane of the renal capsule. The normal glomerular basement membrane appears purple-red in PAS staining, is quasi-circular in shape, and contains capillary loops. When marking, outline a closed pattern along the basement membrane. The basement membrane of some marginal glomeruli is incomplete, requiring the basement membrane to be extended and outlined to restore the complete outline. A normal glomerulus is a non-sclerotic glomerulus.

[0069] Outline of the affected glomerulus: If the basement membrane of the renal capsule of the affected glomerulus is intact, the outline is drawn directly along the basement membrane. In cases of severe lesions, where the basement membrane is unclear, the outline must be determined based on medical knowledge to determine the original basement membrane location. If the basement membrane is incomplete, the complete outline is drawn based on medical knowledge. A affected glomerulus is a glomerulus with sclerosis.

[0070] Tubular basement membrane: The tubular basement membrane appears purple-red in PAS staining and is oval, round, or irregularly tubular in shape. Columnar epithelial cells line the inner lining, and a central cavity is defined. When labeling, outline a closed pattern along the basement membrane. Some glomerular basement membrane edges are incomplete; therefore, outline the basement membrane to restore the complete outline.

[0071] Arteriolar wall margin: The arteriolar wall has an elliptical, circular, or irregular tubular cross-section. The outer margin is the outer edge of the smooth muscle layer and is outlined along the outermost smooth muscle layer. Some marginal arterioles may have incomplete outer margins; therefore, the outer margin must be extended to complete the outline. The outer margin of a diseased arteriolar may be irregular or unclear, so the outer margin must be determined based on medical knowledge before outlining.

[0072] Renal interstitium: After dividing the above components, the remaining irregular continuous part of the renal tissue is the renal interstitial area.

[0073] That is, the four component types of normal glomeruli, diseased glomeruli, renal tubules and arterioles in the renal pathology images are labeled.

[0074] Obtained kidney pathology images often contain hundreds of millions or even billions of pixels, far exceeding the image size that deep learning models can handle. Previous studies have often used simple, equally spaced segmentation, which can easily introduce noise into the model. To improve segmentation quality and efficiency, this paper segments kidney pathology images, selects target images from the segmented images, and constructs a sample set based on the target images and their corresponding component annotations.

[0075] Specifically, the kidney pathology image is segmented, and the target image is screened from the segmented image, including:

[0076] Cutting the kidney pathology image according to a first size to obtain a plurality of cut images;

[0077] The renal pathology image is converted into a grayscale image, and the average grayscale value of the annotated area is calculated according to the component annotation information corresponding to the renal pathology image;

[0078] If, in the corresponding area of ​​the grayscale image, the number of pixels whose grayscale values ​​exceed the average grayscale value accounts for a proportion exceeding a first threshold value in the total number of pixels in the cut image, the cut image is the target image.

[0079] The kidney pathology image is cut according to the first size to obtain a plurality of cut images, including:

[0080] A sliding window of a first size is used to slide on the renal pathology image from left to right and from top to bottom. If the renal pathology image in the current sliding window is smaller than the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window, and the renal pathology image in the current sliding window is used as the cut image.

[0081] During implementation, the first size is set according to the input size of the first encoding module, for example, 512*512. During implementation, a sliding window of the first size is used to slide on the renal pathology image in a left-to-right and top-to-bottom order. The window sliding step is 512. If the size of the renal pathology image in the window does not meet the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window, and the renal pathology image in the current sliding window is used as the cut image, that is, the insufficient part is overlapped with the previous window to ensure that the cut data is all 512*512 in size.

[0082] The original kidney pathology image is converted to a grayscale image. Based on the corresponding component standard information, the average grayscale value a of the annotated area is calculated. Target images are then screened based on this average grayscale value. For each segmented image, the percentage of elements with grayscale values ​​exceeding the average grayscale value in the corresponding region of the grayscale image is calculated. If this percentage exceeds a first threshold, the foreground portion of the segmented image is significant and contains more information, and this segmented image is selected as the target image. The target image and its corresponding annotated information are used as samples to remove noise and improve the accuracy of the segmentation model.

[0083] During implementation, the images in the sample set can be rotated, scaled, flipped, translated, color-changed, and noise added to perform image enhancement. The sample set is divided into a training sample set, a validation sample set, and a test sample set in a ratio of 7:1:2.

[0084] Existing model architectures for medical image segmentation typically use UNet or simple variations, which lack the ability to extract features, resulting in limited segmentation accuracy. This is particularly true for renal components, such as sclerotic glomeruli and arteries. To improve feature extraction and segmentation accuracy, this paper constructs a multi-channel encoding fusion image segmentation model that extracts both semantic and visual features, thereby fusing multi-channel features. This improves the model's feature extraction and generalization capabilities, while reducing data requirements.

[0085] The specific multi-channel coding fusion image segmentation model includes:

[0086] The first encoding module includes a plurality of sequentially connected downsampling units, and is used to extract multi-layer semantic features from the sample image;

[0087] The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image;

[0088] The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module;

[0089] A decoding module, comprising a plurality of upsampling units connected in sequence; configured to recover a feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map;

[0090] The output layer is used to classify components based on the final feature map to obtain a component segmentation map;

[0091] The number of downsampling units in the first encoding module is the same as the number of upsampling units in the encoding module and corresponds one to one;

[0092] A spatial channel cross attention module is connected between the downsampling unit and the corresponding upsampling unit, which is used to perform attention extraction on the semantic features extracted by the downsampling unit and pass them to the corresponding upsampling unit.

[0093] During implementation, semantic features are extracted through the first encoding module. The first encoding module consists of four sequentially connected downsampling units, which gradually reduce the size of the feature map while increasing the number of channels in the feature map, thereby capturing image features at different scales and improving the accuracy of the segmentation results.

[0094] When implemented, the downsampling unit consists of a convolutional layer, an activation function, and a max pooling layer connected sequentially.

[0095] Convolutional layer (Conv Layer): used for feature extraction. Each downsampling unit usually contains two consecutive convolution operations, using a 3×3 convolution kernel.

[0096] Activation function Leaky ReLU: Each convolutional layer is followed by an activation function Leaky ReLU to increase the nonlinear expression ability of the model.

[0097] Max Pooling: Finally, downsampling is performed through a 2×2 max pooling layer to reduce the spatial dimension of the image by half, thereby reducing computational complexity and obtaining more abstract features.

[0098] The last downsampling unit of the first encoding module is connected to the middle layer, which is used to compress and refine the semantic features output by the downsampling unit while preserving the semantic information of the features. In implementation, the middle layer is a convolutional layer.

[0099] The decoding module is connected after the middle layer. The decoding module is symmetrical with the first encoding module, and its upsampling units correspond one-to-one with the downsampling units of the first encoding module. The visual features extracted by the second encoding module also serve as the input of the first upsampling unit of the decoding module.

[0100] During implementation, the second encoding module includes an image filling module and a ViT encoder.

[0101] The ViT encoder is a pre-trained encoder with an input size of 1024*1024, while the first encoding module used for semantic feature extraction typically has a size smaller than 1024*1024. Therefore, the second encoding module uses weighted bilinear interpolation via the image padding module to pad the cropped image so that the image meets the input requirements of the ViT encoder.

[0102] For example, if the original image size is a*b and the image size after interpolation is n*m, then the side length ratios are n / a and m / b respectively. Divide the length and width of the original image into equal parts (n-1) / (a-1) and (m-1) / (b-1) to get the interpolation points. Figure 2 As shown, Figure 2 The original image on the left is 3*3, with a total of 9 pixels. The padded image is 4*4, with a total of 16 pixels. The pixels of the padded image are then divided equally into 2 / 3 and 2 / 3 to obtain the pixels of the padded image. Pixels whose coordinates are exactly the same as those of the original image can simply use the corresponding pixel values ​​in the original image. For example, the pixels in the top-left, top-right, bottom-left, and bottom-right corners of the padded image correspond one-to-one with the pixels in the top-left, top-right, bottom-left, and bottom-right corners of the original image, so the pixel values ​​of the original image can be used directly. The remaining points are interpolation points, and their pixel values ​​need to be calculated using weighted bilinear interpolation.

[0103] For each interpolation point, based on the four original points closest to it (i.e., the pixel points in the original image), for example, the second pixel in the first row, the nearest original points are the four original points on its left, right, lower left corner, and right corner. Then, weighted bilinear interpolation is performed based on these four original points to obtain the pixel value of the interpolation point.

[0104] Specifically, the pixel value of the interpolation point (x, y) is calculated using the following formula:

[0105] Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction:

[0106] f(x,y1)=α1f(x1,y1)+α2f(x2,y1)

[0107] f(x,y2)=α3f(x1,y2)+α4f(x2,y2)

[0108]

[0109] α1=α 1_ori / (α 1_ori +α 2_ori )

[0110] α2=α2_ori / (α 1_ori +α 2_ori )

[0111]

[0112] α3=α 3_ori / (α 3_ori +α 4_ori )

[0113] α4=α 4_ori / (α 3_ori +α 4_ori )

[0114] The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction:

[0115]

[0116] Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α 3_ori and α 4_ori are intermediate parameters, (x1,y1), (x2,y1), (x1,y2) and (x2,y2) represent the four original points closest to the interpolation point. and Represents the weights of the four original points.

[0117] It should be noted that (x1, y1), (x2, y1), (x1, y2), and (x2, y2) are the original points closest to the interpolation point in clockwise order. Among them, (x1, y1) is the original point closest to the upper left corner of the interpolation point. If there is no original point in the upper left corner of the interpolation point, the original point closest to the left of the interpolation point is used.

[0118] When implemented, the weight of the original point is calculated using the following formula:

[0119]

[0120] Among them, θ t Represents the pixel ratio of the tth component in the renal pathology image.

[0121] The sum of weights α1 and α2 is 1, and the sum of weights α3 and α4 is 1, to avoid affecting the stability of the interpolation process. The present invention adopts a weighted bilinear interpolation process that comprehensively considers category information and spatial position information. This can not only maintain good smoothness and transition effects during the image magnification process, but also ensure the consistency of the interpolation process, making the interpolation result more semantically consistent with reality.

[0122] The ViT encoder consists of a block module, a Transformer encoder, and output processing.

[0123] The block module is used to divide the input image into image blocks. Each image block is usually flattened into a vector, and a position encoding is added to preserve the position information of the image block.

[0124] During implementation, a fixed position code is used and the position code is generated based on the sine and cosine functions. The calculation formula is as follows:

[0125]

[0126] Here, pos is the position of an image block in the input sequence, i is the dimension index of the encoding vector, and d is the dimension size of the encoding vector. The use of sine and cosine functions ensures that the position encoding has periodic characteristics and the distance information between different positions is well preserved.

[0127] During implementation, the position code is added to the image block vector to obtain the image block vector with added position information.

[0128] The sequence of image block vectors output by the block segmentation module is passed as input to the subsequent Transformer encoder. The Transformer encoder consists of multiple Transformer blocks, each of which consists of a self-attention layer and a feedforward neural network layer.

[0129] The self-attention layer is used to model the relationship between the input sequences and allows the network to dynamically focus on the correlations between different positions when processing the input sequence.

[0130] Feedforward neural network layers are used to perform nonlinear transformation and dimensionality scaling on features at each position.

[0131] Output processing: The output of the last Transformer block is passed to a multi-layer perceptron (MLP), which combines and optimizes high-level features, maps the input information to the desired output space, and performs dimensionality reduction.

[0132] In order to better utilize the low-level and high-level feature information of the encoder and thus produce more accurate segmentation results, the present invention introduces a spatial channel cross-attention module as a skip connection between the downsampling unit of the first encoder and the corresponding upsampling unit of the decoder to capture more information based on spatial cross-attention and channel cross-attention, thereby filling the semantic gap between the encoder and decoder.

[0133] During implementation, the spatial attention and channel attention of the semantic features extracted by the downsampling unit are calculated respectively, and the obtained spatial attention and channel attention are matrix multiplied with the semantic features respectively. The results are added together to obtain the final attention feature, which is passed to the corresponding upsampling unit.

[0134] It should be noted that the first upsampling unit of the decoding module concatenates the semantic features output by the intermediate layer, the visual features output by the second encoder, and the attention features output by the jump connection, and uses the concatenation result as input for upsampling decoding.

[0135] Since hospital image data often have a serious distribution imbalance problem, in order to eliminate the impact of imbalance on the segmentation effect, the present invention considers the target size distribution information and uses size density focus loss and DICE loss as the training loss function to improve the model accuracy.

[0136] Specifically, the multi-channel coding fusion image segmentation model is trained based on the sample set using the following loss function:

[0137] Loss = L DiceLoss +L SDFL

[0138] Among them, L DiceLoss represents the Dice loss of the training sample, L SDFL Represents the size density focus loss of the training sample. It should be noted that Loss calculates the loss of a sample.

[0139] When implemented, the Dice loss is calculated using the following formula:

[0140]

[0141] Among them, X i Represents the set of pixels predicted by the model as the i-th component, Y i represents the set of pixels marked as the i-th component, N represents the number of component types, and |·| represents the number of elements.

[0142] Specifically, the size density focal loss of the training sample is calculated using the following formula:

[0143]

[0144] in, represents the probability that the i-th pixel of the training sample predicted by the model belongs to the t-th component, γ t Indicates the penalty factor corresponding to the t-th component, k t Represents the number of pixels of the t-th component of the sample, represents the size probability distribution function corresponding to the t-th component, M represents the number of sample pixels, and N represents the number of component types.

[0145] During implementation, the number of pixels Cn belonging to the tth component in each sample of the sample set is counted t ,

[0146] During implementation, the number of pixels Cn t is the independent variable, and the number of pixels of the tth component in the sample is Cn t The proportion of the number of samples to the total number of samples is taken as the dependent variable, and data fitting is performed to obtain the size probability distribution function.

[0147] During implementation, the maximum value Cn of the number of pixels of the t-th component in each sample can also be counted. t (max), μ*Cn t (max) is used as the right endpoint of the interval of pixels containing the tth component in the sample. (Multiplying by the expansion coefficient μ is to better handle the boundary case, μ>1). t (max)] is divided into K sub-intervals of equal length, and the proportion of the number of samples in each interval to the total number of samples is counted, and the proportion is used as the probability, thereby obtaining function.

[0148] By introducing the target size information of different component categories into the loss, that is, calculating the probability distribution function of the size distribution of each category, and calculating the penalty factor based on the probability distribution function, the penalty factor is not a fixed value, but is dynamically determined according to the component size probability. For samples with a larger component size probability, the segmentation difficulty is small and the penalty factor is smaller; for samples with a smaller component size probability, the segmentation difficulty is large and the penalty factor is larger, thereby balancing different distributions and improving the accuracy of segmentation.

[0149] The model is trained on the training set using the aforementioned loss function until the loss accuracy requirement is met or the number of training iterations is reached. The images in the validation set are then fed into the optimal model for validation. Hyperparameters are continuously adjusted based on the validation set's test results to obtain the optimal model. The model is ultimately applied to the test set to evaluate its usability. The resulting optimal model is the kidney segmentation model. The trained kidney segmentation model can be used to perform kidney image recognition.

[0150] During implementation, GPU processors, image processors or smart chips can be used for model training to improve training efficiency.

[0151] For the renal pathology image to be segmented, different from the cutting in step S1, a sliding window of the first size is used to slide on the renal pathology image in the order from left to right and from top to bottom. The sliding step does not exceed the side length of the first size to achieve overlapping cutting. After obtaining the prediction results, splicing is performed, so that the final result is more accurate.

[0152] A target image is selected from the cut image and input into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and then a component segmentation map of the kidney pathology image to be segmented is obtained based on the segmentation result.

[0153] It should be noted that the segmentation result is the probability value of each pixel point of the target image belonging to each component type.

[0154] Since the segmentation result corresponding to the target image is cut from the original image, the prediction results need to be spliced ​​together according to their positions in the original image.

[0155] There are overlapping parts in the segmentation of the original image. The maximum value of the predicted value of the overlapping part is taken during splicing to avoid information loss caused by segmentation. Finally, the component type to which each pixel point in the kidney pathology image to be segmented belongs is obtained, that is, the component segmentation map.

[0156] The component segmentation images are further post-processed as follows:

[0157] During implementation, for the component segmentation image, image binarization is performed for each component to obtain N binary images;

[0158] For each binarized image, perform the following steps:

[0159] The current region index is initialized to 1;

[0160] Traverse each non-zero pixel in turn. If the current pixel is the first non-zero pixel, the area corresponding to the current pixel is marked as the current area index. Otherwise, if there is no non-zero pixel in the four-connected domain of the current pixel, add 1 to the current area index Cd and mark the area corresponding to the current pixel as the current area index. If there is a non-zero pixel in the four-connected domain of the current pixel, mark the area corresponding to the current pixel as the area corresponding to the non-zero pixel in the four-connected domain.

[0161] The four connected domains are the four pixels above, below, left, and right of the current pixel.

[0162] For example, for a binary image of a normal glomerular component, the region of the first non-zero pixel is marked as 1. Then, the image is traversed downwards. If there are no non-zero pixels above, below, or on either side of the current non-zero pixel, the current pixel belongs to a new region, and the current region index is increased by 1. If there are non-zero pixels above, below, or on either side of the current non-zero pixel, the region of the current pixel and its four-connected domain should be consistent. The corresponding region is marked as the region of the pixels within the four-connected domain, thus performing connected domain partitioning on the binary image of the normal glomerular component. Each region after the traversal corresponds to a normal glomerulus instance, thus obtaining a component segmentation instance map.

[0163] During implementation, if there is a hole in the connected domain and the number of pixels in the hole is less than a second threshold, the hole is filled with the connected domain category surrounding the hole.

[0164] A specific embodiment of the present invention discloses a kidney component segmentation system based on deep learning, such as Figure 3 As shown, it includes the following modules:

[0165] A sample set construction module is used to obtain kidney pathology images and corresponding component annotation information, segment the kidney pathology images, screen target images from the segmented images, and construct a sample set based on the target images and corresponding component annotation information;

[0166] A model training module is used to construct a multi-channel coding fusion image segmentation model, and train the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0167] The segmentation module is used to segment the kidney pathology image to be segmented, select the target image from the segmented image and input it into the kidney component segmentation model to obtain the segmentation result corresponding to the target image; and obtain the component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

[0168] The above method embodiment and system embodiment are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process can be found in the above embodiment and will not be repeated here.

[0169] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0170] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A kidney component segmentation method based on deep learning, characterized in that: The following steps are involved: Obtaining a renal pathology image and corresponding component annotation information, segmenting the renal pathology image, screening a target image from the segmented image based on a grayscale image of the renal pathology image, and constructing a sample set based on the target image and the corresponding component annotation information; the components include normal glomeruli, diseased glomeruli, renal tubules, and arterioles; Constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model; Cutting the kidney pathology image to be segmented, screening the target image from the cut image and inputting it into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; Obtaining a component segmentation map of the kidney pathology image to be segmented based on the segmentation result; The multi-channel coding fusion image segmentation model includes: The first encoding module includes a plurality of sequentially connected downsampling units, and is used to extract multi-layer semantic features from the sample image; The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image; The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module; A decoding module, comprising a plurality of upsampling units connected in sequence, configured to recover a feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map; The output layer is used to classify components based on the final feature map to obtain a component segmentation map; The number of downsampling units in the first encoding module is the same as the number of upsampling units in the encoding module and corresponds one to one; A spatial channel cross attention module is connected between the downsampling unit and the corresponding upsampling unit, which is used to extract the semantic features extracted by the downsampling unit and pass them to the corresponding upsampling unit; The pixel value of the interpolation point (x, y) is calculated using the following formula: Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction: f(x,y1)=α1f(x1,y1)+α2f(x2,y1) f(x,y2)=α3f(x1,y2)+α4f(x2,y2) α1＝α 1_ori / (α 1_ori +α 2_ori ) α2＝α 2_ori / (α 1_ori +α 2_ori ) α3＝α 3_ori / (α 3_ori +α 4_ori ) α4＝α 4_ori / (α 3_ori +α 4_ori ) The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction: Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α 3_ori and α 4_ori are intermediate parameters, (x1,y1), (x2,y1), (x1,y2) and (x2,y2) represent the four original points closest to the interpolation point. and Represents the weights of the four original points.

2. The kidney component segmentation method based on deep learning according to claim 1, characterized in that The weight of the original point is calculated using the following formula: Among them, θ t Represents the percentage of pixels of the tth component in the renal pathology image.

3. The kidney component segmentation method based on deep learning according to claim 1, characterized in that The multi-channel coding fusion image segmentation model is trained based on the sample set using the following loss function: Loss=L DiceLoss +L SDFL Among them, L DiceLoss represents the Dice loss of the training sample, L SDFL represents the dimension-density focal loss of training samples.

4. The kidney component segmentation method based on deep learning according to claim 3, characterized in that: The size density focal loss of the training sample is calculated using the following formula: in, represents the probability that the i-th pixel of the training sample predicted by the model belongs to the t-th component, γ t Indicates the penalty factor corresponding to the t-th component, k t Represents the number of pixels of the t-th component of the sample, represents the size probability distribution function corresponding to the t-th component, M represents the number of sample pixels, and N represents the number of component types.

5. The kidney component segmentation method based on deep learning according to claim 4, characterized in that: The size probability distribution function of each type is obtained in the following way: The number of pixels Cn belonging to the tth component in each sample of the statistical sample set t ; The number of pixels Cn t is the independent variable, and the number of pixels of the tth component in the sample is Cn t The proportion of the number of samples to the total number of samples is taken as the dependent variable, and data fitting is performed to obtain the size probability distribution function.

6. The kidney component segmentation method based on deep learning according to claim 1, characterized in that: The kidney pathology images are segmented and the target images are screened from the segmented images, including: Cutting the kidney pathology image according to a first size to obtain a plurality of cut images; The renal pathology image is converted into a grayscale image, and the average grayscale value of the annotated area is calculated according to the component annotation information corresponding to the renal pathology image; If, in the corresponding area of ​​the grayscale image, the number of pixels whose grayscale values ​​exceed the average grayscale value accounts for a proportion exceeding a first threshold value in the total number of pixels in the cut image, the cut image is the target image.

7. The kidney component segmentation method based on deep learning according to claim 6, characterized in that: The kidney pathology image is cut according to the first size to obtain a plurality of cut images, including: A sliding window of a first size is used to slide on the renal pathology image from left to right and from top to bottom. If the renal pathology image in the current sliding window is smaller than the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window; the renal pathology image in the current sliding window is used as the cut image.

8. A kidney component segmentation system based on deep learning, characterized in that: Includes the following modules: a sample set construction module for obtaining renal pathology images and corresponding component annotation information, segmenting the renal pathology images, screening target images from the segmented images based on the grayscale images of the renal pathology images, and constructing a sample set based on the target images and corresponding component annotation information; the components include normal glomeruli, diseased glomeruli, renal tubules, and arterioles; A model training module is used to construct a multi-channel coding fusion image segmentation model, and train the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model; A segmentation module is used to segment the kidney pathology image to be segmented, select a target image from the segmented image, and input it into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; Obtaining a component segmentation map of the kidney pathology image to be segmented based on the segmentation result; The multi-channel coding fusion image segmentation model includes: The first encoding module includes a plurality of sequentially connected downsampling units, and is used to extract multi-layer semantic features from the sample image; The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image; The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module; A decoding module, comprising a plurality of upsampling units connected in sequence, configured to recover a feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map; The output layer is used to classify components based on the final feature map to obtain a component segmentation map; The number of downsampling units in the first encoding module is the same as the number of upsampling units in the encoding module and corresponds one to one; A spatial channel cross attention module is connected between the downsampling unit and the corresponding upsampling unit, which is used to extract the semantic features extracted by the downsampling unit and pass them to the corresponding upsampling unit; The pixel value of the interpolation point (x, y) is calculated using the following formula: Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction: f(x,y1)=α1f(x1,y1)+α2f(x2,y1) f(x,y2)=α3f(x1,y2)+α4f(x2,y2) α1＝α 1_ori / (α 1_ori +α 2_ori ) α2＝α 2_ori / (α 1_ori +α 2_ori ) α3＝α 3_ori / (α 3_ori +α 4_ori ) α4＝α 4_ori / (α 3_ori +α 4_ori ) The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction: Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α 3_ori and α 4_ori are intermediate parameters, (x1,y1), (x2,y1), (x1,y2) and (x2,y2) represent the four original points closest to the interpolation point. and Represents the weights of the four original points.

Images