Kidney component segmentation method and system based on deep learning

Abstract

The invention relates to a kidney component segmentation method and system based on deep learning, belongs to the technical field of medical image segmentation, and solves the problem of low accuracy of kidney component segmentation in the prior art. The method comprises the following steps: acquiring a kidney pathological image and corresponding component labeling information, cutting the kidney pathological image, screening a target image from the cut image, and constructing a sample set based on the target image and the corresponding component labeling information; constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model; cutting a to-be-segmented kidney pathological image, screening a target image from the cut images, and inputting the screened target image into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and obtaining a component segmentation map of the to-be-segmented kidney pathological image based on the segmentation result. And high-precision kidney component segmentation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and system for kidney component segmentation based on deep learning. Background Art

[0002] Chronic kidney disease (CKD) has been recognized as a major public health problem worldwide. A national cross-sectional survey showed that the prevalence of CKD was 10.8%, indicating that nearly 120 million adults in my country suffer from kidney disease. Renal pathology is the gold standard for diagnosing kidney disease and a bridge between pathogenesis and clinical manifestations. The acquisition and analysis of pathological morphological information in renal biopsy specimens is crucial for assessing the severity and prognosis of the disease and the choice of treatment decisions, and is also an important part of basic renal research.

[0003] However, the traditional pathological evaluation method is only based on the existing knowledge system of pathologists, and semi-quantitatively evaluates several pathological indicators of glomeruli, tubulointerstitium and arterioles to reflect the severity of the disease. With the deepening of the study of kidney diseases, traditional evaluation is increasingly unable to meet the needs of clinical and scientific research. The next generation of renal pathology needs to replace traditional evaluation with more sophisticated and comprehensive evaluation methods.

[0004] Renal pathology images contain several main components: glomeruli, tubules, arterioles, and renal interstitium. Each component can have lesions of different types and severity, and these lesions are jointly involved in the progression of kidney disease. Accurate assessment of the lesions of each component is of great significance for the overall assessment of kidney disease, treatment decision-making, and prognosis. Accurate assessment requires measurement and statistics of the lesions of each component separately, which must be based on the fine segmentation of the boundaries of each component in the renal pathology image, and further analysis based on the segmentation results. For example, on the basis of accurate segmentation of the boundaries of glomeruli and tubules, the size, shape, and uniformity of glomeruli and tubules are further measured to comprehensively reflect the severity of kidney disease. Boundary segmentation and identification and classification of each component require medical knowledge and rich experience of renal pathologists, but manual segmentation and classification require a lot of time and effort. One specimen usually takes several hours, which is difficult to implement in daily work. In recent years, with the development of deep learning algorithms, GPUs, smart chips, etc., accurate evaluation of image recognition of large samples has become possible in computer vision.

[0005] The existing deep learning model architectures used in medical image segmentation are mostly traditional UNet and its simple variants, which have insufficient ability to extract features. Therefore, the segmentation accuracy of sclerotic glomeruli, arteries, etc. is low, and the segmentation precision of kidney components is low. Summary of the invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide a kidney component segmentation method and system based on deep learning to solve the problem of low accuracy of existing kidney component segmentation.

[0007] On the one hand, an embodiment of the present invention provides a method for segmenting kidney components based on deep learning, comprising the following steps:

[0008] Obtaining kidney pathology images and corresponding component annotation information, cutting the kidney pathology images, screening target images from the cut images, and constructing a sample set based on the target images and corresponding component annotation information;

[0009] Constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0010] The kidney pathology image to be segmented is cut, and a target image is selected from the cut image and input into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and a component segmentation map of the kidney pathology image to be segmented is obtained based on the segmentation result.

[0011] Based on the further improvement of the above method, the multi-channel coding fusion image segmentation model includes:

[0012] The first encoding module includes a plurality of downsampling units connected in sequence, and is used to extract multi-layer semantic features from the sample image;

[0013] The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image by using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image;

[0014] The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module;

[0015] A decoding module, comprising a plurality of upsampling units connected in sequence, for recovering a feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map;

[0016] The output layer is used to classify components based on the final feature map to obtain a component segmentation map;

[0017] The number of down-sampling units of the first encoding module is the same as the number of up-sampling units of the encoding module and corresponds one to one;

[0018] A spatial channel cross attention module is connected between the down-sampling unit and the corresponding up-sampling unit, which is used to perform attention extraction on the semantic features extracted by the down-sampling unit and pass them to the corresponding up-sampling unit.

[0019] Based on the further improvement of the above method, the pixel value of the interpolation point (x, y) is calculated using the following formula:

[0020] Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction:

[0021] f(x,y1)=α1f(x1,y1)+α2f(x2,y1)

[0022] f(x,y2)=α3f(x1,y2)+α4f(x2,y2)

[0023]

[0024] α1=α 1_ori / (α 1_ori +α 2_ori )

[0025] α2=α 2_ori / (α 1_ori +α 2_ori )

[0026]

[0027] α3=α 3_ori / (α 3_ori +α 4_ori )

[0028] α4=α 4_ori / (α 3_ori +α 4_ori )

[0029] The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction:

[0030]

[0031] Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α3_ori and α 4_ori are intermediate parameters, (x1,y1), (x2,y1), (x1,y2) and (x2,y2) represent the four original points closest to the interpolation point. and Represents the weights of the four original points.

[0032] Based on the further improvement of the above method, the weight of the original point is calculated using the following formula:

[0033]

[0034] Among them, θ t Represents the percentage of pixels of the tth component in the renal pathology image.

[0035] Based on the further improvement of the above method, the multi-channel coding fusion image segmentation model is trained based on the sample set using the following loss function:

[0036] Loss = L DiceLoss +L SDFL

[0037] Among them, L DiceLoss represents the Dice loss of the training sample, L SDFL represents the size density focal loss of training samples, ω1 and ω2 represent weights.

[0038] Based on the further improvement of the above method, the following formula is used to calculate the size density focal loss of the training sample:

[0039]

[0040] in, represents the probability that the i-th pixel of the training sample predicted by the model belongs to the t-th component, γ t represents the penalty factor corresponding to the tth component, k t Represents the number of pixels of the tth component of the sample, represents the size probability distribution function corresponding to the t-th component, M represents the number of sample pixels, and N represents the number of component types.

[0041] Based on the further improvement of the above method, the size probability distribution function of each type is obtained in the following way:

[0042] The number of pixels Cn belonging to the tth component in each sample of the statistical sample set t ;

[0043] The number of pixels Cn t is the independent variable, and the number of pixels of the tth component in the sample is Cn tThe proportion of the number of samples to the total number of samples is taken as the dependent variable, and data fitting is performed to obtain the size probability distribution function.

[0044] Based on the further improvement of the above method, the renal pathological images are cut and the target images are screened from the cut images, including:

[0045] Cutting the kidney pathology image according to a first size to obtain a plurality of cut images;

[0046] The renal pathological image is converted into a grayscale image, and the average grayscale value of the annotated area is calculated according to the component annotation information corresponding to the renal pathological image;

[0047] If, in the corresponding area of ​​the grayscale image, the proportion of the number of pixels whose grayscale values ​​exceed the average grayscale value to the total number of pixels of the cut image exceeds a first threshold, the cut image is the target image.

[0048] Based on the further improvement of the above method, the kidney pathology image is cut according to the first size to obtain multiple cut images, including:

[0049] A sliding window of a first size is used to slide on the renal pathology image in a sequence from left to right and from top to bottom. If the renal pathology image in the current sliding window is smaller than the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window; the renal pathology image in the current sliding window is used as the cutting image.

[0050] On the other hand, an embodiment of the present invention provides a kidney component segmentation system based on deep learning, comprising the following modules:

[0051] A sample set construction module is used to obtain kidney pathology images and corresponding component annotation information, cut the kidney pathology images, select target images from the cut images, and construct a sample set based on the target images and corresponding component annotation information;

[0052] A model training module, used for constructing a multi-channel coded fusion image segmentation model, and training the multi-channel coded fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0053] The segmentation module is used to segment the kidney pathology image to be segmented, select the target image from the segmented image and input it into the kidney component segmentation model to obtain the segmentation result corresponding to the target image; and obtain the component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

[0054] Compared with the prior art, the present invention constructs a sample set by cutting the kidney pathology image and screening the target image from the cut image, thereby introducing less noise and making the trained segmentation model more accurate. By constructing a multi-channel coding fusion image segmentation model and using multi-channel coding to extract image features, the feature extraction capability is improved and the demand for data volume is reduced, thereby improving the segmentation accuracy of the model. The trained kidney component segmentation model can be used to quickly segment the kidney pathology image to be segmented, thereby improving the segmentation efficiency and accuracy.

[0055] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0057] Figure 1 This is a flow chart of a method for segmenting kidney components based on deep learning according to an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of interpolation points in an embodiment of the present invention;

[0059] Figure 3 This is a block diagram of a kidney component segmentation system based on deep learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0061] A specific embodiment of the present invention discloses a kidney component segmentation method based on deep learning, such as Figure 1 As shown, the following steps are included:

[0062] S1. Obtain a kidney pathology image and corresponding component annotation information, cut the kidney pathology image, select a target image from the cut image, and construct a sample set based on the target image and the corresponding component annotation information;

[0063] S2, constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0064] S3, cutting the kidney pathology image to be segmented, selecting a target image from the cut images and inputting it into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and obtaining a component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

[0065] Compared with the prior art, the deep learning-based renal component segmentation method provided in this embodiment constructs a sample set by cutting renal pathological images and screening target images from the cut images, thereby introducing less noise and making the trained segmentation model more accurate. By constructing a multi-channel coding fusion image segmentation model and using multi-channel coding to extract image features, the feature extraction capability is improved and the demand for data volume is reduced, thereby improving the segmentation accuracy of the model. The trained renal component segmentation model can be used to quickly segment the renal pathological images to be segmented, thereby improving the segmentation efficiency and accuracy.

[0066] During implementation, a renal pathology image of the patient is obtained, which is a digital scanned slice (WSI) image of the patient's renal biopsy pathology obtained by scanning the entire slice after PAS staining of the patient's renal biopsy histological slice with a slice scanner.

[0067] The corresponding component annotation information is obtained by professional renal pathologists using annotation software to annotate the contours of each kidney component to obtain standard information. The annotation objects include:

[0068] Normal glomerular outline: that is, the basement membrane of the renal capsule. The normal glomerular basement membrane is purple-red in PAS staining, and is quasi-circular in shape, with capillary loop structures inside. When marking, a closed figure is outlined along the basement membrane. The basement membrane of some marginal glomeruli is incomplete, and the basement membrane needs to be extended to outline the complete restoration outline. Normal glomeruli are non-sclerotic glomeruli.

[0069] Outline of diseased glomerulus: When the basement membrane of the renal capsule of the diseased glomerulus is intact, the outline is drawn directly along the basement membrane. The basement membrane of the glomerulus with severe disease is unclear, and the original basement membrane position needs to be determined based on medical knowledge for outline. When the basement membrane is incomplete, the complete outline is drawn based on medical knowledge. The diseased glomerulus is the glomerular sclerosis glomerulus.

[0070] Tubular basement membrane: The tubular basement membrane is purple-red in PAS staining, and is oval, round or irregularly tubular in shape, with columnar epithelial cells on the inside and a cavity in the middle. When marking, draw a closed pattern along the basement membrane. Some marginal glomerular basement membranes are incomplete, and the basement membrane needs to be extended to outline the complete restoration outline.

[0071] Outer edge of arteriolar wall: The cross section of the arteriolar wall is elliptical, circular or irregularly tubular, and the outer edge is the outer edge of the smooth muscle layer, which is outlined along the outermost smooth muscle boundary. The outer edge of some marginal arterioles is incomplete, and the outer edge needs to be extended to outline the complete restoration outline. The outer edge of the diseased arteriolar may be irregular or unclear, and the outer edge contour needs to be judged based on medical knowledge for outline.

[0072] Renal interstitium: After dividing the above components, the remaining irregular continuous part of the renal tissue is the renal interstitial area.

[0073] That is, the four component types of normal glomeruli, diseased glomeruli, renal tubules and arterioles in the renal pathology images are labeled.

[0074] The obtained renal pathology images often contain hundreds of millions or even billions of pixels, which is far beyond the image size that deep learning models can handle. Previous studies have mostly used simple equal-interval segmentation, which easily introduces noise data into the model. In order to improve the segmentation quality and efficiency, the present invention segments the renal pathology images, selects the target images from the segmented images, and constructs a sample set based on the target images and the corresponding component annotation information.

[0075] Specifically, the kidney pathology image is cut and the target image is selected from the cut images, including:

[0076] Cutting the kidney pathology image according to a first size to obtain a plurality of cut images;

[0077] The renal pathological image is converted into a grayscale image, and the average grayscale value of the annotated area is calculated according to the component annotation information corresponding to the renal pathological image;

[0078] If, in the corresponding area of ​​the grayscale image, the proportion of the number of pixels whose grayscale values ​​exceed the average grayscale value to the total number of pixels of the cut image exceeds a first threshold, the cut image is the target image.

[0079] The kidney pathology image is cut according to the first size to obtain a plurality of cut images, including:

[0080] A sliding window of a first size is used to slide on the renal pathology image in a sequence from left to right and from top to bottom. If the renal pathology image in the current sliding window is smaller than the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window, and the renal pathology image in the current sliding window is used as the cutting image.

[0081] During implementation, the first size is set according to the input size of the first encoding module, for example, 512*512. During implementation, a sliding window of the first size is used to slide on the renal pathology image in the order from left to right and from top to bottom. The window sliding step is 512. If the size of the renal pathology image in the window does not meet the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window, and the renal pathology image in the current sliding window is used as the cutting image, that is, the insufficient part is overlapped with the previous window to ensure that the data obtained by cutting are all 512*512 in size.

[0082] The original kidney pathology image is converted into a grayscale image. According to the corresponding component standard information, the average grayscale value a of the annotated area is calculated, and then the target image is screened based on the average grayscale value. During implementation, for each cut image, the proportion of elements whose grayscale value exceeds the average grayscale value in the area corresponding to the grayscale image is calculated. If the proportion exceeds the first threshold, it means that the foreground part of the cut image accounts for a large proportion and contains more information, and it is used as the target image. The target image and its corresponding annotated information are used as samples to remove noise and improve the accuracy of the segmentation model.

[0083] During implementation, the images in the sample set can be rotated, scaled, flipped, translated, color-changed, and noise added to perform image enhancement. The sample set is divided into a training sample set, a verification sample set, and a test sample set in a ratio of 7:1:2.

[0084] The existing model architecture for medical image segmentation usually uses UNet or its simple deformation, which has insufficient feature extraction capability, so the segmentation accuracy is limited, especially for kidney components, such as sclerotic glomeruli and arteries. In order to improve the feature extraction capability and segmentation accuracy, the present invention constructs a multi-channel coding fusion image segmentation model, which not only extracts image semantic features but also extracts visual features, thereby fusing multi-channel features, improving the model's feature extraction capability and generalization capability, and reducing the demand for data volume.

[0085] The specific multi-channel coding fusion image segmentation model includes:

[0086] The first encoding module includes a plurality of downsampling units connected in sequence, and is used to extract multi-layer semantic features from the sample image;

[0087] The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image by using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image;

[0088] The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module;

[0089] A decoding module, comprising a plurality of upsampling units connected in sequence; used to restore the feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map;

[0090] The output layer is used to classify components based on the final feature map to obtain a component segmentation map;

[0091] The number of down-sampling units of the first encoding module is the same as the number of up-sampling units of the encoding module and corresponds one to one;

[0092] A spatial channel cross attention module is connected between the down-sampling unit and the corresponding up-sampling unit, which is used to perform attention extraction on the semantic features extracted by the down-sampling unit and pass them to the corresponding up-sampling unit.

[0093] During implementation, semantic features are extracted through a first encoding module, which includes four sequentially connected downsampling units, gradually reducing the size of the feature map and increasing the number of channels of the feature map, thereby capturing image features of different scales and improving the accuracy of the segmentation result.

[0094] When implemented, the downsampling unit includes a convolutional layer, an activation function, and a max pooling layer connected sequentially.

[0095] Convolutional layer: used for feature extraction. Each downsampling unit usually contains two consecutive convolution operations, using a 3×3 convolution kernel.

[0096] Activation function Leaky ReLU: Each convolutional layer is followed by an activation function Leaky ReLU to increase the nonlinear expression ability of the model.

[0097] Max Pooling: Finally, downsampling is performed through a 2×2 max pooling layer to reduce the spatial dimension of the image by half, thereby reducing computational complexity and obtaining more abstract features.

[0098] The last downsampling unit of the first encoding module is connected to the middle layer, which is used to compress and refine the semantic features output by the downsampling unit, while maintaining the semantic information of the features. In implementation, the middle layer is a convolutional layer.

[0099] The decoding module is connected after the middle layer. The decoding module is symmetrical with the first encoding module, and its upsampling unit corresponds one-to-one with the downsampling unit of the first encoding module. The visual features extracted by the second encoding module are also used as the input of the first upsampling unit of the decoding module.

[0100] During implementation, the second encoding module includes an image filling module and a ViT encoder.

[0101] The ViT encoder is a pre-trained encoder with an input size of 1024*1024, while the first encoding module used for semantic feature extraction usually has a size smaller than 1024*1024. Therefore, the second encoding module uses weighted bilinear interpolation to fill the cropped image through the image filling module, so that the image meets the input requirements of the ViT encoder.

[0102] For example, if the original image size is a*b and the image size after interpolation filling is n*m, then the side length ratios are n / a and m / b respectively. Divide the length and width of the original image into equal parts (n-1) / (a-1) and (m-1) / (b-1) to get the interpolation points. Figure 2 As shown, Figure 2 The left side is the original image, the image size is 3*3, a total of 9 pixels, the padded image size is 4*4, a total of 16 pixels, then the side length of the original image is divided into 2 / 3, 2 / 3 to obtain the pixels of the padded image. The pixel points that are completely consistent with the pixel coordinates of the original image can directly take the corresponding pixel values ​​in the original image. For example, the pixel points in the upper left corner, upper right corner, lower left corner, and lower right corner of the padded image correspond one-to-one with the pixel points in the upper left corner, upper right corner, lower left corner, and lower right corner of the original image, so the pixel values ​​of the original image can be directly taken. The other points are interpolation points, and the pixel values ​​need to be calculated by weighted bilinear interpolation.

[0103] For each interpolation point, based on the four original points closest to it (i.e., the pixel points in the original image), for example, the second pixel in the first row, the nearest original points are the four original points on its left, right, lower left corner, and right small corner. Then, weighted bilinear interpolation is performed based on these four original points to obtain the pixel value of the interpolation point.

[0104] Specifically, the pixel value of the interpolation point (x, y) is calculated using the following formula:

[0105] Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction:

[0106] f(x,y1)=α1f(x1,y1)+α2f(x2,y1)

[0107] f(x,y2)=α3f(x1,y2)+α4f(x2,y2)

[0108]

[0109] α1=α 1_ori / (α 1_ori +α 2_ori )

[0110] α2=α2_ori / (α 1_ori +α 2_ori )

[0111]

[0112] α3=α 3_ori / (α 3_ori +α 4_ori )

[0113] α4=α 4_ori / (α 3_ori +α 4_ori )

[0114] The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction:

[0115]

[0116] Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α 3_ori and α 4_ori are intermediate parameters, (x1,y1), (x2,y1), (x1,y2) and (x2,y2) represent the four original points closest to the interpolation point. and Represents the weights of the four original points.

[0117] It should be noted that (x1, y1), (x2, y1), (x1, y2) and (x2, y2) are the original points closest to the interpolation points arranged in clockwise order, among which (x1, y1) is the original point closest to the upper left corner of the interpolation point. If there is no original point in the upper left corner of the interpolation point, it is the original point closest to the left side of the interpolation point.

[0118] When implemented, the weight of the original point is calculated using the following formula:

[0119]

[0120] Among them, θ t Represents the pixel proportion of the tth component in the renal pathology image.

[0121] The sum of weights α1 and α2 is 1, and the sum of weights α3 and α4 is 1, to avoid affecting the stability of the interpolation process. The present invention adopts a weighted bilinear interpolation process to comprehensively consider the category information and spatial position information, which can not only keep the image smooth and transition effect during the magnification process, but also ensure the consistency of the meaning of the interpolation process, so that the interpolation result is more semantically consistent with reality.

[0122] The ViT encoder consists of: a block module, a Transformer encoder, and output processing.

[0123] The block module is used to divide the input image into image blocks. Each image block is usually flattened into a vector, and a position encoding is added to preserve the position information of the image block.

[0124] During implementation, fixed position encoding is used and position encoding is generated based on sine and cosine functions. The calculation formula is as follows:

[0125]

[0126] Among them, pos is the position of a certain image block in the input sequence, i is the dimension index of the encoding vector, and d is the dimension size of the encoding vector. The use of sine and cosine functions ensures that the position encoding has periodic characteristics and the distance information between different positions can be well preserved.

[0127] During implementation, the position code is added to the image block vector to obtain the image block vector with added position information.

[0128] The image block vector sequence output by the block module is passed as input to the subsequent Transformer encoder. The Transformer encoder consists of multiple Transformer blocks, each of which consists of a self-attention layer and a feedforward neural network layer.

[0129] The self-attention layer is used to model the relationship between the input sequences and allows the network to dynamically focus on the correlations between different positions when processing the input sequence.

[0130] Feedforward neural network layers are used to perform nonlinear transformation and dimensionality scaling on features at each position.

[0131] Output Processing: The output of the last Transformer block is passed to a Multi-Layer Perceptron (MLP), which combines and optimizes high-level features, maps the input information to the desired output space, and performs dimensionality reduction.

[0132] In order to better utilize the low-level and high-level feature information of the encoder and thus produce more accurate segmentation results, the present invention introduces a spatial channel cross-attention module as a jump connection between the downsampling unit of the first encoder and the corresponding upsampling unit of the decoder, so as to capture more information based on spatial cross-attention and channel cross-attention, thereby filling the semantic gap between the encoder and the decoder.

[0133] During implementation, the spatial attention and channel attention of the semantic features extracted by the downsampling unit are calculated respectively, and the obtained spatial attention and channel attention are matrix multiplied with the semantic features respectively. The results are added together to obtain the final attention feature, which is passed to the corresponding upsampling unit.

[0134] It should be noted that the first upsampling unit of the decoding module concatenates the semantic features output by the intermediate layer, the visual features output by the second encoder, and the attention features output by the jump connection, and uses the concatenated result as input for upsampling decoding.

[0135] Since hospital image data often has a serious distribution imbalance problem, in order to eliminate the impact of imbalance on the segmentation effect, the present invention considers the target size distribution information and uses size density focus loss and DICE loss as the training loss function to improve the model accuracy.

[0136] Specifically, the multi-channel coding fusion image segmentation model is trained based on the sample set using the following loss function:

[0137] Loss = L DiceLoss +L SDFL

[0138] Among them, L DiceLoss represents the Dice loss of the training sample, L SDFL Represents the size density focus loss of the training sample. It should be noted that Loss calculates the loss of a sample.

[0139] When implemented, the Dice loss is calculated using the following formula:

[0140]

[0141] Among them, X i represents the set of pixels predicted by the model as the i-th component, Y i represents the set of pixels marked as the i-th component, N represents the number of component types, and |·| represents the number of elements.

[0142] Specifically, the size density focal loss of the training sample is calculated using the following formula:

[0143]

[0144] in, represents the probability that the i-th pixel of the training sample predicted by the model belongs to the t-th component, γ t represents the penalty factor corresponding to the tth component, k t Represents the number of pixels of the tth component of the sample, represents the size probability distribution function corresponding to the t-th component, M represents the number of sample pixels, and N represents the number of component types.

[0145] During implementation, the number of pixels Cn belonging to the tth component in each sample of the sample set is counted t ,

[0146] During implementation, the number of pixels Cn t is the independent variable, and the number of pixels of the tth component in the sample is Cn t The proportion of the number of samples to the total number of samples is taken as the dependent variable, and data fitting is performed to obtain the size probability distribution function.

[0147] During implementation, the maximum value Cn of the number of pixels of the t-th component in each sample can also be counted. t (max), μ*Cn t (max) is used as the right end point of the interval of pixel points containing the tth component in the sample. (Multiplying by the expansion coefficient μ is to better handle the boundary case, μ>1). t (max)] is divided into K sub-intervals of equal length, and the proportion of the number of samples in each interval to the total number of samples is counted, and the proportion is used as the probability, thereby obtaining function.

[0148] By introducing the target size information of different component categories into the loss, that is, calculating the probability distribution function of the size distribution of each category, and calculating the penalty factor based on the probability distribution function, the penalty factor is not a fixed value, but is dynamically determined according to the component size probability. For samples with a larger component size probability, the segmentation difficulty is small and the penalty factor is smaller; while for samples with a smaller component size probability, the segmentation difficulty is large and the penalty factor is larger, thereby balancing different distributions and improving the accuracy of segmentation.

[0149] The model is trained based on the above loss function on the training sample set until the loss accuracy requirement is met or the number of training iterations is reached, and the training is completed. The images in the validation sample set are input into the optimal model for validation, and the hyperparameters are continuously adjusted according to the test results of the validation set to obtain the optimal model, and the model is finally applied to the test set to evaluate the usability of the model. The optimal model obtained is the kidney component segmentation model. The trained kidney component segmentation model can be used for kidney component image recognition.

[0150] During implementation, GPU processors, image processors or smart chips can be used for model training to improve training efficiency.

[0151] For the renal pathology image to be segmented, different from the cutting in step S1, a sliding window of the first size is used to slide on the renal pathology image in the order from left to right and from top to bottom, and the sliding step length does not exceed the side length of the first size to achieve overlapping cutting. After the prediction results are obtained, splicing is performed, so that the final result is more accurate.

[0152] A target image is selected from the cut images and input into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and then a component segmentation map of the kidney pathology image to be segmented is obtained based on the segmentation result.

[0153] It should be noted that the segmentation result is the probability value of each pixel of the target image belonging to each component type.

[0154] Since the segmentation result corresponding to the target image is obtained by cutting the original image, the prediction results need to be spliced ​​together according to their positions in the original image.

[0155] There are overlapping parts in the segmentation of the original image. The maximum value of the predicted value of the overlapping part is taken during splicing to avoid information loss caused by segmentation. Finally, the component type to which each pixel in the kidney pathology image to be segmented belongs is obtained, that is, the component segmentation map.

[0156] The component segmentation images are further post-processed as follows:

[0157] During implementation, for the component segmentation image, image binarization is performed for each component to obtain N binary images;

[0158] For each binarized image, perform the following steps:

[0159] The current region index is initialized to 1;

[0160] Traverse each non-zero pixel in turn. If the current pixel is the first non-zero pixel, the area corresponding to the current pixel is marked as the current area index; otherwise, if there is no non-zero pixel in the four-connected domain of the current pixel, add 1 to the current area index Cd, and mark the area corresponding to the current pixel as the current area index. If there is a non-zero pixel in the four-connected domain of the current pixel, mark the area corresponding to the current pixel as the area corresponding to the non-zero pixel in the four-connected domain.

[0161] The four-connected domain is the four pixels above, below, left and right of the current pixel.

[0162] For example, for the binary image of the normal glomerular component, the area of ​​the first non-zero pixel is marked as 1, and then traversed downward in sequence. If there are no non-zero pixels above, below, left, and right of the current non-zero pixel, the current pixel belongs to a new area, and the current area index is increased by 1. If there are non-zero pixels above, below, left, and right of the current non-zero pixel, the area of ​​the current pixel and the pixel points in its four-connected domain should be consistent, and the corresponding area is marked as the area of ​​the pixel points in the four-connected domain, thereby dividing the binary image of the normal glomerular component into connected domains. Each area after the traversal corresponds to a normal glomerulus instance. Thus, a component segmentation instance image is obtained.

[0163] During implementation, if there is a hole in the connected domain and the number of hole pixels is less than the second threshold, the hole is filled with the connected domain category surrounding the hole.

[0164] A specific embodiment of the present invention discloses a kidney component segmentation system based on deep learning, such as Figure 3 As shown, it includes the following modules:

[0165] A sample set construction module is used to obtain kidney pathology images and corresponding component annotation information, cut the kidney pathology images, select target images from the cut images, and construct a sample set based on the target images and corresponding component annotation information;

[0166] A model training module, used for constructing a multi-channel coded fusion image segmentation model, and training the multi-channel coded fusion image segmentation model based on the sample set to obtain a kidney component segmentation model;

[0167] The segmentation module is used to segment the kidney pathology image to be segmented, select the target image from the segmented image and input it into the kidney component segmentation model to obtain the segmentation result corresponding to the target image; and obtain the component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

[0168] The above method embodiments and system embodiments are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process refers to the above embodiment, which will not be repeated here.

[0169] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0170] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A kidney component segmentation method based on deep learning, characterized in that: The following steps are involved: Obtaining kidney pathology images and corresponding component annotation information, cutting the kidney pathology images, screening target images from the cut images, and constructing a sample set based on the target images and corresponding component annotation information; Constructing a multi-channel coding fusion image segmentation model, and training the multi-channel coding fusion image segmentation model based on the sample set to obtain a kidney component segmentation model; The kidney pathology image to be segmented is cut, and a target image is selected from the cut image and input into the kidney component segmentation model to obtain a segmentation result corresponding to the target image; and a component segmentation map of the kidney pathology image to be segmented is obtained based on the segmentation result.

2. The kidney component segmentation method based on deep learning according to claim 1, characterized in that: The multi-channel coding fusion image segmentation model includes: The first encoding module includes a plurality of downsampling units connected in sequence, and is used to extract multi-layer semantic features from the sample image; The second encoding module includes an image filling module and a ViT encoder; the image filling module is used to fill the sample image by using weighted bilinear interpolation; the ViT encoder is used to extract visual features from the filled sample image; The middle layer is used to compress the semantic features extracted by the last downsampling unit of the first encoding module and pass the compressed semantic features to the decoding module; A decoding module, comprising a plurality of upsampling units connected in sequence, for recovering a feature map layer by layer based on the semantic features extracted by the first encoding module and the visual features extracted by the second encoding module to obtain a final feature map; The output layer is used to classify components based on the final feature map to obtain a component segmentation map; The number of down-sampling units of the first encoding module is the same as the number of up-sampling units of the encoding module and corresponds one to one; A spatial channel cross attention module is connected between the down-sampling unit and the corresponding up-sampling unit, which is used to perform attention extraction on the semantic features extracted by the down-sampling unit and pass them to the corresponding up-sampling unit.

3. The kidney component segmentation method based on deep learning according to claim 2, characterized in that: The pixel value of the interpolation point (x, y) is calculated using the following formula: Calculate the pixel values ​​of two temporary points (x, y1) and (x, y2) in the horizontal direction: f(x,y1)=α1f(x1,y1)+α2f(x2,y1) f(x,y2)=α3f(x1,y2)+α4f(x2,y2) α1＝α 1_ori / (α 1_ori +α 2_ori ) α2＝α 2_ori / (α 1_ori +α 2_ori ) α3＝α 3_ori / (α 3_ori +α 4_ori ) α4＝α 4_ori / (α 3_ori +α 4_ori ) The pixel value of the interpolation point is obtained based on the pixel values ​​of the two temporary points in the horizontal direction: Among them, f(x,y) represents the pixel value of point (x,y), f(x,y1) represents the pixel value of point (x,y1), f(x,y2) represents the pixel value of point (x,y2), f(x1,y1) represents the pixel value of point (x1,y1), f(x1,y2) represents the pixel value of point (x1,y2), f(x2,y1) represents the pixel value of point (x2,y1), f(x2,y2) represents the pixel value of point (x2,y2), α1, α2, α3 and α4 represent weights, α 1_ori , α 2_ori , α 3_ori and α 4_ori is the intermediate parameter, (x1, y1), (x2, y1), (x1, y2) and (x2, y2) represent the four original points closest to the interpolation point, w (x1,y1) 、w (x2,y1) 、w (x1,y2) and w (x2,y2) Represents the weights of the four original points.

4. The kidney component segmentation method based on deep learning according to claim 3, characterized in that: The weight of the original point is calculated using the following formula: Among them, θ t Represents the percentage of pixels of the tth component in the renal pathology image.

5. The kidney component segmentation method based on deep learning according to claim 1, characterized in that: The multi-channel coding fusion image segmentation model is trained based on the sample set using the following loss function: Loss=L DiceLoss +L SDFL Among them, L DiceLoss represents the Dice loss of the training sample, L SDFL represents the dimension-density focal loss of training samples.

6. The method for kidney component segmentation based on deep learning according to claim 5, characterized in that: The size density focal loss of the training sample is calculated using the following formula: in, represents the probability that the i-th pixel of the training sample predicted by the model belongs to the t-th component, γ t represents the penalty factor corresponding to the tth component, k t Represents the number of pixels of the tth component of the sample, represents the size probability distribution function corresponding to the t-th component, M represents the number of sample pixels, and N represents the number of component types.

7. The method for segmenting kidney components based on deep learning according to claim 6, characterized in that: The size probability distribution function of each type is obtained in the following way: Count the number of pixels Cn belonging to the tth component in each sample of the sample set t ; The number of pixels Cn t is the independent variable, and the number of pixels of the tth component in the sample is Cn t The proportion of the number of samples to the total number of samples is taken as the dependent variable, and data fitting is performed to obtain the size probability distribution function.

8. The kidney component segmentation method based on deep learning according to claim 1, characterized in that: Cut the kidney pathology images and select the target images from the cut images, including: Cutting the kidney pathology image according to a first size to obtain a plurality of cut images; The renal pathological image is converted into a grayscale image, and the average grayscale value of the annotated area is calculated according to the component annotation information corresponding to the renal pathological image; If, in the corresponding area of ​​the grayscale image, the proportion of the number of pixels whose grayscale values ​​exceed the average grayscale value to the total number of pixels of the cut image exceeds a first threshold, the cut image is the target image.

9. The kidney component segmentation method based on deep learning according to claim 8, characterized in that: The kidney pathology image is cut according to the first size to obtain a plurality of cut images, including: A sliding window of a first size is used to slide on the renal pathology image in a sequence from left to right and from top to bottom. If the renal pathology image in the current sliding window is smaller than the first size, the lower right corner of the sliding window is aligned with the lower right corner of the renal pathology image in the current sliding window; the renal pathology image in the current sliding window is used as the cutting image.

10. A kidney component segmentation system based on deep learning, characterized in that: Includes the following modules: A sample set construction module is used to obtain kidney pathology images and corresponding component annotation information, cut the kidney pathology images, select target images from the cut images, and construct a sample set based on the target images and corresponding component annotation information; A model training module, used for constructing a multi-channel coded fusion image segmentation model, and training the multi-channel coded fusion image segmentation model based on the sample set to obtain a kidney component segmentation model; The segmentation module is used to segment the kidney pathology image to be segmented, select the target image from the segmented image and input it into the kidney component segmentation model to obtain the segmentation result corresponding to the target image; and obtain the component segmentation map of the kidney pathology image to be segmented based on the segmentation result.

Images