A classification system for glomerular immunofluorescence images

Abstract

The invention discloses a classification system for glomerular immunofluorescence images, which relates to the field of medical image processing. The current high reliance on manual labor and the general lack of senior pathologists have further exacerbated the difficulty of diagnosing renal biopsy samples in primary medical institutions. The application of auxiliary systems will be able to significantly improve the efficiency and accuracy of recognition. The classification accuracy of fluorescence intensity and distribution morphology of the present invention reached 90.48% and 90.87% respectively. From the perspective of inference time, the system only takes 13 milliseconds and 12 milliseconds to process each image in fluorescence intensity classification and distribution morphology classification, which is much faster than the 11.06 seconds and 7.84 seconds required by junior pathologists and the 17.14 seconds and 18.26 seconds required by independent senior pathologists. From the perspective of Kappa scores, the conclusions given by the system are almost perfect with the labels and highly consistent with the conclusions given by independent senior pathologists (Substantial agreement, 0.6494 for fluorescence intensity classification and 0.7184 for distribution morphology).

Description

Technical Field

[0001] The present invention relates to the field of medical image processing. Background Art

[0002] Nowadays, with the widespread application of deep learning in the field of image diagnosis, some researchers have introduced deep learning-based methods into the diagnosis of renal tissue images obtained by optical microscopy. However, using only optical microscopy images is not enough to obtain a definite diagnostic result. Immunofluorescence images are also one of the most important data for the pathological diagnosis of renal diseases. Zhao et al. constructed a dataset of whole-slice renal immunofluorescence images and annotated the glomeruli in the images using bounding boxes [1]. Their study demonstrated the applicability of existing deep learning methods in evaluating glomerular immunofluorescence images, but they did not propose a detailed method for classifying glomerular descriptive indicators. Ligabue et al. fine-tuned the pre-trained ResNet-101 network to classify glomerular descriptive indicators, including appearance, distribution, location, and intensity [2]. Kitamura et al. used a convolutional neural network automatically established by computer software to extract features from immunofluorescence images of patients with diabetic nephropathy [3]. In general, most existing studies did not design dedicated image preprocessing methods and classification networks based on the characteristics of glomerular immunofluorescence images, which limited the improvement of the automatic diagnosis performance of glomerular immunofluorescence images.

[0003] [1]Zhao K, Yu J, Teng Z, et al. DGDI: A Dataset for Detecting Glomeruli onRenal Direct Immunofluorescence[C] / / 2018Digital Image Computing:Techniquesand Applications(DICTA).2018.

[0004] [2] Ligabue G, Pollastri F, Fontana F, et al. Evaluation of the Classification Accuracy of the Kidney Biopsy Direct Immunofluorescence through Convolutional Neural Networks[J]. Clinical Journal of the American Society of Nephrology, 2020, 15(10).

[0005] [3]Kitamura S,Takahashi K,Sang Y,et al.Deep learning could diagnoseiabetic nephropathy with renal pathological immunofluorescent images[J].Diagnostics,2020,10(7):466. Summary of the Invention

[0006] Existing deep learning models perform poorly on glomerular immunofluorescence images, primarily due to the following inherent challenges: 1. Glomerular immunofluorescence image classification is a fine-grained classification task. Differences in image texture and color determine the categories of descriptive metrics, and these differences are subtle and difficult to extract. 2. Fluorescence intensity is uneven, which occurs widely between different images and between different parts of the same image due to different staining procedures and deposit locations. 3. Different types of noise exist in the images. Residual dye and nonspecific staining create confounding noise in the images, which interferes with the observation of glomerular immunofluorescence images. 4. Although standardized classification rules exist for renal immunofluorescence image features, analyzing immunofluorescence images remains relatively subjective due to their complexity and variability. The current high reliance on manual labor and the widespread lack of skilled pathologists further exacerbate the diagnostic difficulties of renal biopsy samples in primary care settings. The application of auxiliary systems could significantly improve the efficiency and accuracy of recognition.

[0007] The technical solution of the present invention is a classification system for glomerular immunofluorescence images, which includes: a semantic segmentation module, an intensity equalization module, a classification module, and a visualization module;

[0008] The same glomerular immunofluorescence image is input into the semantic segmentation module and the intensity equalization module respectively. After processing, the output of the semantic segmentation module and the intensity equalization module are input into the classification module, and the classification results are visualized by the visualization module.

[0009] The semantic segmentation module performs fluorescence intensity classification, which is used to segment and remove tissue outside the glomerulus in the glomerular immunofluorescence image. The intensity equalization module performs distribution morphology classification, which is used to eliminate intensity differences between images. The classification module combines the output images of the semantic segmentation module and the intensity equalization module to classify the two images, and visualizes them after classification.

[0010] The classification module uses a hierarchical feature fusion attention network to classify the input image. The hierarchical feature fusion attention network includes the following steps: a feature extraction module, a hierarchical feature fusion attention module, and a classifier, and the classifier outputs the classification result. The feature extraction module includes: 6 convolution blocks from the first to the sixth, and the first to the fourth convolution blocks all include: a convolution layer, a batch normalization layer, and a maximum pooling layer in sequence. The filters of the convolution layers in the first to the fourth convolution blocks increase in size, and the fifth and sixth convolution blocks all include: a maximum pooling layer and a convolution layer in sequence. The first to the fourth convolution blocks are connected in sequence, the output of the second convolution block is connected to the fifth convolution block, and the output of the third convolution block is connected to the sixth convolution block. The outputs of the fourth to the sixth convolution blocks are the outputs of the feature extraction module, so the feature extraction module includes three outputs, and the feature maps of these three outputs have the same size.

[0011] The hierarchical feature fusion attention module includes: a splicing module, a branch representative feature map calculation module, a branch importance calculation module, a feature fusion attention mask calculation module, and a Hadamard product module; the splicing module splices the three outputs of the feature extraction module and divides them into three paths, wherein the first path inputs the branch representative feature map calculation module, the second path inputs the branch importance calculation module, the outputs of the branch representative feature map calculation module and the branch importance calculation module input the feature fusion attention mask calculation module, the feature fusion attention mask calculation module and the third path of the splicing module are input into the Hadamard product module together, and the output of the Hadamard product module is the output of the hierarchical feature fusion attention module.

[0012] Furthermore, the processing method of the intensity balancing module is:

[0013] Step 1: Initialize the positive pixel list V, iteration factor n = 0;

[0014] Step 2: Set the target intensity level k, adjustment factor N, threshold M, and constant α;

[0015] Step 3: Extract the red, yellow, and blue channel values of the positive pixels in the image, and add the average of the maximum and minimum values to the positive pixel list V; positive pixels refer to pixels whose red, yellow, and blue channel values are not all 0;

[0016] Step 4: Repeat step 3 for all positive pixels in the image, and increase the iteration factor n by 1 each time;

[0017] Step 5: Arrange the positive pixel list V in descending order and calculate the sum Sum of the values of the <2n / N>th to <6n / N>th elements in the list, where the operator <·> indicates rounding down;

[0018] Step 6: Calculate the global intensity level IF of the image = αSum / (<6n / N>-<2n / N>), and further calculate the intensity level adjustment coefficient k / IF of the image;

[0019] Step 7: Multiply the intensity level adjustment coefficient of the image by the values of the red, yellow, and blue channels of the pixel respectively. If the multiplication result is greater than the threshold M set in step 2, the result is set to M;

[0020] Step 8: Repeat step 7 for all pixels in the image to obtain an intensity-balanced image;

[0021] Step 9: Save the intensity-equalized image.

[0022] Furthermore, the three outputs of the feature extraction module in the classification module are sequentially passed through the channel attention module and the spatial attention module before being used as the three outputs of the feature extraction module; the channel attention module includes the following: the global average pooling layer and the convolution layer, and the output of the convolution layer is multiplied with the input of the channel attention module as the output of the channel attention module, and the spatial attention module includes the following: the cross-channel pooling layer and the Hadamard product module, and the output features of the cross-channel pooling layer and the input features are simultaneously input into the Hadamard product module, and the output of the Hadamard product module is the output of the spatial attention module.

[0023] Furthermore, the classifier in the classification module is a global average pooling layer and two fully connected layers.

[0024] Furthermore, the processing method of the branch importance calculation module in the hierarchical feature fusion attention module is:

[0025] Step 1: Calculate the importance of the i-th feature map output by the k-th branch

[0026]

[0027] Where GMP(·) represents the global maximum pooling operation, f i k Represents the i-th feature map output by the k-th branch;

[0028] Step 2: Calculate the importance representation b of each branch k :

[0029]

[0030] Where C is the number of channels output by each branch.

[0031] Furthermore, the processing method of the branch representative feature map calculation module in the hierarchical feature fusion attention module is:

[0032]

[0033] Among them, R k Represents the branch representative feature map.

[0034] Furthermore, the calculation method of the feature fusion attention mask calculation module in the hierarchical feature fusion attention module is:

[0035]

[0036] Among them, M represents the feature fusion attention mask.

[0037] The classification accuracy of the fluorescence intensity and distribution morphology of the present invention reached 90.48% and 90.87%, respectively. From the perspective of inference time, the system only takes 13 milliseconds and 12 milliseconds to process each image in the fluorescence intensity classification and distribution morphology classification, respectively, which is much faster than the 11.06 seconds and 7.84 seconds required by junior pathologists and the 17.14 seconds and 18.26 seconds required by independent senior pathologists, respectively. From the perspective of Kappa scores, the conclusions given by the system are almost completely consistent with the labels (Almost perfect), and are highly consistent with the conclusions given by independent senior pathologists (Substantialagreement, 0.6494 for fluorescence intensity classification and 0.7184 for distribution morphology). The conclusions given by junior pathologists only show moderate agreement with the labels, and also show moderate agreement with the conclusions given by independent senior pathologists (0.5903 for fluorescence intensity classification and 0.5581 for distribution morphology). The conclusions given by independent senior pathologists are highly consistent with the labels. Therefore, overall, the system's classification performance has exceeded that of junior pathologists and is comparable to that of senior pathologists, which is also reflected in the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the structural block diagram of the computer-aided diagnosis system for glomerular immunofluorescence images;

[0039] Figure 2 This is the network structure diagram of HFANet;

[0040] Figure 3 The network structure of the channel attention (CA) module and the spatial attention (SA) module;

[0041] Figure 4Visualization results of the classification of six samples in the glomerular immunofluorescence image dataset;

[0042] Figure 5 Heat maps of regions for two samples that contain more than one type of descriptive indicator;

[0043] Figure 6 A boxplot comparison of the segmentation effects of U-Net++ with and without deep supervision;

[0044] Figure 7 The semantic segmentation results of glomerular immunofluorescence images based on U-Net++ are shown in (a) and (b). The mis-segmentation in (c) and (d) is mainly located in the low-intensity area, while the mis-segmentation in (e) and (f) is mainly in the high-intensity area.

[0045] Figure 8 (a) is the confusion matrix for fluorescence intensity classification, and (b) is the confusion matrix for distribution morphology classification. DETAILED DESCRIPTION

[0046] The diagnostic system based on Hierarchical Feature Fusion Attention Network (HFANet) for glomerular immunofluorescence images is proposed. Figure 1 As shown in the figure, it consists of two subsystems: preprocessing and classification. The red and blue arrows in the figure represent the workflows for fluorescence intensity (fluorescence intensity) classification and distribution morphology (distribution morphology) classification, respectively. In the preprocessing stage, the original immunofluorescence image is first resized from 2560×1920 to 256×256. Subsequently, for fluorescence intensity classification, tissue outside the glomerulus in the image is removed using semantic segmentation. For distribution morphology classification, the semantic segmentation in the preprocessing is replaced by the proposed intensity equalization (IE) algorithm to eliminate intensity differences between images. The preprocessed images are finally fed into HFANet to obtain the classification results. With the help of the visualization algorithm Grad-CAM, the system can provide the classification basis of descriptive indicators in the form of heat maps, which is consistent with the classification basis commonly used by pathologists.

[0047] During preprocessing, due to hardware limitations, the original images were first resized from 2560×1920 to 256×256. Although image compression inevitably results in a loss of detail, this target size ensures that features relevant to both classification tasks are preserved. Subsequently, a U-Net++-based semantic segmentation algorithm and the proposed IE algorithm were used to preprocess the images for fluorescence intensity classification and distribution morphology classification, respectively.

[0048] The intensity equalization algorithm is designed to adjust the fluorescence intensity of all images in the dataset to the same level. The details of the algorithm are given in the form of pseudo code in Algorithm 1.

[0049] The IF in the algorithm roughly represents the global intensity level of the original image. The IE algorithm adjusts the IF of an image to 60 (close to 60, rather than strictly equal to 60) by adjusting each pixel value, which eliminates the intensity difference between images. The values of the parameters involved in the IE algorithm are obtained through experiments, and they ensure that all processed images have stable and clear non-intensity features. In the present invention, the IE algorithm is used to process images in segmentation data sets and distribution morphology classification data sets.

[0050]

[0051]

[0052] The proposed Hierarchical Feature Fusion Attention Network (HFANet) is as follows Figure 2 shown. Figure 2 In the middle, ⊙ indicates that the feature map of each channel is multiplied by the corresponding element of the weight vector. Represents the Hadamard product between the weight matrix and the feature map of each channel. Represents the average between the feature maps of each channel. GMP and GAP represent the global maximum pooling operation and the global average pooling operation respectively.

[0053] Since pixel values and textures do not involve complex semantic information, a shallow convolutional neural network (CNN) is more suitable for extracting such features. A lightweight CNN is proposed as the backbone of the FE module, which consists of 4 convolutional blocks. Each convolutional block consists of a 3×3 convolutional layer, a batch normalization (BN) layer, a linear rectification function (ReLU) layer, and a maximum pooling layer connected in series. The number of filters in the 4 convolutional blocks is 32, 64, 128, and 128, respectively. Compared with VGG-16, the number of convolutional layers is reduced from 12 to 4, which enables it to avoid overfitting.

[0054] Unlike VGG, HFANet uses a global average pooling (GAP) layer in its classifier instead of the commonly used fully connected (FC) layer. This significantly reduces the number of network parameters and computational complexity, and further reduces the risk of overfitting. Ultimately, the classification result is output by an FC layer activated by a softmax function.

[0055] According to experimental verification, the HFANet used in the fluorescence intensity classification task can achieve the highest classification performance by simply performing weighted fusion of branches 2 and 3 in the HFA module.

[0056] Using the visualization algorithm Grad-CAM to visualize the decision basis of HFANet (features that can derive categories) is a simple way to confirm that the system can work scientifically. The heat map generated by the system can not only provide a reference for clinicians' diagnosis, but also provide guidance in medical training. Figure 4 As shown, from left to right, six glomerular immunofluorescence image samples corresponding to levels II, III, IV, coarse granularity, fine granularity, and linearity are displayed. All six samples are from the test set and are correctly classified by the system. The level I sample is not displayed due to its extremely low fluorescence intensity. The redder areas in the heatmap indicate areas that the classification network pays more attention to during its decision-making process, while the bluer areas indicate areas that receive less attention. The yellow lines in the first row indicate the key areas that senior pathologists focus on when making category judgments. The second row shows the heatmaps obtained from the backbone network trained on the original images (compression only), while the third and fourth rows show the heatmaps obtained from CABNet and HFANet, respectively, trained on appropriately preprocessed images. The corresponding CFRR value for each heatmap is annotated below the heatmap.

[0057] From a qualitative perspective, the heatmaps provided by our system are similar to those annotated by senior pathologists, which means that HFANet makes the correct judgment based on appropriate features.

[0058] Quantitative analysis further supports the above conclusions. Inspired by a metric for measuring the stability of nonlinear systems in dynamical physics, a metric called classification feature recall ratio (CFRR) was designed. CFRR can assess the adequacy of the features focused on during model classification and is calculated as follows:

[0059]

[0060]

[0061] Here, h(·) represents the weight matrix output by Grad-CAM, and its element h(i, j) represents the degree of attention paid to the pixel at position (i, j) in the image during the classification process (ranging from 0 to 1). Δ(·) represents a binarization operation with a threshold of 0.2, where values greater than 0.2 are set to 1 and values less than 0.2 are set to 0. The purpose of the binarization operation is to filter out pixels that the model pays little attention to. A is the area of interest manually annotated by senior pathologists, and all pixel values in area A are set to 1. In fact, CFRR reveals the ratio of the area selected by both the model and the pathologist to the area selected by the pathologist.

[0062] like Figure 4 As shown in the six samples in , the CFRR values when using HFANet are higher than the CFCC values when using backbone and CABNet, which proves the advantage of HFANet in interpretability.

[0063] For images with multiple categories, such as Figure 5 As shown in the figure (the upper sample is from the fluorescence intensity classification task, and the lower sample is from the distribution morphology classification task), HFANet can also mark the corresponding regions of different categories. This heat map of regions corresponding to a certain category can help pathologists distinguish the category distribution in complex images and can also indicate the differences between different categories for students in medical training.

[0064] 1. Implementation Details

[0065] For semantic segmentation, the Adam optimizer is used and the learning rate is set to 3×10 -4 To prevent overfitting, an early stopping mechanism is introduced to monitor the binary accuracy of the model output. The hybrid segmentation loss, shown below, is chosen as the loss function for the segmentation task. It consists of a pixel-wise cross entropy loss function and a soft DICE coefficient loss function. When the iteratively obtained model achieves the highest binary accuracy on the validation set, it is saved.

[0066]

[0067] Among them, g n，c ∈G and p n，c ∈P represents the label and predicted probability of the nth pixel category c in a batch, and N represents the number of pixels in a batch.

[0068] For classification, the RMSprop optimizer was used and the categorical cross entropy was selected as the loss function. The initial value of the learning rate was set to 0.01, which was decayed by a factor of 0.8 during the iterations.

[0069] All experiments were performed using the Keras framework with TensorFlow as the backend. To further increase the number of training samples and improve the model's generalization performance, data augmentation techniques were used to generate more training samples. This technique uses various random transformations to produce plausible-looking images. All experiments were performed using an NVIDIA GeForce GTX 1080Ti with 11GB of video memory. Full implementation details are listed in Table 1.

[0070] Table 1. Implementation details of semantic segmentation and classification

[0071]

[0072] *Learning rate.

[0073] 2 Evaluation indicators

[0074] In the biomedical field, the DICE similarity coefficient (DICE) and intersection over union (IOU) are often used as metrics to evaluate segmentation performance. Given that glomerular immunofluorescence images primarily convey information through pixel intensity, pixels with higher intensities are often more important. Therefore, treating all pixels in an image equally is not objective. A new evaluation metric, DICE-IF, specifically for evaluating the visual performance of glomerular immunofluorescence image segmentation is proposed as follows:

[0075]

[0076] Where P represents the segmentation result and G represents the manually drawn label. |·| represents the counting function, which counts the number of non-zero pixels in the image. V represents a weight matrix of size 256×256, and the operator Denotes the Hadamard product. The color space of the immunofluorescence image is first resized from RGB (red, green, blue) to HSV (hue, saturation, value). The value channel in HSV space is selected as the intermediate matrix V′. The weight matrix V required for DICE-IF calculation is obtained by dividing V′ by the average of all its elements.

[0077] The elements in V represent the intensity level of the pixel at the corresponding spatial location. Because the importance of a pixel is positively correlated with its intensity level, V can also be considered as the importance matrix of pixels in the image. Compared with traditional evaluation metrics such as DICE and IoU, the addition of parameter V makes DICE-IF more efficient and appropriate in evaluating the visual quality of segmentation.

[0078] In summary, in this paper, DICE, IOU, and DICE-IF are selected as evaluation metrics for segmentation, while accuracy and Kappa score are selected as evaluation metrics for classification.

[0079] 3 Comparative Experiments and Ablation Studies

[0080] 3.1 Semantic Segmentation

[0081] FCN, U-Net, and U-Net++ are all viable models for extracting glomerular regions from glomerular immunofluorescence images. As shown in Table 2, four models were trained on the segmentation dataset (two models were trained depending on whether U-Net++ included deep supervision). To ensure fairness, ResNet-101 was selected as the backbone network for all segmentation models. The segmentation dataset included both datasets processed with and without the IE algorithm.

[0082] Table 2 Comparison of segmentation results of different semantic segmentation models

[0083]

[0084] 1 Training time per iteration (seconds / iteration).

[0085] 2 The number of parameters.

[0086] 3 Video memory usage during model training

[0087] Test results show that when the dataset was processed with the IE algorithm, U-Net++ with a deep supervision structure achieved the best segmentation performance, with average DICE, IOU, and DICE-IF values of 93.47%, 88.09%, and 95.10%, respectively. Table 2 shows that compared with other classic segmentation models, U-Net++ is the most suitable model for the glomerular immunofluorescence segmentation task. Furthermore, the proposed IE algorithm can enhance image features through simple pixel operations, significantly improving segmentation performance with a slight increase in computational effort. Due to the increased number of parameters, U-Net++ incurs more training time per iteration than U-Net and FCN. The addition of deep supervision slightly reduces U-Net++'s training time, but significantly increases its video memory usage during training due to its multiple outputs.

[0088] Figure 6 The results show that the mean and upper quartile of DICE, IOU, and DICE-IF all show consistent improvement after the deep supervision structure is added, indicating that deep supervision is helpful for the segmentation of glomerular immunofluorescence images. The outlier values of the three indicators are also improved after the use of deep supervision, which further demonstrates that deep supervision has a positive effect on the segmentation of difficult-to-segment samples.

[0089] Figure 7 Six typical segmentation results are given in Figure 3, which illustrate the flexibility and objectivity of DICE-IF in evaluating the visualization effect of segmentation. In fact, poor segmentation results can be divided into two types. One is that the segmentation error occurs in the low-intensity area (negative area), e.g. Figure 7 (c) and (d). Because immune complex deposit-negative regions are generally not used for diagnosing kidney disease, errors in these regions are tolerable. For the two samples corresponding to this type of segmentation error, the DICE-IF value is improved compared to the DICE and IOU values.

[0090] Besides, another type of segmentation error is segmentation error occurring in high intensity areas, e.g. Figure 7 (e) and (f). This type of error manifests as incomplete glomerular extraction and the introduction of intensity interference. Because both pathologists and classification CNNs are sensitive to high-intensity regions, this type of error cannot be tolerated. For the two samples corresponding to this type of segmentation error, the DICE-IF value is lower than the DICE and IOU values.

[0091] As for the perfect segmentation sample Figure 7 (a) and (b), DICE-IF is basically consistent with traditional DICE and IOU, which shows that DICE-IF also inherits the characteristics of traditional indicators.

[0092] 3.2 Classification

[0093] Fluorescence intensity classification and distribution morphology classification are the last but most core steps to realize the computer-aided diagnosis system of glomerular immunofluorescence images.

[0094] Table 3 Classification performance of different feature fusion parameters

[0095]

[0096] 1 Feature map size during fusion

[0097] 2 Number of branches

[0098] As shown in Table 3, the feature map size and the number of branches during fusion are set to different values to find the optimal parameters of HFANet. When the fusion size is larger than the output size of a certain convolutional block, the maximum pooling layer in the corresponding branch will be replaced by an upsampling layer. When the number of branches is set to 2, only Figure 2 The feature maps output by branches 2 and 3 in the image are fused. Experimental results show that for fluorescence intensity classification and distribution morphology classification, the fusion size should be set to 16×16, and the number of branches should be set to 2 and 3, respectively.

[0099] Table 4. Ablation study of feature fusion and HFA modules. The classifier is omitted in the table.

[0100]

[0101] 1 The backbone of the FE module, which does not include branch structures

[0102] 2 FE modules that include both trunk and branch structures

[0103] 3 Hierarchical feature fusion attention (HFA) model

[0104] piece

[0105] The ablation study shown in Table 4 reveals the significant role of feature fusion and the HFA module in improving classification performance. The parameters of the FE module were set based on the results presented in Table 3. The classification performance achieved by directly concatenating the branch outputs significantly improved compared to using only the backbone network. However, replacing direct concatenation with the HFA module further improved classification performance. Because more branches are used in distribution morphology classification, the number of parameters and training time overhead are greater than those for fluorescence intensity classification.

[0106] Table 5 Ablation study results of CA module and SA module in HFANet

[0107]

[0108] 1 That is, the complete HFANet

[0109] In HFANet, the CA and SA modules are placed at the end of the relevant branches to enhance feature extraction. The ablation results in Table 5 reveal the importance of CA and SA in HFANet. Specifically, unless feature maps from different layers are first weighted by the CA or SA modules in the branches, using the HFA module alone will not lead to high classification performance. However, adding the CA and SA modules in series at the end of the relevant branches is undoubtedly the best option.

[0110] VGG-16, MobileNetV2, ResNet-50, DenseNet-121, CABNet, and HFANet are all candidates for classification. Through ablation studies of module combinations and comparative experiments between these architectures, we analyze the effectiveness of each module and explore the most suitable network architecture for the two classification tasks. The proposed lightweight CNN is referred to as the baseline and serves as the backbone for CABNet and HFANet.

[0111] Table 6 Ablation study results of fluorescence intensity and distribution morphology classification

[0112]

[0113] 1 The image is processed by semantic segmentation based on U-Net++

[0114] The test results for fluorescence intensity classification are shown in the upper half of Table 6. HFANet achieved the highest accuracy and Kappa score on the segmented dataset, at 90.48% and 0.8730, respectively. Semantic segmentation based on U-Net++ eliminates interference from stained tissue outside the glomeruli, focusing the field of view on the glomeruli; the CA and SA modules placed in the relevant branches emphasize key features in feature maps at different levels, while the HFA module further guides the network to focus on levels and regions that are beneficial for classification. The confusion matrix is shown in Figure 1. Figure 8 As shown in (a), all prediction errors occur at adjacent fluorescence levels. Clinically, even experienced pathologists hesitate between adjacent fluorescence levels. In reality, such subtle deviations do not affect the diagnosis of kidney disease, a fact that holds true for both the system and the pathologists.

[0115] The test results of distribution morphology classification are shown in the lower half of Table 6. Using the dataset processed only by the IE algorithm, the accuracy and Kappa score achieved by HFANet were 90.87% and 0.8525, respectively, which is the highest performance achieved. The ablation study under this task shows that the accuracy of the baseline is 74.78%. When HFANet is used alone and the IE algorithm is added, the accuracy is increased to 81.30% and 86.96%, respectively. The simultaneous use of HFANet and IE algorithms further increases the accuracy to 90.87%. However, if semantic segmentation is further added on the basis of the simultaneous use of HFANet and IE algorithms, the classification accuracy will drop from 90.87% to 83.91%. Therefore, HFANet combined with the IE algorithm has a positive effect on improving the performance of distribution morphology classification, but this is not the case for semantic segmentation. The confusion matrix is shown in Figure 2. Figure 8 As shown in (b), misclassification mainly occurs between fine-grained and line-like images, while coarse-grained images are rarely misclassified.

[0116] Table 7 Comparison of classification performance of HFANet and commonly used classification networks on glomerular immunofluorescence image dataset

[0117]

[0118] As shown in Table 7, when combined with appropriate preprocessing methods, HFANet achieves higher classification performance than other networks. Compared to VGG-16, MobileNetV2, ResNet-50, DenseNet-121, and CABNet, HFANet achieves the best classification performance despite its extremely small size. CABNet sequentially connects a CA module, a SA module, and a category attention module to its backbone to improve the model's classification performance and interpretability. Compared to CABNet, HFANet achieves better classification performance and interpretability despite having slightly more trainable parameters.

Claims

1. A classification system for glomerular immunofluorescence images, comprising: Semantic segmentation module, intensity equalization module, classification module, and visualization module; The same glomerular immunofluorescence image is input into the semantic segmentation module and the intensity equalization module respectively. After processing, the output of the semantic segmentation module and the intensity equalization module are input into the classification module, and the classification results are visualized by the visualization module. The semantic segmentation module performs fluorescence intensity classification, which is used to segment and remove tissues other than glomeruli in glomerular immunofluorescence images. The intensity equalization module performs distribution morphology classification, which is used to eliminate intensity differences between images. The classification module combines the output images of the semantic segmentation module and the intensity equalization module to classify the two images and visualize them after classification; The classification module uses a hierarchical feature fusion attention network to classify the input image. The hierarchical feature fusion attention network includes the image passing through: a feature extraction module, a hierarchical feature fusion attention module, and a classifier, and the classifier outputs the classification result; The feature extraction module includes: 6 convolution blocks from the first to the sixth, the first to the fourth convolution blocks all include: a convolution layer, a batch normalization layer, and a maximum pooling layer in sequence, the filters of the convolution layers in the first to the fourth convolution blocks are increased in sequence, and the fifth and sixth convolution blocks all include: a maximum pooling layer and a convolution layer in sequence; the first to the fourth convolution blocks are connected in sequence, the output of the second convolution block is connected to the fifth convolution block, and the output of the third convolution block is connected to the sixth convolution block; the outputs of the fourth to the sixth convolution blocks are the outputs of the feature extraction module, so the feature extraction module includes three outputs, and the feature maps of these three outputs have the same size; The hierarchical feature fusion attention module includes: a splicing module, a branch representative feature map calculation module, a branch importance calculation module, a feature fusion attention mask calculation module, and a Hadamard product module; the splicing module splices the three outputs of the feature extraction module and divides them into three paths, wherein the first path inputs the branch representative feature map calculation module, the second path inputs the branch importance calculation module, the outputs of the branch representative feature map calculation module and the branch importance calculation module input the feature fusion attention mask calculation module, the feature fusion attention mask calculation module and the third path of the splicing module are input into the Hadamard product module together, and the output of the Hadamard product module is the output of the hierarchical feature fusion attention module.

2. A classification system for glomerular immunofluorescence images according to claim 1, characterized in that: The processing method of the intensity balancing module is: Step 1: Initialize the positive pixel list V, iteration factor n = 0; Step 2: Set the target intensity level k, adjustment factor N, threshold M, and constant α; Step 3: Extract the red, yellow, and blue channel values of the positive pixels in the image, and add the average of the maximum and minimum values to the positive pixel list V; Positive pixels represent pixels whose red, yellow, and blue channel values are not all 0; Step 4: Repeat step 3 for all positive pixels in the image, and increase the iteration factor n by 1 each time; Step 5: Arrange the positive pixel list V in descending order and calculate the sum Sum of the values of the <2n / N>th to <6n / N>th elements in the list, where the operator <·> indicates rounding down; Step 6: Calculate the global intensity level IF of the image = αSum / (<6n / N>-<2n / N>), and further calculate the intensity level adjustment coefficient k / IF of the image; Step 7: Multiply the intensity level adjustment coefficient of the image by the values of the red, yellow, and blue channels of the pixel respectively. If the multiplication result is greater than the threshold M set in step 2, the result is set to M; Step 8: Repeat step 7 for all pixels in the image to obtain an intensity-balanced image; Step 9: Save the intensity-equalized image.

3. The classification system for glomerular immunofluorescence images according to claim 1, characterized in that: The three outputs of the feature extraction module in the classification module are sequentially passed through the channel attention module and the spatial attention module before being used as the three outputs of the feature extraction module; the channel attention module includes the following: the global average pooling layer and the convolution layer, and the output of the convolution layer is multiplied with the input of the channel attention module as the output of the channel attention module; the spatial attention module includes the following: the cross-channel pooling layer and the Hadamard product module, and the output features of the cross-channel pooling layer and the input features are simultaneously input into the Hadamard product module, and the output of the Hadamard product module is the output of the spatial attention module.

4. The classification system for glomerular immunofluorescence images according to claim 1, wherein: The classifier in the classification module is a global average pooling layer and two fully connected layers.

5. The classification system for glomerular immunofluorescence images according to claim 1, characterized in that: The processing method of the branch importance calculation module in the hierarchical feature fusion attention module is: Step 1: Calculate the importance of the i-th feature map output by the k-th branch Where GMP(·) represents the global maximum pooling operation, f i k Represents the i-th feature map output by the k-th branch; Step 2: Calculate the importance representation b of each branch k : Where C is the number of channels output by each branch.

6. The classification system for glomerular immunofluorescence images according to claim 5, characterized in that: The processing method of the branch representative feature map calculation module in the hierarchical feature fusion attention module is: Among them, R k Represents the branch representative feature map.

7. A classification system for glomerular immunofluorescence images according to claim 6, characterized in that: The calculation method of the feature fusion attention mask calculation module in the hierarchical feature fusion attention module is: Among them, M represents the feature fusion attention mask.

Images