Multi-scale attention fusion network-based ulcerative colitis image classification method

Abstract

The invention discloses an ulcerative colitis image classification method based on a multi-scale attention fusion network, belongs to the technical field of image information mining and classification, and combines a staged feature extraction backbone network, a multi-scale attention fusion module (SCHL) and a classification module. The ulcerative colitis lesion image classification method has the advantages that significant advantages are shown in ulcerative colitis lesion image automatic classification tasks, an OfficientNet backbone network has staged feature extraction capacity, multi-level information from shallow texture to deep semantics in lesion images can be effectively captured, an SCHL module performs modeling by fusing channel attention, space attention and level high-frequency information, and the ulcerative colitis lesion image classification efficiency is improved. The model is guided to focus on a key lesion area, background interference is inhibited, feature discrimination is enhanced, the recognition ability of the model to a fine-grained lesion form and the generalization ability of the model to a complex clinical image are remarkably improved, and higher accuracy and robustness are achieved in refined classification of ulcerative colitis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image information mining and classification, and particularly relates to an ulcerative colitis image classification method based on a multi-scale attention fusion network. BACKGROUND

[0002] Ulcerative colitis (UC) is a chronic nonspecific inflammatory disease mainly involving the colonic mucosa, and is one of the important subtypes of inflammatory bowel disease. The pathogenesis of UC is relatively complex, and is generally considered to be related to multiple factors such as genetic factors, environmental influences, and abnormal immune systems. UC has the characteristics of repeated course and high heterogeneity, and often presents alternating clinical manifestations of active and remission periods. The mucosal damage can continuously progress, and seriously affects the quality of life of patients.

[0003] At present, the disease state of UC is mainly evaluated by an endoscopic evaluation system in the clinic, such as the Mayo Endoscopic Subscore (MES) and the Ulcerative Colitis Endoscopic Index of Severity (UCEIS). The MES uses a four-level grading system (0 to 3) to evaluate the mucosal state, which can reflect the severity of the disease and predict the treatment response. The UCEIS quantitatively analyzes the lesions through three dimensions of blood vessel morphology, bleeding, and erosion / ulceration, and has higher scoring consistency. However, these evaluation systems still have the following shortcomings: 1) They rely on artificial judgment and have strong subjectivity, resulting in low inter-observer consistency; 2) The whole colonoscopy is time-consuming and complex; and 3) The examination cost is high, which increases the burden of patients. Therefore, how to accurately, efficiently and low-costly evaluate the disease state of UC has become a technical problem to be solved.

[0004] In recent years, deep learning (DL) algorithms have shown significant effects in the diagnosis and classification of gastrointestinal (GI) diseases. Many studies have shown that convolutional neural network (CNN)-based models have the potential to classify colorectal polyps. For example, Bernal et al. evaluated various CNN-based and hybrid frameworks for colorectal polyp detection and found that these methods significantly improved detection accuracy and clinical applicability. Zhang et al. used transfer learning and deep convolutional neural networks (DCNNs) to classify colonoscopy images into hyperplastic, adenomatous polyps, and normal tissue. Ribeiro et al. addressed the class imbalance and data scarcity problem using transfer learning and extracted more images from the CC-iScan database for colorectal polyp classification. Byrne et al. developed a real-time framework based on deep learning that not only classified polyp subtypes during colonoscopy but also extended to wireless capsule endoscopy, improving classification efficiency. Becker et al. proposed a CNN-based classification method for evaluating ulcerative colitis (UC), but due to its shallow structure and limited performance evaluation, it failed to effectively distinguish subtle or multi-scale UC lesions. Lee used three pre-trained CNN architectures—ResNet, VGGNet, and InceptionV3—to classify normal ulcers and malignant ulcers. Turan and Durmus proposed the UC-NfNet model, which used colonoscopy images to evaluate UC and outperformed individual gastroenterologists.Malik et al. developed a CNN-RNN hybrid model for wireless capsule endoscopy (WCE) image classification, but there are still challenges in feature extraction and generalization. Shah et al. introduced a ViT-CNN hybrid model for ulcerative colitis detection, successfully solving the class imbalance problem, but still facing certain difficulties in complex gastrointestinal disease cases. Guo et al. proposed a self-supervised framework C-Mixup for gastrointestinal image classification and achieved excellent performance on the HyperKvasir dataset. Diamantis et al. proposed a multi-scale architecture Look-Behind FCN (LB-FCN) for endoscopic abnormality detection. In addition, DeiT enhances the visual transformer by distilling labeled transfer, and the features extracted from CNN are further optimized. However, existing methods still have many shortcomings in the intelligent analysis of gastrointestinal diseases, especially ulcerative colitis (UC) lesions. On the one hand, these methods are highly dependent on domain expertise and lack the ability to generalize to diverse datasets, making it difficult to capture complex hierarchical features. On the other hand, their dependence on large annotated datasets significantly limits their scalability in real clinical environments. In addition, although some hybrid models have achieved certain success in label relationship modeling and classification performance, they still lack sufficient integration of multi-scale attention mechanisms and high-frequency detail feature extraction, affecting the ability to recognize subtle lesion patterns. More critically, existing loss functions are designed for general visual tasks and do not effectively adapt to the label ambiguity, class imbalance, and subjective differences in manual interpretation of UC lesions, thereby restricting the practicality and accuracy of the model. To address the above problems, an endoscopic image grading method based on a multi-scale attention fusion network (MSAFN) is proposed. SUMMARY

[0005] To solve the technical problems raised in the background art, the present application provides an ulcerative colitis image classification method based on a multi-scale attention fusion network, which adopts the following technical solutions:

[0006] Step 1: Obtain the ulcerative colitis image dataset and divide it into training set, test set and validation set;

[0007] The ulcerative colitis (UC) endoscopic image classification task used in this scheme involves two datasets, LIMUC and Hyper-Kvasir. The LIMUC dataset was collected by the Medical Faculty of Marmara University in Turkey and contains colonoscopy images of 564 patients from 2011 to 2019, a total of 11276 images, with an image resolution of 352x288. All images were blindly reviewed by three experienced gastroenterologists and the final label was determined by majority voting according to the Mayo Endoscopic Score (MES) standard. The score reflects the different severity of UC inflammation from 0 to 3.

[0008] The Hyper-Kvasir dataset was built by the University Hospital of Oslo in cooperation with the University of Simon Fraser, and the data collection time spanned from 2015 to 2019, containing 110,079 gastrointestinal images, of which 10,662 had artificial labels, covering classification images of different grades (including intermediate grades) of UC Mayo scoring;

[0009] Step 2: data preprocessing;

[0010] The training set, test set and validation set are divided according to the ratio of 7:2:1, the training set is enhanced by random horizontal flip, vertical flip and standardization, etc., to improve the generalization ability of the model, combined with the characteristics of ulcerative colitis images, image smoothing, denoising and morphological transformation enhancement operations are introduced to improve the robustness of the model to the diversity of the lesion structure;

[0011] In the above scheme, the images from different data sources are uniformly standardized: the image size of the LIMUC dataset is adjusted to 352x288, and the image of the HyperKvasir dataset is unified to 512x512 resolution, all images are in three-channel RGB format, and the corresponding pathological classification label Y∈{0,1,…,K} is covered, including erosion, inflammatory structural changes and ulcer depth classification, etc. To improve the robustness of the model, a multi-level data enhancement strategy is adopted: spatial enhancement includes random horizontal flip (probability 0.5), rotation (±30°), scaling and cropping (scale 0.8-1.2); light enhancement covers brightness adjustment (±20%), contrast change (±15%) and Gaussian noise injection (standard deviation 0.05). All images are normalized by ImageNet mean ([0.485, 0.456, 0.406]) and standard deviation ([0.229, 0.224, 0.225]). The dataset is divided into training set, validation set and test set according to the ratio of 7:2:1, and 5-fold cross-validation is adopted for small sample data to ensure the stability of model training;

[0012] Step 3: establish and initialize the MSAFN backbone network typing model, which includes the EfficientNet backbone network, the multi-scale feature fusion module (SCHL) and the classification module;

[0013] According to the classification task requirements of the ulcerative colitis (UC) image, the MBConv module and the channel attention mechanism (SE) of each stage of the EfficientNet backbone network are constructed and initialized, which is used for multi-stage feature extraction of the input image, and different levels of feature maps from shallow to deep and from texture details to semantic structure are obtained layer by layer. The feature maps output by each stage are aligned to the same spatial resolution by bilinear interpolation, and are spliced in the channel dimension to form a fused multi-scale global feature representation. Convolution operation is introduced for channel compression to reduce redundancy and compress feature dimension;

[0014] The compressed fused features are input into the SCHL multi-scale attention fusion module. The SCHL module combines channel attention, spatial attention and high-frequency feature modeling mechanism, and gradually guides the model to focus on the key lesion area in the image while effectively suppressing background interference. This module can enhance the response ability of typical and atypical lesions at different scales, and improve the expression and discrimination ability of the model for UC disease characteristics;

[0015] The attention-enhanced high-level features are input into the classification module to complete the final prediction output of the Mayo grading four-classification or UCG six-classification task.

[0016] In the above scheme, first, the backbone network is initialized based on the EfficientNet architecture, and multi-scale features of the image are extracted step by step in four stages. Stage 1 extracts shallow edge and texture information to generate a feature map with a resolution of 128x128. Stage 2 further captures medium-scale lesion morphology features. Stage 3 and stage 4 extract deep semantic information respectively to generate high-level semantic features with a resolution of 16x16. Each stage optimizes feature expression through the MBConv module (including depth separable convolution and channel attention mechanism) to dynamically adjust the importance of channels and enhance the saliency of key lesion areas.

[0017] To fuse multi-scale information, the first three stage features are adjusted to the same resolution (16x16) as the fourth stage by bilinear interpolation, then channel splicing and compression are performed, and the spliced multi-scale features are reduced in dimension by 1x1 convolution to generate lightweight fusion features to reduce computational complexity and retain key information. Further optimization of feature expression is performed by a channel-space-hierarchy attention fusion module (SCHL): a channel attention module extracts global context information by global average pooling and global maximum pooling to generate channel weights to enhance the response of key channels; a high-frequency feature extraction branch uses a double-branch 3x3 convolution to capture lesion edge and detail information, and fuses it into a high-frequency feature map through learnable parameters, which is multiplied element-wise with the channel attention feature to enhance local details; a spatial attention module generates spatial weights based on global pooling and convolution operations to suppress background noise and highlight the spatial distribution of the lesion area. Finally, the original input, channel weighted features and spatial weighted features are fused in a learnable proportion, and a 3x3 convolution is further used to refine the enhanced multi-scale features.

[0018] The classification head part uses global average pooling (GAP) to compress the spatial dimension, and generates class probability distribution through fully connected layer. In the training process, Adam optimizer is used combined with adaptive learning rate adjustment strategy to dynamically balance the convergence speed and stability of the model. Through multi-scale feature fusion and attention enhancement mechanism, MSAFN can effectively distinguish the morphological differences of UC lesions (such as erosion, polypoid hyperplasia and inflammatory structural changes), and still maintain high classification accuracy in complex background and high inter-class similarity scenarios. In the model initialization process, the parameters of each module are optimized and configured according to the characteristics of the input data to ensure the comprehensiveness of feature extraction and the reliability of classification decision.

[0019] Specifically, the initialization of the EfficientNet backbone network fully utilizes its hierarchical feature extraction capability, gradually transitioning from shallow texture to deep semantic expression; the SCHL module significantly improves the focusing ability of the model on the lesion area through the synergistic effect of channel and spatial attention, combined with high-frequency detail enhancement strategy; the lightweight design of the classification head ensures the inference efficiency while avoiding overfitting problems by integrating spatial information through global average pooling. The overall architecture realizes the adaptive optimization of multi-scale features and attention weights through end-to-end training, providing an efficient and reliable solution for accurate classification of UC images.

[0020] Step 4: Set the optimizer and loss function;

[0021] Select the appropriate optimizer and loss function according to the complexity of the model and the requirements of the ulcerative colitis lesion image classification task;

[0022] The optimizer adopts AdamW optimization strategy and combines an exponential decay learning rate scheduler Learning RateScheduler to optimize the model parameters; the learning rate adjustment formula is as follows:

[0023] η t =η0·exp(-λ·t)

[0024] Wherein, η t is the learning rate of the tth round, η0is the initial learning rate, and λ is the decay rate parameter;

[0025] The loss function is a progressive loss function PIloss, aiming at the label ambiguity and lesion heterogeneity problems in ulcerative colitis image classification, PIloss combines polynomial loss and inter-class feature separation strategy, and is regulated by time weight, and its form is:

[0026]

[0027] Wherein, y c and are the real label and the prediction probability respectively, C is the total number of classes; α and γ are the adjustment parameters of the polynomial term; f is the sample feature vector, μ y is the corresponding class center feature, and λ(t) is the weight function changing with the training round t, which is used to smoothly control the dominance of different loss components in the training process to ensure the accurate detection of minority classes in medical data, solve the label ambiguity and lesion heterogeneity problems in UC classification, and combine the dynamic weighting mechanism to adaptively balance feature learning at different training stages, PIloss gradually adjusts its focus, emphasizing fine-grained feature learning in the early stage, and gradually shifting to enhance inter-class separability and intra-class compactness as the model matures, PolyLoss is used to optimize the classification decision boundary in the early training stage, and then encourages InterClassSeparationLoss to smoothly transition to improve the classification separation degree, this adaptive strategy can effectively capture the complexity and heterogeneity of UC lesions, thereby improving the classification accuracy and robustness.

[0028] In the model training process, the following dynamic weight mechanism is defined, λ poly (i): the weight coefficient of PolyLoss at the ith moment, which is large at the beginning and gradually decreases later. λ ics (i): the weight coefficient of InterClassSeparationLoss at the ith moment, which is small at the beginning and gradually increases later, and the total loss of PIloss can be expressed as:

[0029]

[0030] Wherein, is PolyLoss, i.e.:

[0031]

[0032] wherein, is the standard cross-entropy loss, P true is the predicted probability of the true class, and ∈ is the coefficient controlling the polynomial term.

[0033] is InterClassSeparationLoss, i.e.:

[0034]

[0035] wherein, f i is the feature of the i-th sample, is the center of the class to which the i-th sample belongs, and m is a preset class center distance threshold. PIloss dynamically adjusts the weights of the two losses, so that the model can be optimized flexibly at different training stages. In the early stage of training, the learning of difficult-to-classify samples is focused on, and as the training progresses, the class separation and intra-class compactness are gradually enhanced, thereby improving the overall classification accuracy of the model. This gradual transition strategy effectively avoids the local optimization problem of a single loss function during training, improves the fine-grained classification ability and discriminative ability of the model, and ultimately achieves higher classification accuracy. We can quantify the accuracy of the model's prediction;

[0036] Step 5: Model training

[0037] The images in the training set are classified according to the lesion category and annotation, and the EfficientNet backbone network is used to extract structural and semantic features at different levels, and the multi-scale feature maps are uniformly spatially aligned and channel compressed; a multi-scale attention fusion module SCHL is introduced, which performs channel attention, spatial attention and high-frequency feature modeling operations on the fused feature maps, enhances the expression ability of key lesion areas and suppresses background interference; through high-frequency structural information, the perception effect of edge details and small lesions is improved, effectively depicting and accurately classifying typical and atypical lesion morphologies in images, significantly improving the adaptability and discriminative ability of the model to complex clinical images;

[0038] In the above technical solution, the goal of the training stage is to use the existing labeled images and input ulcerative colitis image data to train the MSAFN model, so that it can learn effective features and make accurate predictions. The trained MSAFN model is saved for further preparation for subsequent testing;

[0039] MSAFN model test stage

[0040] The test set samples x test are input into the trained MSAFN model. This stage only performs forward propagation, without backward propagation and parameter updating;

[0041] The model outputs the predicted feature vector y for each test sample predict Through comparison of the feature vector with the classification threshold, the model generates the corresponding predicted label;

[0042] These predicted labels can be compared with the true labels y test of the test set to evaluate the performance of the model.

[0043] Step six: model saving and output

[0044] If the performance of the validation set in the current training round is better than the optimal result in the previous round, the current model parameters are saved, and the relevant performance indicators are recorded to ensure that the final best model parameters can be retained. After training is completed, the final parameters of the model and the best accuracy on the validation set are output;

[0045] In the above scheme, the training loop is from i = 1 to N, and this process is repeated until the model parameters reach the optimal state. The training loop is the core of the model training process. Through continuous iteration and optimization, the model can gradually learn the internal rules and characteristics of the data. In each iteration, the model undergoes a complete forward propagation and backward propagation process, gradually adjusting its internal parameters and improving prediction accuracy;

[0046] When the training loop is repeated N times, the model completes training, and the parameters of the MSAFN model have been adjusted to the optimal state. The prediction performance of the model reaches the expected target, and the optimal model is saved.

[0047] Step seven: model typing

[0048] For endoscopic image data of ulcerative colitis patients, an automatic lesion recognition and typing model is constructed to perform multi-grade classification prediction on the colon lesion images of each patient. The model output can be used for system terminal display and auxiliary diagnosis.

[0049] Further: the classification task includes two types of settings: Mayo grading four classification and ulcerative colitis index six classification, which are used to reflect the clinical activity degree and microscopic pathological feature evolution process of ulcerative colitis lesions. The Mayo grading classification labels are 0, 1, 2, and 3, respectively, corresponding to normal mucosa, mild activity, moderate activity, and severe activity. The ulcerative colitis typing labels are ulcerative colitis 0-1, ulcerative colitis 1, ulcerative colitis 1-2, ulcerative colitis 2, ulcerative colitis 2-3, and ulcerative colitis 3, which are used to refine the evolution state of UC lesions in the pathological continuum.

[0050] Further: the model training process includes image normalization, data enhancement, multi-scale feature extraction, attention module integration and classification module optimization, output prediction label and used for clinical auxiliary judgment. In the classification task setting, for four classification: if the model output label is 0, it represents Mayo 0 level; the output is 1, it represents Mayo 1 level; the output is 2, it represents Mayo 2 level; the output is 3, it represents Mayo 3 level; if it is a six classification task, the model output label is 0, it represents ulcerative colitis 0-1; the output is 1, it represents ulcerative colitis 1; the output is 2, it represents ulcerative colitis 1-2; the output is 3, it represents ulcerative colitis 2, label 4 corresponds to ulcerative colitis 2-3, and label 5 corresponds to ulcerative colitis 3.

[0051] The present application has the following advantages:

[0052] The present application combines the phased feature extraction backbone network, the multi-scale attention fusion module (SCHL) and the classification module, and has significant advantages in the automatic grading task of ulcerative colitis lesion images. The EfficientNet backbone network has phased feature extraction capability and can effectively capture multi-level information from shallow texture to deep semantics in the lesion image. The SCHL module models by fusing channel attention, spatial attention and high-frequency information at different levels, guides the model to focus on the key lesion area, suppresses the background interference, and enhances the feature discriminability. This strategy significantly improves the recognition ability of the model to fine-grained lesion morphology and the generalization ability to complex clinical images, so as to realize higher accuracy and robustness in the fine classification of ulcerative colitis. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is the multi-scale attention fusion network MSAFN model of the present application;

[0054] Figure 2 is the cross-scale space-level module (SCHL) of the present application;

[0055] Figure 3 is the comparison result confusion matrix graph of the present application on the LIMUC dataset;

[0056] Figure 4 is the comparison result ROC curve graph of the present application on the LIMUC dataset;

[0057] Figure 5 is the comparison result ROC curve graph of the present application on the Hyper Kvasir dataset;

[0058] Figure 6 is the Grad-CAM visualization graph of the present application on the LIMUC dataset; DETAILED DESCRIPTION

[0059] The specific technical solutions of the present application are further described below to facilitate further understanding of the present application by those skilled in the art without limiting the rights thereof.

[0060] Embodiment 1, please refer to Figures 1-6 A method for classifying ulcerative colitis images based on a multi-scale attention fusion network:

[0061] Step 1: Obtain the ulcerative colitis image dataset and divide it into a training set and a test set;

[0062] The ulcerative colitis (UC) endoscopic image classification task used in this solution involves two datasets, LIMUC and Hyper-Kvasir. The LIMUC dataset was collected by the Medical Faculty of Marmara University in Turkey and contains colonoscopy images of 564 patients from 2011 to 2019, a total of 11276 images, with an image resolution of 352x288. All images were reviewed by three experienced gastroenterologists and the final label was determined by majority voting according to the Mayo Endoscopic Score (MES) standard. The score reflects the different severity of UC inflammation from 0 to 3.

[0063] The Hyper-Kvasir dataset was constructed by the University Hospital of Oslo and Simon Fraser University. The data was collected from 2015 to 2019 and contains 110,079 gastrointestinal images, of which 10,662 have artificial labels. The dataset covers UC Mayo score classification images of different levels (including intermediate levels).

[0064] Step 2: Data preprocessing;

[0065] The training set and the validation set are divided according to the ratio of 8:2. The training set is enhanced by random horizontal flipping, vertical flipping, and standardization to improve the generalization ability of the model. Combined with the features of ulcerative colitis images, image smoothing, denoising, and morphological transformation enhancement operations are introduced to improve the robustness of the model to lesion diversity structure.

[0066] In the above scheme, the images of different data sources are uniformly standardized: the image size of the LIMUC dataset is adjusted to 352x288, the image of the HyperKvasir dataset is unified to 512x512 resolution, all images are in three-channel RGB format, and correspond to the pathological classification label Y∈{0, 1,…, K}, covering erosion, inflammatory structural changes, and ulcer depth classification, etc. To improve the robustness of the model, a multi-level data augmentation strategy is adopted: spatial enhancement includes random horizontal flip (probability 0.5), rotation (±30°), scaling and cropping (scale 0.8-1.2); photometric enhancement includes brightness adjustment (±20%), contrast change (±15%) and Gaussian noise injection (standard deviation 0.05). All images are normalized by ImageNet mean ([0.485, 0.456, 0.406]) and standard deviation ([0.229, 0.224, 0.225]). The dataset is divided into training set, validation set and test set in the ratio of 7:2:1, and 5-fold cross-validation is adopted for small sample data to ensure the stability of model training.

[0067] Step 3: Establish and initialize the MSAFN backbone network typing model, which includes an EfficientNet backbone network, a multi-scale feature fusion module (SCHL) and a classification module;

[0068] According to the typing classification task requirements of ulcerative colitis (UC) images, the MBConv module and channel attention mechanism (SE) of each stage of the EfficientNet backbone network are constructed and initialized, which are used for multi-stage feature extraction of input images, and different levels of feature maps from shallow to deep and from texture details to semantic structure are obtained layer by layer. The feature maps output by each stage are aligned to the same spatial resolution by bilinear interpolation, and are spliced in the channel dimension to form a fused multi-scale global feature representation. Convolution operation is introduced for channel compression to reduce redundancy and compress feature dimension;

[0069] The compressed fused features are input into the SCHL multi-scale attention fusion module. The SCHL module combines channel attention, spatial attention and high-frequency feature modeling mechanism, and gradually guides the model to focus on the key lesion area in the image while effectively suppressing background interference. This module can enhance the response ability of typical and atypical lesions at different scales, and improve the expression and discrimination ability of the model for UC disease features;

[0070] The attention-enhanced high-level features are input into the classification module to complete the final prediction output of the Mayo grading four-classification or UCG six-classification task;

[0071] Firstly, the backbone network is initialized based on the EfficientNet architecture, and multi-scale features of the image are extracted step by step in four stages. In stage 1, shallow edge and texture information is extracted to generate a feature map with a resolution of 128x128. In stage 2, medium-scale lesion morphology features are further captured. In stages 3 and 4, deep semantic information is extracted, respectively, to generate high-level semantic features with a resolution of 16x16. In each stage, the feature expression is optimized through the MBConv module (including depth separable convolution and channel attention mechanism), and the channel importance is dynamically adjusted to enhance the saliency of the key lesion area.

[0072] To fuse multi-scale information, the features of the first three stages are adjusted to the same resolution (16x16) as the fourth stage through bilinear interpolation, and then channel splicing and compression are performed. The spliced multi-scale features are reduced in dimension through 1x1 convolution to generate lightweight fusion features, reducing the computational complexity and preserving key information. Further, the feature expression is optimized through the channel-space-level attention fusion module (SCHL): the channel attention module extracts global context information through global average pooling and global maximum pooling to generate channel weights to enhance the response of key channels. The high-frequency feature extraction branch uses a double-branch 3x3 convolution to capture lesion edge and detail information, and fuses them into a high-frequency feature map through learnable parameters, which is multiplied element-wise with the channel attention feature to strengthen local details. The spatial attention module generates spatial weights based on global pooling and convolution operations to suppress background noise and highlight the spatial distribution of the lesion area. Finally, the original input, channel weighted features, and spatial weighted features are fused in a learnable proportion, and the enhanced multi-scale features are output after further refinement through a 3x3 convolution.

[0073] The classification head part uses global average pooling (GAP) to compress the spatial dimension, and generates class probability distribution through a fully connected layer. During training, the Adam optimizer is used in combination with an adaptive learning rate adjustment strategy to dynamically balance the model convergence speed and stability. Through multi-scale feature fusion and attention enhancement mechanism, MSAFN can effectively distinguish the morphological differences of UC lesions (such as erosion, polypoid hyperplasia, and inflammatory structural changes), and still maintain high classification accuracy in complex background and high inter-class similarity scenarios. During model initialization, the parameters of each module are optimized and configured according to the characteristics of the input data to ensure the comprehensiveness of feature extraction and the reliability of classification decision-making.

[0074] Specifically, the initialization of the EfficientNet backbone network fully utilizes its hierarchical feature extraction capability, gradually transitioning from shallow texture to deep semantic expression; the SCHL module significantly improves the model's focusing ability on the lesion area through the synergistic effect of channel and spatial attention, combined with high-frequency detail enhancement strategy; the lightweight design of the classification head ensures the inference efficiency while avoiding overfitting problems by integrating spatial information through global average pooling. The overall architecture realizes the adaptive optimization of multi-scale features and attention weights through end-to-end training, providing an efficient and reliable solution for the accurate classification of UC images.

[0075] As shown in Figure 1 , the EfficientNet backbone network is divided into four stages, each corresponding to different scale feature expression, and the input image is After the first stage of EfficientNet, the feature map is converted to where C1 is the channel number of this stage, H1 and W1 are the spatial resolution of the feature map. The subsequent stages extract deeper features in turn:

[0076] F i = f i (F i-1 ), i∈{2,3,4}

[0077] where f i (·) represents the transformation operation of EfficientNet at stage i, which mainly consists of MBConv (Mobile Inverted Bottleneck Convolution) modules and contains depth separable convolution and SE (Squeeze-and-Excitation) channel attention mechanism. The channel numbers of the features of each stage are {C1, C2, C3, C4}, finally, the output F4 of the fourth stage is used as high-level semantic features for subsequent multi-scale fusion and attention enhancement.

[0078] Due to the morphological manifestations of UC lesions having multi-scale features, such as erosion, polypoid hyperplasia, and inflammatory bowel structure changes, different scale information is crucial for accurate classification. In order to enhance multi-scale information expression, all stage features are adjusted to the same spatial resolution through bilinear interpolation, i.e.,

[0079]

[0080] Subsequently, all scale features are concatenated in the channel dimension to form the global multi-scale feature representation F multi :

[0081]

[0082] F multi Channel compression is performed by 1x1 convolution to reduce computational complexity while ensuring information preservation:

[0083] F fusion = W conv * F multi

[0084] where W conv is a 1x1 convolution kernel and * represents the convolution operation.

[0085] Finally, the fused feature F fusion is further extracted by the SCHL module to enhance the information expression of key lesion areas and suppress background noise interference, and is sent to the classification head for prediction. The loss function combines cross-entropy loss, inter-class separation loss, and label smoothing regularization to improve the discriminability and generalization performance of the model.

[0086] As Figure 2 shown, to address the problem of low contrast and subtle structural changes in early UC lesions, the SCHL module is proposed, which integrates high-frequency enhancement to highlight fine-grained texture and boundary features. The input feature map is subjected to channel attention to adaptively adjust the importance of each channel, while spatial attention focuses on key areas to improve feature representation. In addition, hierarchical feature fusion can extract high-frequency details and capture subtle changes and edges. These attention-weighted hierarchical features form more discriminative multi-scale representations, enhancing the expressiveness of the features and improving the performance of the model in classifying fine-grained UC lesions.

[0087] The channel attention mechanism is based on a global information aggregation strategy, which extracts global context information through global max pooling (GMP) and global average pooling (GAP) operations, and performs feature transformation through a fully connected network (Fully Connected, FC) to calculate the channel-level attention weight:

[0088]

[0089] X c = X ⊙ M c

[0090] where M c represents the channel attention weight map, σ represents the Sigmoid activation function, ⊕ represents the feature concatenation operation, and ⊙ represents the element-wise multiplication to achieve dynamic feature enhancement at the channel level.

[0091] The hierarchical HM module aims to extract high-frequency information from the input feature map to enhance the perception of edge details and fine-grained features. Specifically, we use two independent 3x3 convolution transformations H1 = f3×3 (X) and H2 = f 3×3 (X) acts on the input feature X and obtains a high-frequency feature map through weighted fusion:

[0092] M F = δ · (H1 + H2)

[0093] where δ is a learnable scaling coefficient used to adjust the contribution of high-frequency information to the final feature representation.

[0094] The high-frequency feature map M F is further applied to the feature map X C after channel attention to enhance the perception of important regions:

[0095] X' C = X C ⊙ M F

[0096] The spatial attention mechanism aims to enhance the importance of local regions and further optimize the spatial distribution of feature maps. Global spatial information is extracted based on GAP and GMP operations, and spatial attention weights are generated through convolution operations:

[0097]

[0098] X s = X' C ⊙ M s

[0099] where M s represents the spatial attention weight, and Conv acts on the concatenated features to generate a spatial weight map.

[0100] In the multi-scale feature fusion process, we use a weighted fusion strategy to integrate channel attention, spatial attention, and high-frequency features to obtain a multi-scale feature representation:

[0101] F out = αX + βX C + γX S

[0102] F' out = F out ⊙ M F

[0103] where α, β, γ are learnable weight parameters used to balance the contribution of different attention mechanisms in feature representation, and the final feature map X final is further refined through 3 × 3 convolution to improve the discriminative ability of the classification task.

[0104] The fusion strategy maintains global information while enhancing local feature expression, improving the model's ability to suppress complex background, distinguish fine-grained classes, and extract high-frequency lesion features, thereby optimizing the representation and classification performance of UC lesions.

[0105] Step 4: Set optimizer and loss function

[0106] Select appropriate optimizer and loss function according to the complexity of the model and the requirements of the ulcerative colitis lesion image classification task;

[0107] The optimizer uses the AdamW optimization strategy and combines the exponential decay learning rate scheduler Learning Rate Scheduler to optimize the model parameters; its learning rate adjustment formula is as follows:

[0108] η t = η0·exp(-λ·t)

[0109] where η t is the learning rate of the t-th round, η0 is the initial learning rate, and λ is the decay rate parameter;

[0110] The loss function is the progressive loss function PIloss, which integrates the polynomial loss and the inter-class feature separation strategy to address the label ambiguity and lesion heterogeneity in ulcerative colitis image classification. PIloss is controlled by a time weight, and its form is:

[0111]

[0112] where y c and are the true label and predicted probability, respectively, C is the total number of classes, α and γ are the adjustment parameters of the polynomial term, f is the sample feature vector, μ y is the corresponding class center feature, and λ(t) is the weight function that changes with the training round t, which is used to smoothly control the dominance of different loss components in the training process to ensure accurate detection of minority classes in medical data, solving the label ambiguity and lesion heterogeneity problems in UC classification. It combines a dynamic weighting mechanism to adaptively balance feature learning at different training stages. PIloss gradually adjusts its focus, emphasizing fine-grained feature learning in the early stage, and gradually shifting to enhancing inter-class separability and intra-class compactness as the model matures. In the early training stage, PolyLoss is used to optimize the classification decision boundary, while in the later stage, it encourages a smooth transition to InterClassSeparationLoss to improve classification separation. This adaptive strategy effectively captures the complexity and heterogeneity of UC lesions, thereby improving classification accuracy and robustness.

[0113] In the model training process, the following dynamic weight mechanism is defined, λ poly (i): the weight coefficient of PolyLoss at the ith moment, which is large at the beginning and gradually decreases later. λ ics (i): the weight coefficient of InterClassSeparationLoss at the ith moment, which is small at the beginning and gradually increases later. The total loss of PIloss can be expressed as:

[0114]

[0115] wherein, is PolyLoss, i.e.:

[0116]

[0117] wherein, is the standard cross-entropy loss, P true is the predicted probability of the real class, and ∈ is the coefficient controlling the polynomial term.

[0118] is InterClassSeparationLoss, i.e.:

[0119]

[0120] wherein, f i is the feature of the ith sample, is the center of the class to which the ith sample belongs, and m is the preset class center distance threshold. PIloss adjusts the weights of the two losses dynamically, so that the model can be optimized flexibly at different training stages. In the early stage of training, the learning of difficult-to-classify samples is focused on, and as the training progresses, the class separation and class compactness are gradually enhanced, thereby improving the overall classification accuracy of the model. This gradual transition strategy effectively avoids the local optimization problem of a single loss function in the training process, improves the fine-grained classification ability and discrimination ability of the model, and ultimately achieves higher classification accuracy. We can quantify the accuracy of the model prediction;

[0121] Step 5: Model training

[0122] The images in the training set are classified according to the lesion category and annotation, the structural and semantic features of different levels are extracted by means of the EfficientNet backbone network, and the multi-scale feature maps are uniformly spatially aligned and channel compressed; a multi-scale attention fusion module SCHL is introduced, which performs channel attention, spatial attention and high-frequency feature modeling operations on the fused feature maps, enhances the expression ability of key lesion areas and suppresses background interference; the perception effect of edge details and small lesions is improved through high-frequency structural information, effectively depicting and accurately classifying the typical and atypical lesion morphology in the image, and significantly improving the adaptability and discriminability of the model to complex clinical images;

[0123] In the above technical solution, the goal of the training phase is to train the MSAFN model using existing labeled images and input ulcerative colitis image data, so that it can learn effective features and make accurate predictions. The trained MSAFN model is saved for further testing;

[0124] MSAFN model test phase

[0125] The test set sample x test is input into the trained MSAFN model. This phase only performs forward propagation, and does not need to perform back propagation and parameter update;

[0126] The model outputs the prediction feature vector y predict of each test sample. By comparing the feature vector with the classification threshold, the model generates the corresponding prediction label;

[0127] These prediction labels can be compared with the true labels y test of the test set to evaluate the performance of the model.

[0128] Step six: model saving and output

[0129] If the performance of the current training round on the validation set is better than the optimal result of the previous round, save the current model parameters and record the relevant performance indicators to ensure that the final best model parameters can be retained. After training is completed, output the final parameters of the model and the best accuracy on the validation set;

[0130] In the above solution, the training loop runs from i = 1 to N, and this process is repeated until the model parameters reach the optimal state. The training loop is the core of the model training process. Through continuous iteration and optimization, the model can gradually learn the internal rules and features of the data. In each iteration, the model will go through a complete forward propagation and back propagation process, gradually adjusting its internal parameters and improving prediction accuracy;

[0131] When the training cycle is repeated for N times, the model is trained, and the parameters of the MSAFN model have been adjusted to the optimal state, and the prediction performance of the model reaches the expected target. The optimal model is saved.

[0132] Step seven: model typing

[0133] For the endoscopic image data of ulcerative colitis patients, an automatic lesion recognition and typing model is constructed to perform multi-grade classification prediction on the colon lesion images of each patient. The model output results can be used for system terminal display and auxiliary diagnosis.

[0134] Further: the classification task includes two types of Mayo grading four classification and ulcerative colitis index six classification, which are used to reflect the clinical activity degree and microscopic pathological feature evolution process of ulcerative colitis lesions, respectively. The Mayo grading classification labels are 0, 1, 2, and 3, respectively, corresponding to normal mucosa, mild activity, moderate activity, and severe activity state. The ulcerative colitis typing labels are ulcerative colitis 0-1, ulcerative colitis 1, ulcerative colitis 1-2, ulcerative colitis 2, ulcerative colitis 2-3, and ulcerative colitis 3, which are used to refine the evolution state of UC lesions in the pathological continuum.

[0135] Further: the model training process includes image normalization, data enhancement, multi-scale feature extraction, attention module integration, and classification module optimization, and outputs the prediction label for clinical auxiliary judgment. In the classification task setting, for four classification: if the model output label is 0, it means Mayo 0 level; the output is 1, which means Mayo 1 level; the output is 2, which means Mayo 2 level; the output is 3, which means Mayo 3 level; if it is a six classification task, the model output label is 0, which means ulcerative colitis 0-1; the output is 1, which means ulcerative colitis 1; the output is 2, which means ulcerative colitis 1-2; the output is 3, which means ulcerative colitis 2, label 4 corresponds to ulcerative colitis 2-3, and label 5 corresponds to ulcerative colitis 3.

[0136] Verification method of the above technical solutions and comparative methods: the Adam optimizer is used to update the network parameters, and the learning rate is set to 1e-4. For training epochs, when the training loss tends to be stable, the training process stops. The training epochs are uniformly set to 300, and the training loss is stable under this setting. The size is set to 64. All comparative methods use the code provided by the original author, and are retested on the same hardware configuration to ensure the consistency of the experiment;

[0137] The number of samples used in the experiment was divided. 0.8% of the samples were selected for training on the LIMUC dataset and Hyper Kvasir, and all training samples were randomly selected to ensure the stability of the algorithm, except for the training samples, which were used for testing;

[0138] The method and all comparison methods are implemented based on the PyTorch framework of PyTorch 2.0.0 and cuda11.8. Training and testing all experiments are performed on an NVIDIA GeForce RTX 4060 Laptop GPU. In order to quantitatively evaluate the classification effect of each method, four main indicators, accuracy, precision, recall and F1-score, are used to quantitatively evaluate the proposed model. The definitions of each indicator are as follows:

[0139]

[0140] Among them, TP, TN, FP and FN represent the number of true positives, true negatives, false positives and false negatives, respectively.

[0141] The present scheme is compared with the deep learning image classification methods in the prior art, covering 5 kinds of classic CNN architectures (AlexNet, VGGNet, GoogLeNet, MobileNet, ResNet) and 3 kinds of representative Transformer architectures (Swin-Transformer, MobileViT, ConViT).

[0142] Each method is trained and tested on the UC image dataset constructed in this study, and a unified hyperparameter configuration is used to ensure fairness. The classification accuracy statistics are shown in Tables 1 and 2.

[0143] Specifically, for the classification results on the LIMUC dataset in Figure 3 and Figure 4 , the results are shown in (a) AlexNet (58.3%), (b) VGGNet (57.4%), (c) GoogLeNet (60.2%), (d) MobileNet (65.1%), (e) ResNet (64.9%), (f) Swin-Transformer (50.4%), (g) EfficientNet (75.1%), (h) ConViT (51.8%), (i) MSAFN proposed in this paper (80%). It can be seen that MSAFN performs best among all models.

[0144] As Figure 5The classification results on the Hyper Kvasir dataset are shown in (a) AlexNet (57.9%), (b) VGGNet (51.5%), (c) GoogLeNet (58.3%), (d) MobileNet (60.1%), (e) ResNet (58.6%), (f) Swin-Transformer (52.3%), (g) EfficientNet (61.1%), (h) ConViT (51.2%), (i) MSAFN proposed in this paper (69.3%). It can be seen that MSAFN performs best among all models.

[0145] From the result analysis, the traditional CNN architecture mainly relies on local convolution kernels in the feature extraction process, and has limited modeling ability for complex lesion boundaries and diversified morphology; while the Transformer structure has the ability to model global dependence, but in the scene of medical images with small sample size, it is prone to overfitting, and the classification effect is not as expected. In contrast, MSAFN integrates semantic information at different levels through a multi-scale fusion strategy, and combines with the SCHL attention mechanism, which can effectively focus on the lesion area while suppressing the interference of atypical structures, thereby showing better discrimination performance in scenarios where the difference between classes is small. In addition, the introduction of PILoss further improves the intra-class consistency and inter-class separation degree in the feature space, and shows stronger discrimination ability for complex and fine-grained UC lesions.

[0146] As Figure 6 shown, in order to verify the feature extraction ability of the scheme, the inventors used the Grad-CAM visualization tool to conduct explanatory analysis on the discrimination area of the model combined with the Grad-CAM visualization method, and the results showed that the model could effectively locate the inflammatory mucosa area, and the highlighted area had high consistency with the lesion area labeled by experts, had good clinical interpretability and lesion recognition ability, and the clarity and accuracy of lesion visualization showed that the method of the inventors was a valuable supplement to the clinical workflow, and could provide diagnostic assistance for radiologists and gastroenterologists. In summary, the Grad-CAM-based visualization of the lesion area of the colorectal image shows the significant benefits of the MSAFN model proposed by the inventors in the clinical aspect, especially in improving the diagnostic accuracy, interpretability and robustness. These features highlight the superiority of the method of the inventors in research and real clinical application.

[0147]

[0148] Table 1 Comparison of the model of the application and other models on the LIMUC dataset

[0149] Table 2 Comparison of the present model with other models on the Hyper Kvasir dataset

[0150]

[0151] Table 2 Comparison of the present model with other models on the Hyper Kvasir dataset.

Claims

1. A method for classifying ulcerative colitis images based on a multi-scale attention fusion network, wherein the image classification method classifies input images based on image features, characterized in that, Includes the following steps: Step 1: Obtain the ulcerative colitis image dataset and divide it into training, testing, and validation sets; Step 2: Data preprocessing; The training set, test set, and validation set were divided in a 7:2:1 ratio. The training set was enhanced by random horizontal flipping, vertical flipping, and standardization to improve the model's generalization ability. Combined with the image features of ulcerative colitis, image smoothing, denoising, and morphological transformation enhancement operations were introduced to improve the model's robustness to the diverse structures of lesions. Step 3: Establish and initialize the MSAFN backbone network classification model; The MSAFN backbone network classification model includes the EfficientNet backbone network, the SCHL multi-scale feature fusion module (SCHL), and a classification module; Based on the classification task requirements of ulcerative colitis (UC) images, the MBConv module and channel attention mechanism (SE) of each stage of the EfficientNet backbone network are constructed and initialized to perform multi-stage feature extraction on the input image. The feature maps at different levels, from shallow to deep and from texture details to semantic structure, are obtained layer by layer. The feature maps output by each stage are aligned to the same spatial resolution through bilinear interpolation and then stitched together in the channel dimension to form a fused multi-scale global feature representation. Convolution operation is introduced to compress the channels, reduce redundancy and compress the feature dimension. The compressed fusion features are input into the SCHL multi-scale attention fusion module. The SCHL multi-scale attention fusion module combines channel attention, spatial attention and high-frequency feature modeling mechanism to gradually guide the model to focus on key lesion areas in the image, while effectively suppressing background interference. The attention-enhanced high-level features are input into the classification module to complete the final prediction output for the Mayo hierarchical four-class classification or UCG six-class classification task; Step 4: Set up the optimizer and loss function; Step 5: Model Training; Images in the training set are classified according to lesion type and annotation. Structural and semantic features at different levels are extracted using the EfficientNet backbone network. Unified spatial alignment and channel compression are performed on the multi-scale feature maps. The multi-scale attention fusion module SCHL is introduced to perform channel attention, spatial attention and high-frequency feature modeling operations on the fused feature maps, which enhances the expressive ability of key lesion regions and suppresses background interference. The perception effect of edge details and small lesions is improved by high-frequency structural information, so as to achieve effective characterization and accurate classification of typical and atypical lesion morphology in the image, and improve the model's adaptability and discrimination ability to complex clinical images. Step Six: Model Saving and Output; If the validation set performance of the current training round is better than the best result of the previous round, the current model parameters are saved and the relevant performance metrics are recorded to ensure that the final best model parameters can be retained. After training is completed, the final parameters of the model and the best accuracy on the validation set are output. Step 7: Model typing; An automated lesion identification and classification model was constructed based on endoscopic image data of patients with ulcerative colitis. The model performs multi-level classification and prediction of colonic lesions for each patient, and the model output is used for display on the system terminal and for assisted diagnosis.

2. The ulcerative colitis image classification method based on a multi-scale attention fusion network according to claim 1, wherein the optimizer adopts the AdamW optimization strategy and combines it with an exponentially decaying learning rate scheduler (Learning RateScheduler) to optimize model parameters, and its learning rate adjustment formula is as follows: or t =η0·exp(-λ·t) in, η t Let η be the learning rate in round t, η0 be the initial learning rate, and λ be the decay rate parameter.

3. The ulcerative colitis image classification method based on a multi-scale attention fusion network according to claim 1, wherein the loss function is a progressive loss function (PIloss), which addresses the problems of label ambiguity and lesion heterogeneity in ulcerative colitis image classification. PIloss integrates multinomial loss and inter-class feature separation strategies, and is controlled by time weights, taking the following form: in, y c and Let C be the true label and the predicted probability, respectively; C be the total number of categories; α and γ be the adjustment parameters of the multinomial terms; f be the sample feature vector; and μ be the value of μ. y For the corresponding class center feature, λ(t) is a weight function that varies with the number of training rounds t, used to smoothly control the dominance of different loss components during the training process to ensure accurate detection of minority classes in medical data.

4. The image classification method for ulcerative colitis based on a multi-scale attention fusion network according to claim 1, characterized in that: The classification task includes two categories: the Mayo Criterion (4 categories) and the Ulcerative Colitis (UC) Index (6 categories). These categories are used to reflect the clinical activity level and the evolution of microscopic pathological features of UC lesions. The Mayo Criterion classification labels are 0, 1, 2, and 3, corresponding to normal mucosa, mild activity, moderate activity, and severe activity, respectively. The Ulcerative Colitis subtype labels are Ulcerative Colitis 0-1, Ulcerative Colitis 1, Ulcerative Colitis 1-2, Ulcerative Colitis 2, Ulcerative Colitis 2-3, and Ulcerative Colitis 3, which are used to refine the evolution of UC lesions in the pathological continuum.

5. The image classification method for ulcerative colitis based on a multi-scale attention fusion network according to claim 1, characterized in that: The model training process includes image normalization, data augmentation, multi-scale feature extraction, attention module integration, and classification module optimization. It outputs predicted labels for clinical auxiliary diagnosis. In the classification task settings, for a four-class classification task: if the model output label is 0, it represents Mayo level 0; if the output is 1, it represents Mayo level 1; if the output is 2, it represents Mayo level 2; and if the output is 3, it represents Mayo level 3. For a six-class classification task, if the model output label is 0, it represents ulcerative colitis 0-1; if the output is 1, it represents ulcerative colitis 1; if the output is 2, it represents ulcerative colitis 1-2; and if the output is 3, it represents ulcerative colitis 2. Label 4 corresponds to ulcerative colitis 2-3, and label 5 corresponds to ulcerative colitis 3.

Images