A colorectal cancer microscopic fluorescence image classification method based on ResNet deep learning

Abstract

The application discloses a kind of based on ResNet deep learning's colorectal cancer microscopic fluorescence image classification method, comprising: obtaining the microscopic fluorescence image of colorectal tissue labeled after fluorescence probe HSP90-CY5;Image is pretreated;The image after pretreatment is input into the light weight ResNet-8 network model of embedding high efficiency local attention module, and network model extracts molecular signal features and tissue microstructure features in image by residual structure and spatial attention mechanism;Classification probability output by network model is obtained in turn, and finally the classification corresponding to the maximum probability is output.The application solves the problems that existing colonoscopy image diagnosis relies on subjective experience of doctors, it is difficult to distinguish inflammation and early adenocarcinoma, and lacks interpretability, realizes high-precision, low-delay, interpretable automatic classification of colorectal cancer microscopic fluorescence image.

Description

Technical Field

[0001] This invention belongs to the field of image processing and deep learning technology, and specifically relates to an image classification method based on deep learning. Background Technology

[0002] Early diagnosis of colorectal cancer is crucial for improving patient prognosis. Colonoscopy is the gold standard for colorectal cancer screening and diagnosis, but traditional white light colonoscopy mainly relies on the doctor's subjective judgment of the morphological characteristics of the lesions, which has the following technical defects: (1) High rate of missed diagnosis of early flat and micro adenomas, with conventional white light colonoscopy missing about one-third of adenomas; (2) Difficulty in effectively distinguishing between benign inflammatory lesions and early adenocarcinoma, especially in patients with inflammatory bowel disease, where the difficulty of identifying dysplasia increases significantly; (3) Poor consistency in diagnosis among different doctors, resulting in low efficiency of manual interpretation.

[0003] In recent years, fluorescent probe technology targeting tumor-specific biomarkers has provided a new direction for molecular imaging of colorectal cancer. Heat shock protein 90 (HSP90) exhibits tumor-specific high expression in colorectal cancer tissues, making it an ideal molecular imaging target. The HSP90-CY5 fluorescent probe, based on HSP90, can generate strong near-infrared fluorescence signals in tumor regions, while showing only weak background signals in normal and inflamed mucosa. However, existing fluorescent probe diagnostic systems mainly rely on fluorescence signal intensity as the basis for interpretation, failing to automatically capture the microstructural information of tumor tissues and lacking intelligent image analysis methods, making it difficult to achieve high-throughput, standardized diagnosis.

[0004] Deep learning technology, especially residual networks, has made significant progress in the field of medical image analysis. However, most existing models are trained based on conventional white light endoscopy images, failing to fully utilize the specific fluorescence signals provided by molecularly targeted probes. Furthermore, these models have a large number of parameters, slow inference speed, and insufficient interpretability, making it difficult to meet the needs of real-time clinical auxiliary diagnosis.

[0005] Therefore, developing a colorectal cancer microscopic fluorescence image classification method that can integrate molecularly targeted fluorescence signals with tissue microstructure features, and possesses high precision, low latency, and interpretability, is of great significance for addressing the aforementioned clinical challenges. Summary of the Invention

[0006] Purpose of the invention: To address the aforementioned existing technologies, this invention proposes a ResNet deep learning-based method for classifying colorectal cancer microscopic fluorescence images. This method integrates molecularly targeted fluorescence signals with tissue microstructure features, and offers high accuracy, low latency, and interpretability.

[0007] Technical Solution: A ResNet deep learning-based method for classifying colorectal cancer microfluorescence images, wherein the colorectal cancer microfluorescence images are colorectal tissue microfluorescence images labeled with the fluorescent probe HSP90-CY5 targeting heat shock protein 90, and the method includes:

[0008] Step 1: Preprocess the microscopic fluorescence image;

[0009] Step 2: Input the preprocessed image into a pre-trained deep learning network model, which is a lightweight ResNet-8 network model with embedded high-efficiency local attention modules;

[0010] Step 3: Sequentially obtain the classification results of each classifier in the network model, and determine the category to which the microscopic fluorescence image belongs based on the obtained classification results. The category includes normal tissue, inflamed tissue, and adenocarcinoma tissue.

[0011] Furthermore, in step 3, the network model uses a binary classifier to obtain two probability values, namely: probability value P1 = Max[P normal , P carcinoma The probability value P2 = Max[P] colitis , P carcinoma ], where P normal For normal organization probability, P carcinoma P represents the probability of adenocarcinoma tissue. colitis Let P1 be the probability of inflamed tissue, and Max be the function that maximizes the probability. The final output is the classification corresponding to Max[P1,P2].

[0012] Furthermore, in step 1, the preprocessing includes: scaling the image to 224×224×3 pixels, cropping to remove redundant edge areas, randomly flipping the image horizontally, and standardizing the pixel values.

[0013] Furthermore, in step 2, the structure of the lightweight ResNet-8 network model is as follows:

[0014] Input layer: Receives a 224×224×3 three-channel image;

[0015] First convolutional layer: 32 kernels, 7×7 kernel size, stride 2, output 112×112×32;

[0016] Maximum pooling layer: 3×3 pooling kernel, 2-step stride, 56×56×32 output;

[0017] Four residual blocks, each containing two 3×3 convolutional layers, with the number of convolutional kernels in each residual block being 32, 64, 128, and 256, respectively. A high-efficiency local attention module is embedded at the end of each residual block.

[0018] Global average pooling layer;

[0019] Flatten layer;

[0020] Fully connected layer, with an output dimension of 2;

[0021] A Dropout layer with a deactivation rate of 0.5 is set before the fully connected layer.

[0022] Furthermore, the high-efficiency local attention module is implemented as follows:

[0023] One-dimensional global average pooling is performed on the input feature map along the height and width directions respectively, while preserving the original channel dimensions;

[0024] By replacing batch normalization with group normalization, one-dimensional convolution with adjustable receptive field independently encodes the spatial dependencies in the horizontal and vertical directions;

[0025] The attention maps generated in the two directions are broadcast and multiplied to generate pixel-level spatial weights.

[0026] The spatial weights are multiplied element-wise with the original feature map to output the weighted feature map.

[0027] Furthermore, the high-efficiency local attention module includes a biaxial global average pooling layer, a set of parameter-shared 1D grouped convolutional layers, a grouped normalization layer, and a sigmoid activation layer. Finally, the original features are weighted and output by multiplying the biaxial attention weights.

[0028] Furthermore, the training method for the deep learning network model includes:

[0029] Obtain a sample dataset, which includes microscopic fluorescence images of normal colorectal tissue, inflamed colorectal tissue, and adenocarcinoma colorectal tissue labeled with the fluorescent probe HSP90-CY5 targeting heat shock protein 90;

[0030] Preprocessing and data augmentation of sample images;

[0031] The feature extraction backbone of the initial ResNet-8 network model is trained using labeled sample images. The classification result is obtained through a classifier. The classification result and the corresponding true label of the sample are compared using the cross-entropy loss function to calculate the loss.

[0032] Iterate through the training process until the model converges, and save the optimal model weight parameters.

[0033] A ResNet deep learning-based classification system for colorectal cancer microscopic fluorescence images, the system comprising:

[0034] The image input module is used to input microscopic fluorescence images of colorectal tissue labeled with the fluorescent probe HSP90-CY5, which targets heat shock protein 90.

[0035] The preprocessing module is used to preprocess the image, including scaling the image, cropping to remove redundant edge areas, randomly flipping the image horizontally, and normalizing the pixel values.

[0036] The deep learning classification module deploys a pre-trained lightweight ResNet-8 network model embedded with a high-efficiency local attention module. The binary classifier of the network model outputs two probability values, P1 and P2, which correspond to the probability of normal tissue and adenocarcinoma tissue, and the probability of normal tissue and inflammatory tissue, respectively. Finally, it outputs the classification corresponding to Max[P1,P2].

[0037] Beneficial effects: (1) This invention is the first to combine the microscopic fluorescence image of HSP90-CY5, a fluorescent probe targeting heat shock protein 90, with a deep learning classification method, giving full play to the high tumor specificity of molecular probes and the feature extraction capability of deep learning, and realizing the accurate differentiation of normal tissues, inflamed tissues and adenocarcinoma tissues.

[0038] (2) This invention constructs a lightweight ResNet-8 network with embedded high-efficiency local attention (ELA) module. Compared with the traditional ResNet-50, the number of parameters is reduced by about 95%, the single-sample inference time is shortened to about 120ms, and the classification accuracy of "normal-adenocarcinoma" on the test set reaches 0.98, and the classification accuracy of "inflammation-adenocarcinoma" reaches 0.95. It maintains high classification accuracy while significantly reducing computational overhead.

[0039] (3) The present invention confirms through SmoothGradCAMpp visualization analysis that the model decision is highly focused on the adenocarcinoma glandular structure region enriched by HSP90-CY5 signal, which is consistent with the pathological gold standard and has good interpretability. Attached Figure Description

[0040] Figure 1 A flowchart of a colorectal cancer microscopic fluorescence image classification method based on ResNet deep learning provided in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of a lightweight Res1et-8 network structure with an embedded high-efficiency local attention module provided in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the high-efficiency local attention module structure provided in an embodiment of the present invention.

[0043] Figure 4 The confusion matrix and ROC curve of the "normal-adenocarcinoma" classification model provided in the embodiments of the present invention.

[0044] Figure 5 The confusion matrix and ROC curve of the "inflammation-adenocarcinoma" classification model provided in the embodiments of the present invention.

[0045] Figure 6 The SmoothGradCAMpp interpretable heatmap provided for embodiments of the present invention. Detailed Implementation

[0046] The invention will now be further explained with reference to the accompanying drawings.

[0047] Example 1:

[0048] like Figure 1 As shown, a method for classifying colorectal cancer microscopic fluorescence images based on ResNet deep learning is described, with the following specific steps:

[0049] 1. Construction of the sample dataset

[0050] Animal model: A mouse colorectal cancer model induced by AOM / DSS was used. Seven- to eight-week-old C57BL / 6 mice were intraperitoneally injected with 10 mg / kg of azomethane. After drinking normal water for one week, the drinking water was replaced with 2.5% DSS solution for one week, followed by two weeks of normal water intake. The DSS administration cycle was repeated three times.

[0051] Image Acquisition: In vivo fluorescence microscopy was performed on mice at week 0 (normal stage), week 4 (inflammatory stage), and week 10 (adenocarcinoma stage). Before imaging, 100 μL of 0.3% HSP90-CY5 probe solution was instilled rectally, incubated in the dark for 10 minutes, and rinsed four times with physiological saline. An integrated fiber optic probe with both beam and image transmission bundles was used to acquire images layer by layer along the colorectal wall at a working distance of 1.5 mm, at a depth of 2–30 mm. 10–30 images were acquired from each mouse based on the stage of the lesion, and blurry or improperly exposed low-quality images were discarded.

[0052] Dataset partitioning: The internal cohort consisted of 15 mice, yielding 450 valid images: 150 each for normal, inflammatory, and adenocarcinoma mice. These images were randomly divided into training and validation sets at an 8:2 ratio, with 120 images per class in the training set and 30 images per class in the validation set. The external independent validation cohort, consisting of 15 mice, was independently collected by another experimenter, yielding 153 valid images (51 per class) to evaluate the method's generalization ability.

[0053] Data preprocessing and augmentation: All images were scaled to 224×224×3, and the black redundant areas at the four corners were cropped. They were then randomly flipped horizontally with a probability of 0.5, and pixel normalization was performed. Data augmentation was applied to the training set using random rotation (0~90°), horizontal flipping, vertical flipping, center cropping, and edge cropping. After augmentation, the number of images in the training set increased to 720, i.e., 240 images per class, while the validation set remained unchanged at 90 images.

[0054] 2. Construction of Deep Learning Network Models

[0055] This embodiment provides the specific structure of a lightweight ResNet-8 network model with an embedded high-efficiency local attention (ELA) module. For example... Figure 2 As shown, the network structure is as follows:

[0056] Input layer: Receives a 224×224×3 three-channel image.

[0057] Convolutional layer 1: 32 7×7 convolutional kernels, stride 2, padding 3, output 112×112×32; batch normalization + ReLU activation.

[0058] Pooling layer: 3×3 max pooling, step size 2, output 56×56×32.

[0059] Residual Block 1: Contains two 3×3 convolutional layers, each with 32 kernels. The first convolutional layer is followed by batch normalization and ReLU, and the second convolutional layer is followed by batch normalization. The residual connection adds the input and output and then activates it via ReLU. A high-efficiency local attention (ELA) module is embedded at the end of the residual block, resulting in an output of 56×56×32.

[0060] Residual Block 2: The structure is the same as that of Residual Block 1, but the number of convolution kernels is increased to 64, and the output is 28×28×64.

[0061] Residual block 3: The number of convolution kernels is increased to 128, and the output is 14×14×128.

[0062] Residual block 4: The number of convolution kernels is increased to 256, and the output is 7×7×256.

[0063] Global average pooling layer: Output 1×1×256.

[0064] Flatten layer: Outputs a 256-dimensional feature vector.

[0065] Dropout layer: 0.5% inactivation rate.

[0066] Fully connected layer: Output dimension 2, that is, two independent binary classification heads, corresponding to "normal / adenocarcinoma" and "inflammation / adenocarcinoma" respectively.

[0067] like Figure 3As shown, the High Efficiency Local Attention (ELA) module has the following structure: a biaxial global average pooling layer, a set of parameter-shared 1D grouped convolutional layers, a grouped normalization layer, and a sigmoid activation layer. Finally, the original features are weighted and output by multiplying the biaxial attention weights.

[0068] This scheme is adapted to the ELA-T specification of lightweight ResNet, configured with a kernel size of 5, channel-wise grouped convolutions, and a normalized group number of 32. Specifically, the input feature map X∈R B×C×H×W First, global average pooling is performed along the width and height axes respectively, splitting the feature map into a one-dimensional spatial feature in the height direction with size B×C×H and a one-dimensional spatial feature in the width direction with size B×C×W, where B is the batch size, and C, H, and W are the number of channels, height, and width of the input feature map, respectively. Then, the two one-dimensional features are processed by a parameter-shared one-dimensional grouped convolution (with C input channels and C output channels) to extract axial spatial context features, and then the feature distribution is adjusted by grouping normalization. Then, a normalized biaxial spatial attention weight with a range of 0~1 is generated by the Sigmoid activation function, and then multiplied point by point with the original input features, that is, each spatial position is multiplied by its corresponding attention weight coefficient, and then output to the next network stage.

[0069] 3. Model Training

[0070] Optimizer: Adam, initial learning rate 0.001, β1=0.9, β2=0.999, weight decay 1e-4.

[0071] Loss function: Cross-entropy loss, which calculates the loss by comparing the classification result obtained by the classifier with the true label corresponding to the sample.

[0072] Training strategy: Batch size 32, maximum training epochs 100, early stopping method, stopping if the validation set loss does not decrease for 10 consecutive epochs. Learning rate scheduling: multiplied by 0.1 every 30 epochs.

[0073] The ResNet-8 network model is trained in a supervised manner using labeled sample images from the training set, and the model weights with the highest accuracy on the validation set are saved.

[0074] 4. Specific implementation of classification methods

[0075] Classify microscopic fluorescence images using a trained network model:

[0076] Step 1: Obtain a micro-fluorescence image of the colorectal tissue to be classified. This micro-fluorescence image is a micro-fluorescence image of the colorectal tissue labeled with the fluorescent probe HSP90-CY5 that targets heat shock protein 90.

[0077] Step 2: Preprocess the image: scale it to 224×224×3, crop to remove redundant edge areas, randomly flip it horizontally with a probability of 0.5, and normalize the pixels.

[0078] Step 3: Input the preprocessed image into the trained ResNet-8 network model.

[0079] Step 4: Forward propagation of the model. For the binary classification subtask, the Softmax function outputs the probability distributions for the two classes respectively: normal tissue probability P. normal Probability P of adenocarcinoma tissue carcinoma and the probability P of inflamed tissue colitis Probability P of adenocarcinoma tissue carcinoma .

[0080] Step 5: For each task, the category with the highest probability P is taken as the classification result for that task, where P = Max[P]. normal ,P carcinoma ] or P = Max[P colitis , P carcinoma For example, in a task classifying normal cells and adenocarcinoma, if P... carcinoma If the result is the largest, then the corresponding subtask output candidate result is "adenocarcinoma".

[0081] For a single image, the average inference time of the method in this embodiment is approximately 120ms.

[0082] 5. Evaluate the classification performance of the method in this embodiment.

[0083] After excluding inflammatory samples, a binary classification task of "normal-adenocarcinoma" was performed, yielding the following results:

[0084] like Figure 4 , Figure 5 As shown, on an internal test set consisting of 90 normal images and 90 adenocarcinoma images, the overall classification accuracy was 0.98. For adenocarcinoma, the precision was 0.98, recall was 0.99, and F1-score was 0.98; for normal samples, the precision was 0.99, recall was 0.98, and F1-score was 0.98. The confusion matrix showed that 0.99% of adenocarcinoma samples were correctly classified, with only 0.01 misclassified as normal; for normal samples, the correct classification rate was 0.98%, with 0.02 misclassified as adenocarcinoma. The area under the ROC curve (AUC) was 1.00, with a 95% confidence interval of 0.99–1.00.

[0085] On an external independent test set consisting of 51 normal images and 51 adenocarcinoma images, the correct classification rate for normal samples was 0.91, the correct classification rate for adenocarcinoma samples was 1.00, the AUC was 0.99, and the 95% confidence interval was 0.98–1.00.

[0086] After excluding normal samples, a binary classification task of "inflammation-adenocarcinoma" was performed, yielding the following results:

[0087] On an internal test set consisting of 90 inflammation images and 90 adenocarcinoma images, the overall classification accuracy was 0.95. For adenocarcinoma, the precision was 0.92, recall 0.99, and F1-score 0.95; for inflammation, the precision was 0.99, recall 0.91, and F1-score 0.95. The confusion matrix showed that the proportion of correctly classified adenocarcinoma samples was 0.99, and the proportion misclassified as inflammation was 0.01; the proportion of correctly classified inflammation samples was 0.91, and the proportion misclassified as adenocarcinoma was 0.09. The AUC was 0.99, with a 95% confidence interval of 0.99–1.00.

[0088] On an external independent test set consisting of 51 inflammation images and 51 adenocarcinoma images, the correct classification rate for adenocarcinoma was 0.98, and the correct classification rate for inflammation was 0.88, with an AUC of 0.94 and a 95% confidence interval of 0.91–0.97.

[0089] 6. Interpretability Analysis

[0090] This embodiment uses the SmoothGradCAMpp method to visualize and interpret the model's decisions.

[0091] like Figure 6 As shown, for normal tissue images, the heatmap shows no obvious highlighted areas, indicating a scattered focus of the model; for inflamed tissue images, the heatmap shows weak, diffuse highlighting in localized areas, but the intensity is low; for adenocarcinoma tissue images, the highlighted areas on the heatmap are concentrated and significant, highly overlapping with the areas of strong fluorescence signal in the original microscopic fluorescence image. Pathologists confirmed that the highlighted areas correspond precisely to the location of the adenocarcinoma glandular structure shown in the HE-stained sections.

[0092] These results indicate that the method of the present invention does not learn non-specific artifacts, but rather truly learns the pathological features related to the HSP90-CY5 molecular targeting signal and adenocarcinoma microstructure, demonstrating good interpretability.

[0093] Example 2:

[0094] A ResNet deep learning-based classification system for colorectal cancer microscopic fluorescence images includes:

[0095] The image input module is used to input microscopic fluorescence images of colorectal tissue labeled with the fluorescent probe HSP90-CY5, which targets heat shock protein 90.

[0096] The preprocessing module is used to preprocess the image, including image resizing, cropping to remove redundant edge areas, random horizontal flipping, and pixel value normalization.

[0097] The deep learning classification module deploys a pre-trained lightweight ResNet-8 network model with embedded high-efficiency local attention modules. The binary classifier of the network model outputs two probability values, P1 and P2, which correspond to the probability of normal tissue and adenocarcinoma tissue, and the probability of normal tissue and inflammatory tissue, respectively. Finally, it outputs the classification corresponding to Max[P1,P2].

[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for classifying colorectal cancer microscopic fluorescence images based on ResNet deep learning, characterized in that, The colorectal cancer fluorescence microscopy image is a colorectal tissue fluorescence microscopy image labeled with the fluorescent probe HSP90-CY5 targeting heat shock protein 90, and the method includes: Step 1: Preprocess the microscopic fluorescence image; Step 2: Input the preprocessed image into a pre-trained deep learning network model, which is a lightweight ResNet-8 network model with embedded high-efficiency local attention modules; Step 3: Sequentially obtain the classification results of each classifier in the network model, and determine the category to which the microscopic fluorescence image belongs based on the obtained classification results. The category includes normal tissue, inflamed tissue, and adenocarcinoma tissue.

2. The method according to claim 1, characterized in that, In step 3, the network model uses a binary classifier to obtain two probability values, namely: probability value P1 = Max[P normal , P carcinoma The probability value P2 = Max[P] colitis ,P carcinoma ], where P normal For normal organization probability, P carcinoma P represents the probability of adenocarcinoma tissue. colitis Let P1 be the probability of inflamed tissue, and Max be the function that maximizes the probability. The final output is the classification corresponding to Max[P1,P2].

3. The method according to claim 1, characterized in that, In step 1, the preprocessing includes: scaling the image to 224×224×3 pixels, cropping to remove redundant edge areas, randomly flipping the image horizontally, and standardizing the pixel values.

4. The method according to claim 1, characterized in that, In step 2, the structure of the lightweight ResNet-8 network model is as follows: Input layer: Receives a 224×224×3 three-channel image; First convolutional layer: 32 kernels, 7×7 kernel size, stride 2, output 112×112×32; Maximum pooling layer: 3×3 pooling kernel, 2-step stride, 56×56×32 output; Four residual blocks, each containing two 3×3 convolutional layers, with the number of convolutional kernels in each residual block being 32, 64, 128, and 256, respectively. A high-efficiency local attention module is embedded at the end of each residual block. Global average pooling layer; Flatten layer; Fully connected layer, with an output dimension of 2; A Dropout layer with a deactivation rate of 0.5 is set before the fully connected layer.

5. The method according to claim 1 or 4, characterized in that, The high-efficiency local attention module is implemented as follows: One-dimensional global average pooling is performed on the input feature map along the height and width directions respectively, while preserving the original channel dimensions; By replacing batch normalization with group normalization, one-dimensional convolution with adjustable receptive field independently encodes the spatial dependencies in the horizontal and vertical directions; The attention maps generated in the two directions are broadcast and multiplied to generate pixel-level spatial weights. The spatial weights are multiplied element-wise with the original feature map to output the weighted feature map.

6. The method according to claim 1 or 4, characterized in that, The high-efficiency local attention module includes a biaxial global average pooling layer, a set of parameter-shared 1D grouped convolutional layers, a grouped normalization layer, and a sigmoid activation layer. Finally, the original features are weighted and output by multiplying the biaxial attention weights.

7. The method according to claim 1, characterized in that, The training methods for the deep learning network model include: Obtain a sample dataset, which includes microscopic fluorescence images of normal colorectal tissue, inflamed colorectal tissue, and adenocarcinoma colorectal tissue labeled with the fluorescent probe HSP90-CY5 targeting heat shock protein 90; Preprocessing and data augmentation of sample images; The feature extraction backbone of the initial ResNet-8 network model is trained using labeled sample images. The classification result is obtained through a classifier. The classification result and the corresponding true label of the sample are compared using the cross-entropy loss function to calculate the loss. Iterate through the training process until the model converges, and save the optimal model weight parameters.

8. A colorectal cancer microscopic fluorescence image classification system based on ResNet deep learning, characterized in that, The system includes: The image input module is used to input microscopic fluorescence images of colorectal tissue labeled with the fluorescent probe HSP90-CY5, which targets heat shock protein 90. The preprocessing module is used to preprocess the image, including scaling the image, cropping to remove redundant edge areas, randomly flipping the image horizontally, and normalizing the pixel values. The deep learning classification module deploys a pre-trained lightweight ResNet-8 network model embedded with a high-efficiency local attention module. The binary classifier of the network model outputs two probability values, P1 and P2, which correspond to the probability of normal tissue and adenocarcinoma tissue, and the probability of normal tissue and inflammatory tissue, respectively. Finally, it outputs the classification corresponding to Max[P1,P2].

Images