Endoscope image classification method based on multi-granularity perception and residual attention
By introducing a multi-granularity perception module and a residual attention module into the endoscopic image classification method, the problem of redundant local features in the shallow network of the visual Transformer model is solved, the classification accuracy is improved, and the workload of doctors is reduced.
Patent Information
- Application Number
- CN202510651082.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-20
- Publication Date
- 2025-09-23
AI Technical Summary
Traditional endoscopic image classification methods based on visual Transformer focus too much on redundant local features in shallow networks, resulting in feature collapse in deep networks and insufficient classification accuracy.
A multi-granularity perception module is used to capture multi-scale features and expand the receptive field in the shallow layer of the network. The residual attention module is combined with low-level features in the deep layer to retain the global context capability. After a four-stage feature extraction process, a multi-layer perceptron is finally used for classification.
It improves the accuracy of endoscopic image classification, reduces the pressure on doctors to read images, enhances the feature diversity of deep networks, and assists doctors in rapid diagnosis.
Smart Images
Figure CN120689661A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image classification in computer vision, and specifically relates to an endoscopic image classification method based on multi-granularity perception and residual attention. Background Art
[0002] Existing technologies, such as computer-assisted endoscopic diagnosis, can significantly increase doctors' workload and time pressure when interpreting images. This technology utilizes computer vision to process images and automatically analyze and classify endoscopic images. This not only improves image classification accuracy but also significantly reduces doctors' workload. Computer-assisted diagnosis not only automatically identifies lesions but also provides real-time diagnostic recommendations, helping doctors more quickly determine the presence of cancer or other lesions, thereby improving the accuracy of early diagnosis.
[0003] In recent years, deep learning frameworks have attracted widespread attention in the field of computer vision, and researchers have begun applying them to various classification tasks. Models based on the visual Transformer architecture have demonstrated strong feature extraction capabilities across a wide range of tasks. However, endoscopic image classification methods based on the visual Transformer architecture overly focus on highly redundant local features in shallow layers of the network, leading to feature collapse in deeper layers, which limits the model's endoscopic image classification accuracy. Summary of the Invention
[0004] The purpose of the present invention is to provide an endoscopic image classification method based on multi-granularity perception and residual attention, so as to solve the problem that the traditional endoscopic image classification method based on visual Transformer over-focuses on redundant local features in shallow networks and features collapse in deep networks, resulting in insufficient classification accuracy.
[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:
[0006] The present invention proposes an endoscopic image classification method based on multi-granularity perception and residual attention, the method comprising:
[0007] Obtaining the labeled Nth type of endoscopic image;
[0008] Inputting the image into a pre-trained endoscopic image classifier to obtain an endoscopic image classification prediction result;
[0009] The endoscopic image classifier is trained using four stages of feature extraction based on a pre-established deep learning network. The first two stages consist of a 2×2 convolution and a multi-granularity perception module, and the last two stages consist of an image patch embedding module and a residual attention module.
[0010] In the feature extraction process of the four stages, the multi-granularity perception module performs super-label attention and multi-head convolution calculations on the input features to output the first-stage feature map X1; X1 is processed again with multi-granularity perception to output the second-stage feature map X2; X2 is input into the residual attention module, and the third-stage feature map X3 is output through residual attention calculation; X3 is calculated again with residual attention to output the fourth-stage feature map X4, and X4 is input into the multi-layer perceptron for processing to obtain the classification prediction result.
[0011] Furthermore, before training the deep learning network, the method further includes:
[0012] Obtain N types of endoscopic image samples of C×H×W categories, preprocess the samples to obtain a training set E, and use the training set E as the input feature;
[0013] E={E1,E2,...,E n ,...,E N}; E n represents the nth class of endoscopic image samples, and there are P images in the nth class. represents the p-th image in the n-th type of preprocessed endoscopic image samples; C represents the image channel, H represents the image height, W represents the image width, and n = 1, 2, ..., N.
[0014] Furthermore, before training the deep learning network, the method further includes: determining that the feature size of each of the four stages of the deep learning network training is:
[0015] Phase 1:
[0016] Phase 2:
[0017] Phase 3: N1×C3;
[0018] Stage 4: N2×C4;
[0019] Among them, H and W are the height and width of the input image, C1 to C4 are the number of channels, and N1 and N2 are the sequence lengths.
[0020] Furthermore, the multi-granularity perception module includes:
[0021] The hyper-label attention module is used to perform dimensionality reduction, hyper-label mapping, and multi-head attention calculation on input features to capture global dependencies;
[0022] Multi-head convolution module, used to process input features in parallel through convolution kernels of different sizes to extract local multi-granularity features;
[0023] The feature fusion unit is used to add the output features of the two branches as the module output.
[0024] Furthermore, the specific operations of the super tag attention module include:
[0025] For input features Perform linear transformation and dimensionality reduction to obtain the transformed features ,in
[0026] Hyper-label mapping Calculate the super-label feature S t =(Q tT )X", where S t is the super-label feature of the previous layer, and t is the number of stacking layers;
[0027] After S is segmented along the channel dimension, multi-head attention calculation is performed to obtain the resulting feature group (H1, H2, ..., H h ), and then stack it along the channel to get the output result feature X MHSA ;
[0028] X MHSA After labeling downsampling and then dimensionality increase, the calculation result features of the super labeling attention module are obtained.
[0029] Furthermore, the specific operations of the multi-head convolution module include:
[0030] Input features Divide the channel into J parts, process each part with k×k convolution kernel, and output the convolution result.
[0031] After concatenating the convolution results along the channel, the 1×1 convolution aggregation is performed to obtain
[0032] Furthermore, the specific operations of the feature fusion unit include:
[0033] Will and After adding, it is used as the output of the multi-granularity perception module. The multi-granularity perception module is stacked 3 times to obtain the output of the first stage. The output of the second stage is
[0034] Furthermore, the residual attention module includes an image block embedding unit, a residual attention unit and a multi-layer perceptron, wherein the specific operations of the residual attention module include:
[0035] Performing image block embedding and position encoding on the second stage feature map X2 by the image block embedding unit to obtain input feature X2';
[0036] The input X2' is first normalized to obtain the feature X"2, and then the residual attention unit is used to calculate the feature based on the learnable parameter matrix Combined with the upper layer residual features Perform attention calculation to obtain features Update the residual features to
[0037] The output of the residual attention module is obtained through multi-layer perceptron processing Among them, MLP(·) represents multi-layer perceptron processing, LN(·) represents normalization processing, and in the third stage, the above calculation process is stacked three times to obtain the third stage feature map Fourth stage feature map
[0038] Furthermore, the classification prediction result is obtained by inputting X4 into the multi-layer perceptron for processing, including:
[0039] Calculate the predicted probability distribution through the SoftMax function: In the formula Represents the prediction result of feature X4;
[0040] Use the cross entropy loss function to optimize network parameters: Where Y i represents the true category of the sample, and N represents the total number of images in the dataset.
[0041] The beneficial effects of the present invention are:
[0042] The image classification method proposed in the present invention introduces a multi-granularity perception module in the shallow layer of the network, enabling the network to capture multi-scale features and expand the receptive field, thereby effectively avoiding the redundancy problem of local features; at the same time, a residual attention module is used in the deep layer of the network to fuse low-level features and retain its ability to extract global context, thereby enhancing the feature diversity in the deep network, improving the accuracy of endoscopic image classification, assisting doctors in diagnosis and reading, and reducing the pressure on endoscopists in reading films. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A schematic diagram of a process of the present invention;
[0044] Figure 2 This is a structural diagram of the deep learning network in the present invention;
[0045] Figure 3is a schematic diagram of a multi-granularity perception module of the present invention;
[0046] Figure 4 Schematic diagram of the residual attention module of the present invention. DETAILED DESCRIPTION
[0047] The present application is described in further detail below in conjunction with the accompanying drawings. It is necessary to point out that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technicians in this field can make some non-essential improvements and adjustments to the present application based on the above application content.
[0048] like Figure 1 As shown, this embodiment proposes an endoscopic image classification method based on multi-granularity perception and residual attention, characterized in that the method includes: obtaining a labeled N-th category endoscopic image; inputting the image into a pre-trained endoscopic image classifier to obtain an endoscopic image classification prediction result. The endoscopic image classifier is trained using a four-stage feature extraction based on a pre-established deep learning network. The first two stages consist of a 2×2 convolution and a multi-granularity perception module, and the last two stages consist of an image block embedding module and a residual attention module.
[0049] More specifically, the technical solution of the present application introduces a multi-granularity perception module in the shallow layer of the network, enabling the network to capture multi-scale features and expand the receptive field, thereby effectively avoiding the redundancy problem of local features; at the same time, a residual attention module is used in the deep layer of the network to fuse low-level features and retain its ability to extract global context, thereby enhancing the feature diversity in the deep network.
[0050] Correspondingly, in the feature extraction process of the above four stages, the multi-granularity perception module performs super-label attention and multi-head convolution calculations on the input features to output the first-stage feature map X1; X1 is processed again with multi-granularity perception to output the second-stage feature map X2; X2 is input into the residual attention module, and the third-stage feature map X3 is output through the residual attention calculation; X3 is calculated again with residual attention to output the fourth-stage feature map X4, and X4 is input into the multi-layer perceptron for processing to obtain the classification prediction result.
[0051] According to the above technical solution, the method, when implemented, includes training an endoscopic image classifier based on deep learning, and the steps are as follows:
[0052] S1. Obtain N types of endoscopic image samples of C×H×W categories, pre-process the samples to obtain a training set E, and use the training set E as the input feature; E={E1,E2,...,E n ,...,E N}; E n represents the nth class of endoscopic image samples, and there are P images in the nth class. represents the p-th image in the n-th type of preprocessed endoscopic image samples; C represents the image channel, H represents the image height, W represents the image width, and n = 1, 2, ..., N.
[0053] S2, such as Figure 2 As shown, a deep learning network is established and used to process the sample dataset of endoscopic images. The deep learning network uses four stages for feature extraction. The first two stages consist of a 2×2 convolution and a multi-granularity perception module, and the last two stages consist of an image block embedding module and a residual attention module. During the four-stage operation process, the feature size of each of the four stages of deep learning network training is determined as follows:
[0054] Phase 1:
[0055] Phase 2:
[0056] Phase 3: N1×C3;
[0057] Stage 4: N2×C4;
[0058] Among them, H and W are the height and width of the input image, C1 to C4 are the number of channels, and N1 and N2 are the sequence lengths.
[0059] S3, such as Figure 3 As shown in the figure, a multi-granularity perception module is constructed. The input image X undergoes super-label attention and multi-head convolution calculations in the multi-granularity perception module to obtain the feature map X1 output in the first stage. X1 then undergoes feature extraction in the multi-granularity perception module to obtain the feature map X2 output in the second stage.
[0060] The multi-granularity perception module includes:
[0061] The hyper-label attention module is used to perform dimensionality reduction, hyper-label mapping, and multi-head attention calculation on input features to capture global dependencies;
[0062] Multi-head convolution module, used to process input features in parallel through convolution kernels of different sizes to extract local multi-granularity features;
[0063] The feature fusion unit is used to add the output features of the two branches as the module output.
[0064] S3.1. Build a super-label attention module:
[0065] The left branch is the Super Tag Attention module. Before performing the Super Tag Attention calculation, the input image needs to go through a linear layer and dimensionality reduction. The purpose of dimensionality reduction is to convert the image feature dimensions into a dimension suitable for attention calculation. The linear layer and dimensionality reduction process can be expressed as follows:
[0066] X'=Linear(X)#(1)
[0067] X”=Reshape(AvgPool(X'))#(2)
[0068] In formula (1), Linear(·) represents linear processing, in formula (2), Reshape(·) represents dimension conversion operation, and AvgPool(·) represents average pooling operation. Represents the image input to the multi-granularity perception module, H represents the height of the image, W represents the width of the image, and C1 represents the number of channels of the image. in
[0069] Super tag attention is to mine global features as much as possible in shallow networks. For a given input feature Identify each mark of the feature All belong to m super tags So here we need to define a i ” to S’s mapping relation Q∈R N×m ,correspond Figure 3 The hyper-label calculation module in
[15] is then used. The output of the hyper-label calculation module is then multiplied by the input image to obtain the hyper-label feature. The calculation process can be expressed as follows:
[0070]
[0071] S t =(Q tT )X” #(4)
[0072] In formula (3), Q t ∈R N×m , t = 1, 2, 3 represents a multi-granularity perception module with 3 layers stacked in each of the first and second stages, where S 0 It is calculated by taking the weighted average of the markers in the 3×3 area around each marker.
[0073] The output of the super-label & label aggregation module is Then the multi-head attention calculation will be performed. t Split along the channel dimension to obtain feature groups Where h is a hyperparameter, and the result after feature group calculation (H1, H2, ..., H z ,…,H h );
[0074]
[0075] In formula (5)
[0076] is a learnable parameter matrix. Then, we use Equation (6) to concatenate along the channel dimension to obtain the result of multi-head attention calculation.
[0077] X MHSA =Concat(H1,H2,…,H h )#(6)
[0078] In formula (6), Concat(·) means stacking along the channel dimension;
[0079] Then the output of the multi-head attention Perform label downsampling and then perform a dimensionality increase operation to restore the image features to the previous dimension. The calculation process is:
[0080] X' sup =Q t X MHSA # (7)
[0081] X sup =Interpolate(Reshape(X' sup ))#(8)
[0082] In formula (8), Interpolate(·) represents the interpolation operation; Reshape(·) represents the dimension conversion operation.
[0083] S3.2. Build a multi-head convolution module:
[0084] Another branch of the multi-granularity perception module is the multi-head convolution module, which uses convolution kernels of different sizes to perform convolution processing on features. First, the input feature Perform linear processing and then divide it into J equal parts along the channel, where J is a hyperparameter. The characteristics of each part are Then, each feature is processed with a k×k convolution kernel, k=1,2,3,...,K, where K is a hyperparameter. After multi-head convolution processing, each feature x i You will get an output And these feature dimensions are of the same size. Then, padding is performed for operations with different convolution kernel sizes. The above process can be described as:
[0085] y i =Conv k×k (x i )#(9)
[0086] In formula (9), Conv k×k (·) represents a convolution operation with a kernel size of k×k.
[0087] Then each y i The output of is divided into M parts along the channel dimension, where each feature is Each y i There will be M Splicing in the same dimension can get M features, each feature size is Then, 1×1 convolution is used to convolve these M features separately, and the results are concatenated to obtain the output of the dimension aggregation module. The above process can be described by the following formula:
[0088]
[0089] X mhc =Concat(Conv 1×1 (y 1 ),…,Conv 1×1 (y M ))#(11)
[0090] S3.3, Fusion of two features:
[0091] After processing the above two branches, we can get and Then the two features are added together as the output of the multi-granularity perception module. The multi-granularity perception module is stacked three times to obtain the output of the first stage. The second stage output is similarly
[0092] S4, such as Figure 4 As shown, a residual attention module is constructed, and the feature map X2 obtained above is input into the residual attention module. After the residual attention calculation, the feature map X3 is output. X3 undergoes another residual attention calculation to obtain the output feature map X4.
[0093] For the third stage input First, we enter the image block embedding module, which consists of a convolution layer with a kernel size of 16×16 and a flattening operation. Then, we add a positional encoding to each token of the feature sequence to obtain the input of the residual attention module.
[0094] The residual attention module first performs normalization to obtain features. Then it sets a set of learnable parameter matrices Multiply them with X'2 to obtain three feature matrices Then the residual attention calculation is performed. The residual attention calculation needs to be combined with the residual features of the upper layer. The above calculation process can be described by the following formula:
[0095]
[0096] In formula (16) Represents the residual features of the residual attention module of the next layer, which is then normalized and the multi-layer perceptron is used to obtain the output of the residual attention module:
[0097]
[0098] In formula (17), MLP(·) represents multi-layer perceptron processing, LN(·) represents normalization processing, and the above calculation process is stacked three times in the third stage to obtain the output of the third stage Similarly, the output of the fourth stage
[0099] S5. Construct a classification module, input the feature map X4 into a multi-layer perceptron, obtain a prediction result after processing, and thus classify the endoscopic image.
[0100] The output of the fourth stage is Use the multi-layer perceptron to obtain the final prediction result of the model:
[0101]
[0102] In formula (18) Represents the prediction result of feature X4.
[0103] The cross entropy loss function is used to calculate the cross entropy loss between the predicted results and the true labels.
[0104]
[0105] In formula (19), Y iRepresents the true category of the sample, and N represents the total number of images in the dataset. The network backpropagation loss is used to optimize the parameters and an endoscopic image classifier is obtained for endoscopic image classification.
[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0107] In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0108] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An endoscopic image classification method based on multi-granularity perception and residual attention, characterized in that: The method comprises: Obtaining the labeled Nth type of endoscopic image; Inputting the image into a pre-trained endoscopic image classifier to obtain an endoscopic image classification prediction result; The endoscopic image classifier is trained using four stages of feature extraction based on a pre-established deep learning network. The first two stages consist of a 2×2 convolution and a multi-granularity perception module, and the last two stages consist of an image patch embedding module and a residual attention module. In the feature extraction process of the four stages, the multi-granularity perception module performs super-label attention and multi-head convolution calculations on the input features to output the first-stage feature map X1; X1 is processed again with multi-granularity perception to output the second-stage feature map X2; X2 is input into the residual attention module, and the third-stage feature map X3 is output through residual attention calculation; X3 is calculated again with residual attention to output the fourth-stage feature map X4, and X4 is input into the multi-layer perceptron for processing to obtain the classification prediction result.
2. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 1 is characterized in that: Before training the deep learning network, the method further includes: Obtain N types of endoscopic image samples of C×H×W categories, preprocess the samples to obtain a training set E, and use the training set E as the input feature; E={E1,E2,...,E n ,...,E N }; E n represents the nth class of endoscopic image samples, and there are P images in the nth class. represents the p-th image in the n-th type of preprocessed endoscopic image samples; C represents the image channel, H represents the image height, W represents the image width, and n = 1, 2, ..., N.
3. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 2 is characterized in that: Before training the deep learning network, the method further includes: determining the feature size of each of the four stages of deep learning network training to be: Phase 1: Phase 2: Phase 3: N1×C3; Stage 4: N2×C4; Among them, H and W are the height and width of the input image, C1 to C4 are the number of channels, and N1 and N2 are the sequence lengths.
4. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 3 is characterized in that: The multi-granularity perception module includes: The hyper-label attention module is used to perform dimensionality reduction, hyper-label mapping, and multi-head attention calculation on input features to capture global dependencies; Multi-head convolution module, used to process input features in parallel through convolution kernels of different sizes to extract local multi-granularity features; The feature fusion unit is used to add the output features of the two branches as the module output.
5. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 4 is characterized in that: The specific operations of the super tag attention module include: For input features Perform linear transformation and dimensionality reduction to obtain the transformed features in Hyper-label mapping Calculate the super-label feature S t =(Q tT )X", where S t is the super-label feature of the previous layer, and t is the number of stacking layers; After S is segmented along the channel dimension, multi-head attention calculation is performed to obtain the resulting feature group (H1, H2, ···, H h ), and then stack it along the channel to get the output result feature X MHSA ; X MHSA After labeling downsampling and then dimensionality increase, the calculation result features of the super labeling attention module are obtained.
6. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 5 is characterized in that: The specific operations of the multi-head convolution module include: Input features Divide the channel into J parts, process each part with k×k convolution kernel, and output the convolution result. After concatenating the convolution results along the channel, the 1×1 convolution aggregation is performed to obtain 7. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 6 is characterized in that: The specific operations of the feature fusion unit include: Will and After adding, it is used as the output of the multi-granularity perception module. The multi-granularity perception module is stacked 3 times to obtain the output of the first stage. The output of the second stage is 8. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 1 is characterized in that: The residual attention module includes an image block embedding unit, a residual attention unit and a multi-layer perceptron, wherein the specific operations of the residual attention module include: Performing image block embedding and position encoding on the second stage feature map X2 by the image block embedding unit to obtain input feature X2'; The input X2' is first normalized to obtain the feature X"2, and then the residual attention unit is used to calculate the feature based on the learnable parameter matrix Combined with the upper layer residual features Perform attention calculation to obtain features X'2; Update the residual feature to The output of the residual attention module is obtained through multi-layer perceptron processing Among them, MLP(·) represents multi-layer perceptron processing, LN(·) represents normalization processing, and in the third stage, the above calculation process is stacked three times to obtain the third stage feature map Fourth stage feature map 9. The endoscopic image classification method based on multi-granularity perception and residual attention according to claim 8 is characterized in that: The classification prediction result obtained by inputting X4 into the multi-layer perceptron for processing includes: Calculate the predicted probability distribution through the SoftMax function: In the formula Represents the prediction result of feature X4; Use the cross entropy loss function to optimize network parameters: Where Y i represents the true category of the sample, and N represents the total number of images in the dataset.