Medical image lesion detection algorithm with scale enhancement and attention fusion
By introducing scale enhancement and attention fusion into the medical image lesion detection algorithm, and using larger convolutional kernels and attention mechanisms to fuse feature information, the accuracy and efficiency problems of medical image lesion detection are solved, and more efficient lesion localization is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEBEI UNIV OF TECH
- Filing Date
- 2022-01-24
- Publication Date
- 2026-05-29
AI Technical Summary
Existing medical image lesion detection algorithms are insufficient in terms of accuracy and efficiency, especially due to the differences between medical images and natural images, resulting in poor lesion detection performance.
A detection algorithm with scale enhancement and attention fusion is adopted. Features are extracted through residual network, and scale enhancement and attention mechanism with larger convolutional kernel are combined to fuse feature information of different scales, generate candidate boxes and perform classification and regression correction.
It improves the accuracy and efficiency of lesion detection in medical images, enables more accurate localization of lesion edges, and has end-to-end portability.
Smart Images
Figure CN114612381B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image lesion detection technology, specifically a medical image lesion detection algorithm with scale enhancement and attention fusion. Background Technology
[0002] Medical imaging lesion detection plays a crucial role in clinical diagnosis and treatment based on computer-aided diagnostic systems. For example, in the initial screening and localization of tumors using imaging, intelligent lesion detection systems can quickly determine, identify, detect, and locate tumors. Furthermore, lesion localization in lesion detection can be applied to other intelligent assisted diagnostic tasks, such as lesion segmentation. Localization through lesion detection can narrow down the segmentation area, thereby improving segmentation accuracy. Currently, given the complexity of clinical medical images and the highly experience-dependent nature of image interpretation and analysis, fully end-to-end automated medical image interpretation and analysis technology is still immature. It primarily relies on physicians with extensive professional experience to determine the location of lesions in images. Moreover, due to objective factors such as the imaging instruments used, the imaging environment, and physician fatigue, as well as the physician's subjective experience, the analysis results for the same case (especially for smaller lesions) may often show some deviation. Therefore, achieving efficient and accurate end-to-end automated detection of clinical medical images has significant theoretical value and socio-economic benefits.
[0003] In recent years, convolutional neural networks have achieved excellent image detection performance due to their superior feature extraction capabilities in Euclidean space. Medical image detection based on deep neural networks refers to using deep learning methods such as deep convolutional neural networks to learn target features in medical images and locate target organs, tissues, or lesions such as inflammation, cysts, and tumors.
[0004] Current deep neural network object detection methods for natural images are mainly divided into two categories: two-stage detection networks (such as R-CNN, Fast-RCNN, and Faster-RCNN) and single-stage detection networks (such as YoLo and RetinaNet). Both two-stage and single-stage networks first extract features from medical images using backbone networks such as VGG or ResNet. The difference lies in that two-stage networks first perform preliminary classification and localization of the extracted features, and then further refine the localization of features within the localized region for more accurate localization. Single-stage networks, on the other hand, skip the preliminary classification and localization steps and directly classify and localize the extracted features. Obviously, two-stage networks, due to the preliminary localization step, are slower but more accurate. However, lesion detection in medical images presents significant differences and challenges compared to natural images. For example, medical images lack the rich colors of natural images, and lesions are not easily distinguishable from the background. This results in lesion detection performance in medical images not being as good as in natural images.
[0005] Therefore, a medical image lesion detection algorithm with scale enhancement and attention fusion is needed to overcome the difficulty of detecting lesions in medical images compared to natural images. Summary of the Invention
[0006] In order to overcome the shortcomings of the prior art, this invention provides a medical image lesion detection algorithm with scale enhancement and attention fusion, which effectively solves the problems mentioned in the background art.
[0007] To achieve the above objectives, the present invention provides the following technical solution: a medical image lesion detection algorithm with scale enhancement and attention fusion, comprising the following steps:
[0008] S1: Case data loading and preprocessing: Standard annotation of medical imaging data and lesion localization information;
[0009] S2: Feature Extraction: The existing residual network is used to extract features from the input raw data, and the feature information of the last 5 layers is retained as the initial feature information;
[0010] S3: Weight generation: Perform max pooling, 2D convolution, normalization and ReLU activation operations on the feature maps of each of the 5 feature maps generated in step S2, and finally generate the weights of each feature map of different scales for that layer.
[0011] S4: Scale Enhancement: Convolve the feature map generated in step S2 with convolution kernels of different sizes to obtain two feature maps of the same size but different scales;
[0012] S5: Feature fusion: Multiply the existing features at different scales and their corresponding weights separately and then sum them;
[0013] S6: Obtain candidate boxes: The final feature map is fed into the region generation network to obtain a series of candidate boxes;
[0014] S7: Classification and Regression: The features in the candidate boxes obtained in step S6 are classified and regressed again, and a score is generated for each predicted box.
[0015] S8: Output prediction: By setting a threshold for the score parameter generated in step S7, prediction boxes that exceed this threshold will be used as the final prediction. For the test part, there will be no standard annotation in step S1, and the prediction result will be directly output as the prediction.
[0016] Preferably, step S1, case data loading and preprocessing, includes the following steps:
[0017] S11: Collect and acquire a database of cases for the target diagnostic condition;
[0018] S12: Name the image folders of each patient in the database of step S11 with ordered numbers, and generate corresponding XML format annotation files for lesion localization with the same names.
[0019] Preferably, the residual network in step S2 is a ResNet network.
[0020] Preferably, the feature information of the last 5 layers in step S2 consists of feature information of different semantic depths and different scales.
[0021] Preferably, the feature information sizes are 4×4×256, 8×8×256, 16×16×256, 32×32×256, and 64×64×256, respectively.
[0022] Preferably, the scale enhancement step S4 further includes the following steps:
[0023] S41: Enhances multi-scale details of feature maps;
[0024] S42: Perform single-scale contrast stretching on the feature map.
[0025] Preferably, the region generation network in step S6 is an RPN network.
[0026] Preferably, step S6, generating candidate boxes, includes the following steps:
[0027] S61: For each point on the feature map generated in step S5, generate anchor boxes with different scales and aspect ratios.
[0028] S62: Input the anchor box from step S61 into the rpn_cls_score network layer for classification, determine whether the feature map inside the anchor box belongs to the foreground, and input it into the rpn_bbox_pred network layer to output four position coordinates (offsets relative to the real object box), and then use GT to learn the loss.
[0029] S63: Compare the IoU of the anchor box with the label box in the Ground Truth. If the IoU is higher than a certain threshold, the anchor box is labeled as a foreground box; otherwise, it is a background box. For the foreground box, calculate its four positional offsets from the real label box. Compare the loss of this labeled anchor box (with foreground / background category and positional offset labels) with the two outputs of the convolutional network layer (category: CrossEntry loss and positional regression: smooth L1 loss) to learn how to extract the foreground box.
[0030] S64: The foreground box is determined based on the output probability value of the rpn_cls_score layer, and the position offset value is integrated into the coordinates of the anchor box to obtain the actual box coordinates, thus obtaining the candidate box.
[0031] S65: Based on the candidate boxes obtained in step S64, classify the features in the candidate boxes and regress to correct the position of the boxes as the final prediction.
[0032] Compared with the prior art, the beneficial effects of the present invention are:
[0033] (1) The medical image lesion detection algorithm of the present invention adds a scale enhancement module to the existing feature pyramid network (FPN), and adds an attention mechanism to assign different weights to different scales in the fusion process after scale enhancement. The scale enhancement module adds a larger convolutional kernel, which can contain a wider range of information for fusion. This is crucial for medical images, because the lesion boundaries in medical images are often not as clear as those in natural images. This leads to inaccurate edge localization of lesions when the network is localizing. The addition of a larger convolutional kernel can fuse a wider range of feature information, which is conducive to more accurate lesion localization. The weights given by the attention mechanism can extract more feature information that is more conducive to classification and regression from features of different scales, thereby improving the accuracy of lesion detection. With the support of scale enhancement and attention mechanism, more accurate detection accuracy can be obtained for lesion detection in medical images.
[0034] (2) The network structure designed by the medical image lesion detection algorithm of the present invention can be trained end-to-end and has good portability. Attached Figure Description
[0035] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0036] In the attached diagram:
[0037] Figure 1 This is a flowchart of a medical image lesion detection algorithm with scale enhancement and attention fusion according to an embodiment of the present invention;
[0038] Figure 2 This is a schematic diagram of max pooling in a medical image lesion detection algorithm with scale enhancement and attention fusion according to an embodiment of the present invention;
[0039] Figure 3 This is a schematic diagram of two-dimensional convolution in a medical image lesion detection algorithm with scale enhancement and attention fusion according to an embodiment of the present invention;
[0040] Figure 4 This is a schematic diagram of normalization in a medical image lesion detection algorithm with scale enhancement and attention fusion according to an embodiment of the present invention;
[0041] Figure 5 This is a schematic diagram of the ReLU function in a medical image lesion detection algorithm with scale enhancement and attention fusion according to an embodiment of the present invention;
[0042] Figure 6 This is a flowchart illustrating the generation of candidate bounding boxes in a medical image lesion detection algorithm with scale enhancement and attention fusion according to an embodiment of the present invention. Detailed Implementation
[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0044] According to an embodiment of the present invention, a medical image lesion detection algorithm with scale enhancement and attention fusion is provided, comprising the following steps:
[0045] S1: Case data loading and preprocessing: Standard annotation of medical imaging data and lesion localization information;
[0046] S2: Feature Extraction: The existing residual network is used to extract features from the input raw data, and the feature information of the last 5 layers is retained as the initial feature information;
[0047] S3: Weight generation: Perform max pooling, 2D convolution, normalization and ReLU activation operations on the feature maps of each of the 5 feature maps generated in step S2, and finally generate the weights of each feature map of different scales for that layer.
[0048] S4: Scale Enhancement: Convolve the feature map generated in step S2 with convolution kernels of different sizes to obtain two feature maps of the same size but different scales;
[0049] S5: Feature fusion: Multiply the existing features at different scales and their corresponding weights separately and then sum them;
[0050] S6: Obtain candidate boxes: The final feature map is fed into the region generation network to obtain a series of candidate boxes;
[0051] S7: Classification and Regression: The features in the candidate boxes obtained in step S6 are classified and regressed again, and a score is generated for each predicted box.
[0052] S8: Output prediction: By setting a threshold for the score parameter generated in step S7, prediction boxes that exceed this threshold will be used as the final prediction. For the test part, there will be no standard annotation in step S1, and the prediction result will be directly output as the prediction.
[0053] In a preferred embodiment, step S1, case data loading and preprocessing, includes the following steps:
[0054] S11: Collect and acquire a database of cases for the target diagnostic condition;
[0055] S12: Name the image folders of each patient in the database of step S11 with ordered numbers, and generate corresponding XML format annotation files for lesion localization with the same names.
[0056] In a preferred embodiment, the residual network in step S2 is a ResNet network.
[0057] In a preferred embodiment, the feature information of the last 5 layers in step S2 consists of feature information of different semantic depths and different scales.
[0058] In a preferred embodiment, the feature information sizes are 4×4×256, 8×8×256, 16×16×256, 32×32×256, and 64×64×256, respectively.
[0059] In a preferred embodiment, step S4 scale enhancement further includes the following steps:
[0060] S41: Enhances multi-scale details of feature maps;
[0061] S42: Perform single-scale contrast stretching on the feature map.
[0062] In a preferred embodiment, the region generation network in step S6 is an RPN network.
[0063] In a preferred embodiment, step S6, generating candidate boxes, includes the following steps:
[0064] S61: For each point on the feature map generated in step S5, generate anchor boxes with different scales and aspect ratios.
[0065] S62: Input the anchor box from step S61 into the rpn_cls_score network layer for classification, determine whether the feature map inside the anchor box belongs to the foreground, and input it into the rpn_bbox_pred network layer to output four position coordinates (offsets relative to the real object box), and then use GT to learn the loss.
[0066] S63: Compare the IoU of the anchor box with the label box in the Ground Truth. If the IoU is higher than a certain threshold, the anchor box is labeled as a foreground box; otherwise, it is a background box. For the foreground box, calculate its four positional offsets from the real label box. Compare the loss of this labeled anchor box (with foreground / background category and positional offset labels) with the two outputs of the convolutional network layer (category: CrossEntry loss and positional regression: smooth L1 loss) to learn how to extract the foreground box.
[0067] S64: The foreground box is determined based on the output probability value of the rpn_cls_score layer, and the position offset value is integrated into the coordinates of the anchor box to obtain the actual box coordinates, thus obtaining the candidate box.
[0068] S65: Based on the candidate boxes obtained in step S64, classify the features in the candidate boxes and regress to correct the position of the boxes as the final prediction.
[0069] To facilitate understanding of the above technical solutions of the present invention, the flow of the above solutions of the present invention will be described in detail below with reference to the accompanying drawings, as follows:
[0070] According to embodiments of the present invention, a medical imaging lesion detection network based on scale enhancement with an attention mechanism is provided, such as... Figure 1 As shown, it includes the following steps:
[0071] Step S101: Case data loading and preprocessing: Obtain standard annotations for medical imaging data and lesion localization information;
[0072] Specifically, a database of cases for the target diagnosis is first collected. The image folders for each patient in the database are named with sequential numbers, and corresponding XML annotation files with the same names are generated for lesion localization. The localization information is in coordinate format, specifically the upper left (xmin, ymin) and lower right (xmax, ymax) corners of the lesion localization frame, as shown below:
[0073] - <annotation>
[0074] <folder> VOC2007< / folder>
[0075] <filename> BraTS19_2013_2_1-30.jpg< / filename>
[0076] - <source>
[0077] <database> The UAV autolanding< / database>
[0078] <annotation> UAV AutoLanding< / annotation>
[0079] flickr
[0080] <flickrid> NULL< / flickrid>
[0081]
[0082] - <owner>
[0083] <flickrid> NULL< / flickrid>
[0084] <name> ChaojieZhu< / name>
[0085] < / owner>
[0086] - <size>
[0087] <width> 240< / width>
[0088] <height> 240< / height>
[0089] <depth> 3< / depth>
[0090] < / size>
[0091] <segmented> 0< / segmented>
[0092] - <object>
[0093] <name> lesion< / name>
[0094] <pose> Unspecified< / pose>
[0095] <truncated> 1< / truncated>
[0096] <difficult> 0< / difficult>
[0097] - <bndbox>
[0098] <xmin> 80< / xmin>
[0099] <ymin> 93< / ymin>
[0100] <xmax> 169< / xmax>
[0101] <ymax> 142< / ymax>
[0102] < / bndbox>
[0103] < / object>
[0104] < / annotation> .
[0105] Step S103: Feature extraction: The existing residual network is used to extract features from the input raw data, and the feature information of the last 5 layers is retained as the initial feature information. The feature information of the last 5 layers are feature information of different semantic depths and different scales, with sizes of 4×4×256, 8×8×256, 16×16×256, 32×32×256, and 64×64×256, respectively.
[0106] Step S105: Perform max pooling, 2D convolution, normalization and ReLU activation operations on the feature maps of each of the 5 feature maps generated in step S103 to finally generate the weights of each feature map of different scales for that layer. The top layer has only two scales of features, while the four layers below have three scales of features. Therefore, the first layer generates a weight of size 2×256 and the four layers below generate a weight of size 3×256.
[0107] Step S107: Scale enhancement: Each feature map in the 5-layer feature map generated in step S103 will be convolved with two different sizes of convolution kernels, 3×3 and 5×5, to obtain two feature maps of the same size.
[0108] Specifically, the convolution is configured as follows:
[0109] torch.nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=1, padding=1),
[0110] torch.nn.Conv2d(in_channels=256, out_channels=256, kernel_size=5, stride=1, padding=2).
[0111] Step S109: Feature fusion: Multiply the feature information of the last layer with only two different scales after scale enhancement and the feature information of the other several layers with three different scales (which are the feature information of the two different scales after scale enhancement and the feature information of the next layer after upsampling) with the weights obtained in step S105 and then add them together to perform feature fusion, and obtain the final feature map used for classification and regression localization.
[0112] Step S111: Generate candidate boxes: A series of candidate boxes are generated by the region generation network. The feature map generated in step S109 is fed into the region generation network to obtain multiple preliminary candidate boxes.
[0113] Step S113: Classification and Regression Localization: The features in the candidate boxes are classified and regressed using a fully connected form to obtain the final predicted boxes, and a score is generated for each predicted box.
[0114] Step S115: Output prediction: By setting a threshold for the score parameter generated in step S113, prediction boxes that exceed this threshold will be used as the final prediction. For the test part, there will be no standard annotation in step S101, and the prediction result will be directly output as the prediction.
[0115] In summary, by utilizing the above-mentioned technical solution of this invention, this invention mainly adopts a two-stage network structure and introduces scale enhancement to achieve the fusion of multi-scale information. Because a larger convolutional kernel is used in the scale enhancement process, the network can fuse a wider range of feature information. This is beneficial for extracting more lesion features in complex backgrounds, thus aiding in classification and localization. Furthermore, after scale enhancement, this method incorporates an attention mechanism to capture more advantageous features for classification and localization from feature information at different scales. The neural network mainly adopts the main structure of Faster-RCNN for lesion detection. This network uses a ResNet network as the backbone to extract features, with a feature pyramid network as the main architecture. A feature enhancement module with an attention mechanism is added to the horizontal channels of the feature pyramid. After feature enhancement, the attention mechanism is used to perform weighted fusion of different scales. The fused features are then used as the final feature input to two parallel fully connected layers for classification and localization. Specifically, the ResNet network uses a depth of 50... The ResNet-50 network is used, and the feature maps of the last five layers (4×4×256, 8×8×256, 16×16×256, 32×32×256, and 64×64×256 respectively) are taken as input to the feature pyramid network. These features from the top to the bottom five layers are all enhanced. The feature enhancement of each layer uses two convolutional kernels of different sizes (one is 3×3 and the other is 5×5) to extract features and obtain two feature maps of the same size. Finally, the two feature maps of each layer are fused with the feature map of the previous layer after upsampling. An attention mechanism is used in the fusion process. Specifically, convolution, pooling, and activation operations are performed on the input features of each layer to obtain the weights of the feature maps of different scales of that layer. As training progresses, backpropagation optimizes these weights. Thus, the features of different scales of each layer and the corresponding weights of the features of different scales are obtained. Finally, the features of different scales are multiplied by their corresponding weights and then added together to obtain the final feature map of each layer. For the feature map of each layer, two fully connected neural networks are used for classification and regression localization.
[0116] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0117] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for detecting lesions in medical images with scale enhancement and attention fusion, characterized in that, Includes the following steps: S1: Case data loading and preprocessing: Standard annotation of medical imaging data and lesion localization information; S2: Feature Extraction: The existing residual network is used to extract features from the input raw data, and the feature information of the last 5 layers is retained as the initial feature information; S3: Weight generation: Perform max pooling, 2D convolution, normalization and ReLU activation operations on the feature maps of each of the 5 feature maps generated in step S2, and finally generate the weights of each feature map of different scales for that layer. S4: Scale Enhancement: Convolve the feature map generated in step S2 with convolution kernels of different sizes to obtain two feature maps of the same size but different scales; S5: Feature fusion: Multiply the existing features at different scales and their corresponding weights separately and then sum them; S6: Obtain candidate boxes: The final feature map is fed into the region generation network to obtain a series of candidate boxes; S7: Classification and Regression: The features in the candidate boxes obtained in step S6 are classified and regressed again, and a score is generated for each predicted box. S8: Output prediction: By setting a threshold for the score parameter generated in step S7, prediction boxes that exceed this threshold will be used as the final prediction. For the test part, there will be no standard annotation in step S1, and the prediction result will be directly output as the prediction.
2. The method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 1, characterized in that, Step S1, case data loading and preprocessing, includes the following steps: S11: Collect and acquire a database of cases for the target diagnostic condition; S12: Name the image folders of each patient in the database of step S11 with ordered numbers, and generate corresponding XML format annotation files for lesion localization with the same names.
3. The method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 1, characterized in that, In step S2, the residual network is a ResNet network.
4. The method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 1, characterized in that, The feature information in the last 5 layers of step S2 consists of feature information with different semantic depths and scales.
5. A method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 4, characterized in that, The feature information sizes are 4×4×256, 8×8×256, 16×16×256, 32×32×256, and 64×64×256, respectively.
6. The method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 1, characterized in that, The scale enhancement step S4 also includes the following steps: S41: Enhances multi-scale details of feature maps; S42: Perform single-scale contrast stretching on the feature map.
7. The method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 1, characterized in that, In step S6, the region generation network is an RPN network.
8. The method for detecting lesions in medical images with scale enhancement and attention fusion according to claim 1, characterized in that, Step S6, generating candidate boxes, includes the following steps: S61: For each point on the feature map generated in step S5, generate anchor boxes with different scales and aspect ratios. S62: Input the anchor box from step S61 into the rpn_cls_score network layer for classification, determine whether the feature map inside the anchor box belongs to the foreground, and input it into the rpn_bbox_pred network layer to output four position coordinates, and then use GT to learn the loss. S63: Compare the IoU of the anchor box with the label box in the Ground Truth. If the IoU is higher than a certain threshold, the anchor box is marked as a foreground box; otherwise, it is a background box. For the foreground box, calculate its four positional offsets from the real label box. Compare the loss of this marked anchor box with the two outputs of the convolutional network layer to learn how to extract the foreground box. S64: The foreground box is determined based on the output probability value of the rpn_cls_score layer, and the position offset value is integrated into the coordinates of the anchor box to obtain the actual box coordinates, thus obtaining the candidate box. S65: Based on the candidate boxes obtained in step S64, classify the features in the candidate boxes and regress to correct the position of the boxes as the final prediction.