Layered multi-scale feature merging fusion analysis method for endoscope focus area perception
Through the hierarchical multi-scale feature fusion analysis method of endoscopic lesion area perception, the U-shaped network and multi-layer supervision mechanism are used to solve the problem of high lesion area miss detection rate in endoscopic image processing, and the multi-scale feature is realized The effective extraction and fusion of the system improves detection accuracy and robustness.
Patent Information
- Application Number
- CN202411152172.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-13
AI Technical Summary
The existing endoscopic image processing technology has high missed detection rate in the lesion area, difficulty in effectively extracting and fusion of multi-scale features, and difficulty in balancing detailed information and semantic information.
The hierarchical multi-scale feature fusion and fusion analysis method of endoscopic lesions is adopted, and feature extraction is performed through U-shaped network architecture and P2T network, and feature fusion is performed using unbalanced information aggregation module and multi-scale information aggregation module, and multi-layered The supervision mechanism implements layer by layer loss constraints and generates prediction information at different scales.
Effectively reduce missed detection rates, improve the detection rate of early cancer screening, enhance the model's perception of complex lesion areas, avoid gradient disappearance problems, and improve detection accuracy and robustness.
Smart Images
Figure CN119131351A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception. Background Art
[0002] Endoscopy is a minimally invasive diagnostic and treatment method widely used in modern medicine, especially in the screening and treatment of internal human diseases. Endoscopic equipment uses a camera to obtain real-time images of the human body, allowing doctors to directly observe the condition of the patient's inner wall, thereby detecting and treating potential lesions. Gastroscopy and colonoscopy are the most common items in digestive endoscopy applications, and are widely used for screening of early gastric cancer and colorectal cancer. Early detection of these two cancers is crucial to the patient's prognosis.
[0003] Although the widespread use of endoscopic technology has effectively improved the detection rate of internal diseases in the human body, the results are still highly dependent on the experience and skills of the operating physician. Due to the complexity of the lesion area and changes in image quality during endoscopic examination, missed detection may occur. Missed detection of key lesions may not only delay diagnosis, but may also lead to canceration in the lesion area, which in turn seriously threatens the life and health of patients.
[0004] In order to solve this problem, computer-aided diagnosis (CAD) technology has developed rapidly in recent years. In particular, the application of deep learning-based algorithms in medical image processing has significantly improved the accuracy of disease detection. CAD technology can assist doctors in quickly and accurately identifying lesion areas during endoscopic examinations, thereby reducing the missed detection rate. However, existing CAD systems still face multiple challenges when processing endoscopic images, including the effective extraction of features at different scales, the fusion of multi-level information, and the balance between detail information and semantic information.
[0005] Traditional convolutional neural networks (CNNs) have achieved certain success in medical image processing, but their limitation is that they are difficult to take into account both global semantic information and local detail information at the same time. In addition, with the improvement of endoscopic image resolution, how to process multi-scale features and fuse information at all levels has become a key challenge to further improve detection accuracy. Summary of the invention
[0006] In order to solve the technical problems in the prior art of high missed detection rate of lesion areas, difficulty in effectively extracting and fusing multi-scale features, and difficulty in balancing detail information and semantic information during endoscopic image processing, the present invention provides a hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception.
[0007] The technical solution provided by the present invention is as follows:
[0008] The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception provided by the present invention comprises:
[0009] S1. Acquire internal human body image data through an endoscope device, wherein the data includes multi-scale image feature information;
[0010] S2. Using a U-shaped network architecture, a P2T network is selected as a backbone network for feature extraction, wherein the P2T network includes feature maps of shallow features and deep features;
[0011] S3, the features of adjacent layers are fused step by step through the non-balanced information aggregation module (NBIA), the NBIA module upsamples the deep features and guides the fusion of shallow features, and the channel attention mechanism is realized by using the mechanism of global average pooling (GAP) and convolution in series, so as to optimize the fusion effect;
[0012] S4. The fused features are further extracted and fused using a multi-scale information aggregation module (MIA). The MIA module uses convolution kernels of different sizes to extract multi-scale features, and fuses features of different receptive fields through trainable weight parameters. At the same time, a channel attention mechanism is used to redistribute weights to enhance the fusion perception of features at all levels.
[0013] S5. Through a multi-layer supervision mechanism, the features are constrained layer by layer to generate prediction information of different scales to achieve accurate perception and extraction of the lesion area. The loss function of each layer includes weighted cross-link loss (IoU loss) and binary cross entropy loss (BEC loss) to optimize the performance of the model at each level and optimize the prediction performance of the lesion area.
[0014] The beneficial effects brought about by the technical solution provided by the present invention include at least:
[0015] (1) In the present invention, by adopting a hierarchical multi-scale feature merging and fusion analysis method, the present invention can more accurately identify the lesion area in the endoscopic image, effectively reduce the missed detection rate, and especially greatly improve the detection rate in early cancer screening, ensuring timely diagnosis and treatment of patients;
[0016] (2) In the present invention, the non-balanced information aggregation module (NBIA) and the multi-scale information aggregation module (MIA) are used to realize the effective extraction and fusion of features at different levels and scales, so that the network can take into account both global semantic information and local detail information at the same time, thereby improving the model's perception ability of complex lesion areas;
[0017] (3) In the present invention, through the optimization of multi-layer supervision mechanism and loss function, the present invention effectively avoids the gradient vanishing problem in deep networks and improves the stability of the training process, so that the model can be optimized evenly at different levels, thereby improving the overall detection accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0019] Figure 1 A schematic flow chart of a hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion region perception provided by an embodiment of the present invention;
[0020] Figure 2 A schematic diagram of the overall network structure of the hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion region perception provided by an embodiment of the present invention;
[0021] Figure 3 A structural diagram of the NBIA module of the hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion region perception provided by an embodiment of the present invention;
[0022] Figure 4 The MIA module structure diagram of the hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0024] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0025] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0026] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.
[0027] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0028] Reference Manual Attached Figure 1 , showing a flow chart of a hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception provided by an embodiment of the present invention.
[0029] The embodiment of the present invention provides a hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion region perception, and the processing flow may include the following steps:
[0030] S1. Obtaining internal human body image data through an endoscope, the data including multi-scale image feature information;
[0031] S2, using the U-shaped network architecture, select the P2T network as the backbone network for feature extraction, the P2T network includes feature maps of shallow features and deep features;
[0032] S3, the features of adjacent layers are fused step by step through the non-balanced information aggregation module (NBIA). The NBIA module upsamples the deep features and guides the fusion of shallow features. The channel attention mechanism is realized by using the global average pooling (GAP) and convolution cascade mechanism to optimize the fusion effect.
[0033] S4. The fused features are further extracted and fused using the multi-scale information aggregation module (MIA). The MIA module uses convolution kernels of different sizes to extract multi-scale features, and fuses features of different receptive fields through trainable weight parameters. At the same time, the channel attention mechanism is used to redistribute weights to enhance the fusion perception of features at all levels.
[0034] S5. Through a multi-layer supervision mechanism, the features are constrained layer by layer to generate prediction information of different scales to achieve accurate perception and extraction of the lesion area. The loss function of each layer includes weighted cross-link loss (IoU loss) and binary cross entropy loss (BEC loss) to optimize the performance of the model at each level and optimize the prediction performance of the lesion area.
[0035] It should be noted that the backbone network specifically includes:
[0036] The backbone network is replaced with other convolutional neural networks (CNNs) or Transformer networks to adapt to different endoscopic image feature extraction requirements.
[0037] It should be noted that NBIA specifically includes:
[0038] NBIA is used to fuse shallow features and deep features. The deep features are upsampled to guide the shallow features. The global average pooling and convolution of the deep features are used in series to guide the fusion of the shallow features.
[0039] It should be noted that MIA specifically includes:
[0040] MIA uses convolution kernels of different sizes for multi-scale feature extraction and redistributes the weights of feature maps through the channel attention mechanism. The convolution kernels include 3×3, 5×5 and 7×7.
[0041] It should be noted that the channel attention mechanism specifically includes:
[0042] Linear transformation is performed through convolution, and the weight coefficients are normalized using the sofrmax function. Finally, the training process is stabilized through ReLU activation and LayerNormalization (LN).
[0043] It should be noted that the loss function specifically includes:
[0044] The loss function is a combination of weighted inter-connection loss (IoU loss) and binary cross entropy loss (BCE loss), which is used to optimize the network MI and prevent the deviation of predicted region shape and pixel probability distribution.
[0045] It should be noted that the loss function specifically includes:
[0046] The IoU loss is used to optimize the shape of the predicted region, and the BCE loss is used to stabilize the pixel-level probability distribution prediction during the training process.
[0047] It should be noted that the total network loss is calculated by the joint constraints of the losses of each layer to ensure balanced optimization of the network at each level.
[0048] It should be noted that the shallow features and deep features of the NBIA module are aligned in width and height dimensions. The width and height dimensions of the deep features are half of those of the shallow features, and unbalanced fusion is achieved through a dual-branch feature extraction strategy.
[0049] It should be noted that the multi-layer supervision mechanism generates multi-scale prediction information through layer-by-layer loss constraints, and finally completes the extraction and analysis of the lesion area through joint fusion.
[0050] In a possible implementation, the overall network structure is as follows: Figure 2 As shown, the network uses the multi-layer features of the backbone network as the feature source. The P2T network is selected as the backbone network, and the backbone network can also be changed according to needs. The different scale features of the P2T network are recorded as x1, x2, x3, and x4, and the features are from shallow to deep (the P2T network is not the only choice as the backbone network, and other typical networks such as CNN and Transformer can also be selected as backbone networks to extract visual features of different degrees). Shallow features have higher resolution and contain more position and detail information, but their semantic content is weaker and contains more noise. In contrast, although the deep features have better semantic expression, their ability to capture detail information is limited. Based on the characteristics of features at all levels, the algorithm of the embodiment of the present invention hierarchically and additively fuses the features of adjacent layers through a non-balanced information aggregation module (Non-Balanced Information Aggregation, NBIA), and the corresponding fused features are further fused through a multi-scale information aggregation module (Multi-scale Information Aggregation, MIA) to extract key supporting features, where the deep features are processed by MIA after a secondary NBIA. In order to make full use of the multi-scale feature information of each clinical area, the deep features are additively fused step by step, and the prediction information of different scales is generated through layer-by-layer loss constraints. Finally, the polyp area is extracted through joint fusion.
[0051] NBIA structure diagram Figure 3 As shown. NBIA is mainly used to fuse shallow (Low Level) and deep (High Level) features. The deep features guide the shallow features after upsampling. In the embodiment of the present invention, the deep feature global average pooling (GlobalAverage Pooling, GAP) and convolution are used in series. This mechanism can be regarded as a channel attention mechanism, thereby guiding the fusion of shallow features. Figure 3 It can be seen that NBIA adopts the upward alignment strategy. In formula (1), x L is the shallow input feature, x H is the deep input feature. Figure 2 It can be seen that the width and height dimensions of all NBIA deep features are half of those of the shallow ones. Both and adopt dual-branch feature extraction, each with one path processed by 3×3 convolution, where xH Upsampling is performed in advance to align the shallow feature dimensions. When shallow features guide deep features, a balance strategy with the shallow layer is adopted, and the convolution kernel size is set to 3. The imbalance of NBIA is mainly reflected in the misalignment of the processing strategies from shallow to deep and from deep to shallow.
[0052]
[0053] MIA is used to extract and fuse feature information at each stage. Its structure is as follows: Figure 4 As shown in the figure. MIA extracts multi-scale features from feature maps using 3×3, 5×5, and 7×7 convolution kernels, and fuses features through trainable weight parameters to eliminate the differences in different receptive fields. Then, the channel attention mechanism is used to redistribute the weights of feature maps extracted from different receptive fields, thereby obtaining a feature module with more scale feature information and channel attention weights. Finally, after the extracted channel attention feature images of different scales are fused, the residual connection is input into the feature image to grasp the characteristics of the segmented object under different receptive fields, and the association with the patches obtained in the encoder is maintained to prevent the gradient disappearance caused by network feature fusion. The attention module in MIA first linearly transforms the features of all positions through 1×1 convolution to obtain the weight coefficients, and normalizes the weight coefficients using the softmax function to obtain the weights of each position. Then the features are weighted averaged to obtain the global context feature. Then, the channel is reduced to one-sixteenth of its original dimension using 1×1 convolution, and then activated by ReLU, and LN is added to stabilize the training, and finally restored by 1×1 convolution.
[0054] like Figure 2 As shown in the figure, the strategy of joint constraint of each layer estimation loss is adopted, and the weighted cross-link loss L is selected. wIoU and binary cross entropy loss L wBCE . IoU loss focuses on predicting the shape of the region, while BCE loss focuses on predicting the probability distribution of each pixel. BCE loss is more stable during training, while IoU loss may fluctuate during initial training. Combining these two losses can more comprehensively optimize model performance. The calculation method of each layer loss is as shown in formula (4), P i represents the prediction result, G represents the manual labeling result. The total network loss L total The calculation method of is as shown in formula (5).
[0055] L i =L wIoU (P i ,G)+L wBCE (P i ,G)(i=1,2) (4)
[0056] L total =L+L1+L2 (5)
[0057] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0058] (1) In the present invention, by adopting a hierarchical multi-scale feature merging and fusion analysis method, the present invention can more accurately identify the lesion area in the endoscopic image, effectively reduce the missed detection rate, and especially greatly improve the detection rate in early cancer screening, ensuring timely diagnosis and treatment of patients;
[0059] (2) In the present invention, the non-balanced information aggregation module (NBIA) and the multi-scale information aggregation module (MIA) are used to realize the effective extraction and fusion of features at different levels and scales, so that the network can take into account both global semantic information and local detail information at the same time, thereby improving the model's perception ability of complex lesion areas;
[0060] (3) In the present invention, through the optimization of multi-layer supervision mechanism and loss function, the present invention effectively avoids the gradient vanishing problem in deep networks and improves the stability of the training process, so that the model can be optimized evenly at different levels, thereby improving the overall detection accuracy and robustness.
[0061] The above contents are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
[0062] There are a few points to note:
[0063] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.
[0064] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.
[0065] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.
[0066] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception, characterized in that: include: S1. Acquire internal human body image data through an endoscope device, wherein the data includes multi-scale image feature information; S2. Using a U-shaped network architecture, a P2T network is selected as a backbone network for feature extraction, wherein the P2T network includes feature maps of shallow features and deep features; S3, the features of adjacent layers are fused step by step through the non-balanced information aggregation module (NBIA), the NBIA module upsamples the deep features and guides the fusion of shallow features, and the channel attention mechanism is realized by using the mechanism of global average pooling (GAP) and convolution in series, so as to optimize the fusion effect; S4. The fused features are further extracted and fused using a multi-scale information aggregation module (MIA). The MIA module uses convolution kernels of different sizes to extract multi-scale features, and fuses features of different receptive fields through trainable weight parameters. At the same time, a channel attention mechanism is used to redistribute weights to enhance the fusion perception of features at all levels. S5. Through a multi-layer supervision mechanism, the features are constrained layer by layer to generate prediction information of different scales to achieve accurate perception and extraction of the lesion area. The loss function of each layer includes weighted cross-link loss (IoU loss) and binary cross entropy loss (BCE loss) to optimize the performance of the model at each level and optimize the prediction performance of the lesion area.
2. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 1 is characterized in that: The backbone network specifically includes: The backbone network is replaced by other convolutional neural networks (CNN) or Transformer networks to adapt to different endoscopic image feature extraction requirements.
3. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 1 is characterized in that: The NBIA specifically includes: The NBIA is used to fuse shallow features and deep features. The deep features guide the shallow features after upsampling. The deep features are globally averaged pooled and convolved in series, thereby guiding the fusion with the shallow features.
4. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 1 is characterized in that: The MIA specifically includes: The MIA uses convolution kernels of different sizes to extract multi-scale features and redistributes the weights of feature maps through a channel attention mechanism. The convolution kernels include 3×3, 5×5 and 7×7.
5. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 4 is characterized in that: The channel attention mechanism specifically includes: Linear transformation is performed through convolution, and the weight coefficients are normalized using the softmax function. Finally, the training process is stabilized through ReLU activation and LayerNormalization (LN).
6. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 1 is characterized in that: The loss function specifically includes: The loss function is a combination of weighted inter-connection loss (IoU loss) and binary cross entropy loss (BCE loss), which is used to optimize network performance and prevent deviations in predicted region shape and pixel probability distribution.
7. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 6 is characterized in that: The loss function specifically includes: The IoU loss is used to optimize the shape of the predicted region, and the BCE loss is used to stabilize the pixel-level probability distribution prediction during the training process.
8. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 1 is characterized in that: include: The total loss of the network is calculated by the joint constraints of the losses of each layer to ensure that the network is balanced and optimized at each level.
9. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion area perception according to claim 1 is characterized in that: include: The shallow features and the deep features of the NBIA module are aligned in width and height dimensions, the width and height dimensions of the deep features are half of the shallow features, and unbalanced fusion is achieved through a dual-branch feature extraction strategy.
10. The hierarchical multi-scale feature merging and fusion analysis method for endoscopic lesion region perception according to claim 1, characterized in that: include: The multi-layer supervision mechanism generates multi-scale prediction information through layer-by-layer loss constraints, and finally completes the extraction and analysis of the lesion area through joint fusion.