An esophageal early cancer lesion segmentation method based on attention double-branch feature fusion

Abstract

In view to the problems that the early esophageal cancer lesion has low foreground-background contrast, various shapes and irregularities, an early esophageal cancer lesion segmentation method based on attention double branch feature fusion is provided, and the network structure is an AMFF-DUNet combining feature fusion and double branch up-sampling network (AMFF-DUNet) of attention mechanism, which combines channel and spatial double attention mechanism, multi-scale feature fusion and double branch up-sampling. The method comprises the following steps: step 1, building the AMFF-DUNet network, adding the pyramid guided feature fusion module (PGFM) and the double branch up-sampling module (DBUM) to the backbone network; step 2, reading the endoscope image, performing image preprocessing such as clipping and color space transformation; step 3, using the AMFF-DUNet to perform accurate semantic segmentation on the esophagus endoscope image; and step 4, comparing and analyzing the experimental results with the current advanced early esophageal cancer lesion segmentation method. The results show that the method improves the segmentation accuracy of the early esophageal cancer lesion with unclear edge features and various shapes.

Description

[0001] This application claims domestic priority to the invention application filed on June 1, 2022, with application number 202210610948.3 and patent name “A method for segmenting esophageal early cancer lesions based on attention dual-branch feature fusion”. The entire contents of the application are incorporated herein by reference. Technical Field

[0002] The present invention relates to the field of artificial intelligence technology, in particular to the field of machine vision technology and the field of semantic segmentation technology, and more specifically to a method for segmenting early esophageal cancer lesions by fusing attention dual-branch features. Background Art

[0003] Esophageal images have complex backgrounds and the diseased areas vary widely across patients, making screening for esophageal lesions and early cancer a challenging task. Esophageal lesions have low foreground-background contrast, diverse and irregular shapes, and are susceptible to endoscopic imaging noise, making them prone to bright spots. This makes it difficult for traditional segmentation algorithms to segment lesions in endoscopic images. Compared with traditional methods, deep learning-based segmentation methods can effectively capture both low-level detail features and high-level semantic features of images, offering advantages for segmenting esophageal images with complex backgrounds. In particular, the U-Net proposed by Ronneberger et al. and subsequent U-Net variants have been widely used in medical image segmentation. These methods employ symmetrical structures and skip connections to effectively integrate low-level and high-level image features, addressing the inaccurate positioning issues of conventional convolutional neural networks used for medical image segmentation.

[0004] In recent years, deep learning-based artificial intelligence methods have made significant progress in various medical fields, especially as a medical image screening system. These fields include radiation oncology diagnosis, skin cancer classification, diabetic retinopathy segmentation, histological classification of gastric biopsy specimens, and characterization of colorectal lesions using endoscopy. Deep learning has also become a powerful supporting tool in the field of early esophageal cancer screening. Xue et al. detected early esophageal cancer by classifying microvascular morphological types. They developed a model in Caffe that used convolutional neural networks (CNN) for feature extraction and support vector machines (SVM) for classification, pioneering the use of deep learning methods for early esophageal cancer screening. Hong et al. used CNN to distinguish between gastric biochemicals, intestinal biochemicals, and gastric tumors. The architecture consisted of four convolutional layers, two maximum pooling layers, and two fully connected layers (FC), achieving a classification accuracy of 80.77%. In 2019, Professor Xu Ruihua's team used the Clopper-Pearson method to estimate the diagnostic accuracy of cancerous lesions. In five external validation sets, the diagnostic accuracy ranged from 0.915 to 0.977. In 2021, Professor Hu Bing's team proposed an esophageal cancer diagnosis algorithm based on a deep learning model. Using 6,473 precancerous lesions and esophageal squamous cell carcinoma (Narrow Band Imaging, NBI) images labeled by professional physicians, the CNN model SegNet was used to extract image features and construct an early screening model with a sensitivity (Sensitivity, Se) and specificity (Specificity, Sp) exceeding 90%. In summary, in the field of esophageal early cancer segmentation, the main direction is to use deep learning methods to obtain high precision, especially high sensitivity and high specificity. Summary of the Invention

[0005] In view of this, the present invention provides an esophageal early cancer lesion segmentation method based on attention dual-branch feature fusion.

[0006] The method for segmenting early esophageal cancer lesions based on attention dual-branch feature fusion includes: preprocessing an esophageal endoscopic image to obtain a target endoscopic image; processing the target endoscopic image using an encoder network included in a deep learning model to obtain a first feature image; processing the first feature image using a dilated spatial pyramid pooling module included in the deep learning model to obtain a second feature image; for each of the multiple pyramid-guided fusion modules included in the deep learning model, fusing the output features of multiple first target stage networks in the multiple stage networks included in the encoder network using the pyramid-guided fusion module included in the deep learning model to obtain a third feature image; and processing the first feature image and the multiple third feature images using a decoder network included in the deep learning model to obtain a semantic segmentation image of the esophageal endoscopic image.

[0007] According to an embodiment of the present invention, the pyramid-guided fusion module included in the deep learning model is used to fuse the output features of multiple first target stage networks in the multiple stage networks included in the encoder network to obtain a third feature image, including: determining a second target stage network from the multiple first target stage networks; sampling the output features of the multiple first target stage networks based on the size of the output features of the second target stage network to obtain multiple fifth encoded feature images; splicing the multiple fifth encoded feature images to obtain a sixth encoded feature image; using multiple hole convolution layers to process the sixth encoded feature image to obtain multiple seventh encoded feature images, wherein the hole rates between the multiple hole convolution layers are different; and splicing the multiple seventh encoded feature images to obtain the third feature image.

[0008] According to an embodiment of the present invention, the method further includes: determining a plurality of third target stage networks from a plurality of network stages included in the decoder network based on the sizes of the output features of the plurality of second target stage networks.

[0009] According to an embodiment of the present invention, the above-mentioned decoder network includes a dual-branch on-sampling module; wherein, the above-mentioned decoder network included in the above-mentioned deep learning model processes the above-mentioned first feature image and multiple third feature images to obtain the semantic segmentation image of the above-mentioned esophageal endoscopy image, including: for each stage network of the multiple stage networks included in the above-mentioned decoder network, when the above-mentioned stage network is the above-mentioned third target stage network, the output features of the above-mentioned stage network and the third feature image corresponding to the above-mentioned third target stage network are spliced ​​to obtain a first decoding feature image, and the above-mentioned first decoding feature image is used as the input feature of the next stage network of the above-mentioned stage network; when the output end of the above-mentioned stage network is connected to the dual-branch on-sampling module, the output features of the above-mentioned stage network are processed by the above-mentioned dual-branch on-sampling module to obtain a second decoding feature image, and the above-mentioned second decoding feature image is used as the input feature of the next stage network of the above-mentioned stage network; and when the above-mentioned stage network is the last stage network, the above-mentioned semantic segmentation image is obtained based on the output features of the above-mentioned stage network.

[0010] According to an embodiment of the present invention, the above-mentioned dual-branch upsampling module is used to process the output features of the above-mentioned stage network to obtain the second decoding feature image, including: performing bicubic interpolation processing on the output features of the above-mentioned stage network to obtain a third decoding feature image; performing pixel reconstruction upsampling processing on the output features of the above-mentioned stage network to obtain a fourth decoding feature image; and splicing the above-mentioned third decoding feature image and the above-mentioned fourth decoding feature image to obtain the above-mentioned second decoding feature image.

[0011] According to an embodiment of the present invention, the above-mentioned preprocessing of the esophageal endoscopic image to obtain the target endoscopic image includes: performing color space transformation on the above-mentioned esophageal endoscopic image to obtain an intermediate endoscopic image; and performing data standardization on the above-mentioned intermediate endoscopic image using a standard normal distribution to obtain the above-mentioned target endoscopic image.

[0012] According to an embodiment of the present invention, the multiple stage networks included in the above-mentioned encoder network include a starting stage network, multiple intermediate stage networks and an ending stage network, and the above-mentioned encoder network includes a spatial channel dual attention module; wherein the above-mentioned encoder network included in the deep learning model processes the above-mentioned target endoscopic image to obtain a first feature image, including: inputting the above-mentioned target endoscopic image into the above-mentioned starting stage network to obtain a first encoded feature image; using the spatial channel dual attention module to process the above-mentioned first encoded feature image to obtain a second encoded feature image; using the above-mentioned multiple intermediate stage networks to process the above-mentioned second encoded feature image to obtain a third encoded feature image; using the above-mentioned spatial channel dual attention module to process the above-mentioned third encoded feature image to obtain a fourth encoded feature image; and inputting the above-mentioned fourth encoded feature image into the above-mentioned ending stage network to obtain the above-mentioned first feature image.

[0013] According to an embodiment of the present invention, the backbone network of the above-mentioned deep learning model is ResNet1 01. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] Figure 1 A schematic diagram of the network structure of a deep learning model according to an embodiment of the present invention is shown schematically.

[0016] Figure 2 The figure schematically shows a pyramid-guided fusion module according to an embodiment of the present invention.

[0017] Figure 3 The figure schematically shows a schematic diagram of a dual-branch sampling module according to an embodiment of the present invention.

[0018] Figure 4 The figure schematically shows a bicubic interpolation method according to an embodiment of the present invention.

[0019] Figure 5 A heat map comparison diagram of the esophageal early cancer lesion segmentation method based on attention dual-branch feature fusion according to an embodiment of the present invention and a method of the related art is schematically shown.

[0020] Figure 6 A comparison diagram of segmentation images of an esophageal early cancer lesion segmentation method based on attention dual-branch feature fusion according to an embodiment of the present invention and a method of related art is schematically shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] In the screening of early esophageal cancer, the endoscopic manifestations of early esophageal cancer are very mild, and it is not easy for endoscopists to accurately detect the lesion area. The present invention uses hollow convolution and depthwise separable convolution to propose a pyramid-guided feature fusion module, which guides the fusion of features at different levels to enhance the expression of effective information. It uses bicubic interpolation and pixel reconstruction methods to propose a dual-branch upsampling module, which performs upsampling in space and channels simultaneously, reducing the loss of useful information during the upsampling process. The DeepLabV3+ network parameters are adjusted to make it suitable for the binary semantic segmentation task. At the same time, the convolutional attention module, the pyramid feature fusion module and the dual-branch upsampling module are combined to build a dual-branch feature fusion network combined with the attention mechanism to improve the segmentation accuracy of the esophageal early cancer lesion area.

[0026] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security and national security. In the embodiments of the present invention, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0027] According to an embodiment of the present invention, a method for segmenting early esophageal cancer lesions based on attention dual-branch feature fusion may include the following operations:

[0028] The esophageal endoscopic image is preprocessed to obtain a target endoscopic image; the target endoscopic image is processed using an encoder network included in a deep learning model to obtain a first feature image; the first feature image is processed using a dilated spatial pyramid pooling module included in the deep learning model to obtain a second feature image; for each of the multiple pyramid-guided fusion modules included in the deep learning model, the output features of multiple first target stage networks in the multiple stage networks included in the encoder network are fused using the pyramid-guided fusion module included in the deep learning model to obtain a third feature image; and the first feature image and multiple third feature images are processed using a decoder network included in the deep learning model to obtain a semantic segmentation image of the esophageal endoscopic image.

[0029] According to an embodiment of the present invention, using a pyramid guided fusion module included in a deep learning model to fuse the output features of multiple first target stage networks in the multiple stage networks included in the encoder network to obtain a third feature image may include the following operations:

[0030] A second target stage network is determined from multiple first target stage networks; based on the size of the output features of the second target stage network, the output features of the multiple first target stage networks are sampled respectively to obtain multiple fifth encoded feature images; the multiple fifth encoded feature images are spliced ​​to obtain a sixth encoded feature image; the sixth encoded feature images are processed using multiple hole convolution layers respectively to obtain multiple seventh encoded feature images, wherein the hole rates between the multiple hole convolution layers are different; and the multiple seventh encoded feature images are spliced ​​to obtain a third feature image.

[0031] According to an embodiment of the present invention, a plurality of third target stage networks may be determined from a plurality of network stages included in the decoder network based on the sizes of the output features of the respective plurality of second target stage networks.

[0032] According to an embodiment of the present invention, the decoder network may include a dual-branch upsampling module.

[0033] According to an embodiment of the present invention, processing the first feature image and the plurality of third feature images using a decoder network included in a deep learning model to obtain a semantic segmentation image of the esophageal endoscopy image may include the following operations:

[0034] For each stage network of the multiple stage networks included in the decoder network, when the stage network is the third target stage network, the output features of the stage network and the third feature image corresponding to the third target stage network are spliced ​​to obtain a first decoding feature image, and the first decoding feature image is used as the input feature of the next stage network of the stage network; when the output end of the stage network is connected to a dual-branch on-sampling module, the output features of the stage network are processed by the dual-branch on-sampling module to obtain a second decoding feature image, and the second decoding feature image is used as the input feature of the next stage network of the stage network; and when the stage network is the last stage network, a semantic segmentation image is obtained based on the output features of the stage network.

[0035] According to an embodiment of the present invention, obtaining a second decoded feature image by processing the output features of the stage network using the dual-branch upsampling module may include the following operations:

[0036] The output features of the stage network are subjected to bicubic interpolation processing to obtain a third decoding feature image; the output features of the stage network are subjected to pixel reconstruction upsampling processing to obtain a fourth decoding feature image; and the third decoding feature image and the fourth decoding feature image are spliced ​​to obtain a second decoding feature image.

[0037] According to an embodiment of the present invention, preprocessing the esophageal endoscopic image to obtain a target endoscopic image may include the following operations:

[0038] Performing color space transformation on the esophageal endoscopic image to obtain an intermediate endoscopic image; and performing data normalization on the intermediate endoscopic image using a standard normal distribution to obtain a target endoscopic image.

[0039] According to an embodiment of the present invention, the multiple stage networks included in the encoder network include a starting stage network, multiple intermediate stage networks and an ending stage network, and the encoder network includes a spatial channel dual attention module.

[0040] According to an embodiment of the present invention, processing the target endoscopic image using the encoder network included in the deep learning model to obtain the first feature image may include the following operations:

[0041] The target endoscopic image is input into the starting stage network to obtain a first coding feature image; the first coding feature image is processed using a spatial channel dual attention module to obtain a second coding feature image; the second coding feature image is processed using multiple intermediate stage networks to obtain a third coding feature image; the third coding feature image is processed using a spatial channel dual attention module to obtain a fourth coding feature image; and the fourth coding feature image is input into the ending stage network to obtain the first feature image.

[0042] According to an embodiment of the present invention, the backbone network of the deep learning model is ResNet101.

[0043] The following is a further detailed description of the esophageal early cancer lesion segmentation method based on attention dual-branch feature fusion of the present invention with reference to examples and drawings.

[0044] Step 1: Build a deep learning network, which can be specifically an Attention Mechanism combined Feature Fusion and Dual-branch Upsampling Network (AMFF-DUNet). The network uses ResNet101 as the backbone network, and the overall framework is a codec structure. The encoder introduces spatial and channel dual attention to enhance the feature expression ability of non-significant lesion areas; the Pyramid-Guided Feature Fusion Module (PGFM) proposed in the present invention is used between the codec to guide the fusion of features at different levels to enhance the expression of effective information; the Dual-Branch Upsampling Module (DBUM) proposed in the present invention is used in the decoder stage to reduce information loss during the upsampling process.

[0045] Figure 1 A schematic diagram of the network structure of a deep learning model according to an embodiment of the present invention is shown schematically.

[0046] like Figure 1As shown in the figure, the legend names of the relevant operations are listed on the right side of the figure. The network input size is 224×224×3. The network structure uses DeepLabV3+ as the basic framework. The entire network is divided into five stages from stage1 to stage5. At the beginning and the last stage of the encoder, the convolutional block attention module (CBAM) is used to introduce channel and spatial dual attention to enhance the feature expression of non-significant lesion areas. The pyramid-guided feature fusion module (PGFM) proposed in the present invention and the atrous spatial pyramid pooling (ASPP) of the network itself are used between the encoder and decoder to capture contextual information and enhance feature expression at multiple scales. In the decoder stage, the dual-branch upsampling module (DBUM) proposed in the present invention is used. This module reduces the loss of detail information in the upsampling process by fusing the spatial and channel information of the image, thereby enhancing the network segmentation capability.

[0047] The modules proposed in the present invention are specifically explained as follows:

[0048] (1) Use the pyramid-guided feature fusion module (PGFM) for multi-scale feature extraction. In order to extract global context information from feature maps at different levels and prevent the loss of spatial information during the upsampling process, the pyramid-guided feature fusion module proposed in step 1 uses regular 3×3 convolution to map the features of different stages of ResNet101 to the same channel space as the selected stage, and then upsamples the generated feature maps to the same size and splices them; then, dilated convolutions with different dilation rates are used to expand the receptive field and compensate for the loss of correlation. Considering that the above operation will increase the model parameters and affect the calculation speed of the network, depthwise separable convolution is performed on the spliced ​​feature maps before dilated convolution, and finally ordinary convolution is used to obtain the final feature map.

[0049] Figure 2 The figure schematically shows a pyramid-guided fusion module according to an embodiment of the present invention.

[0050] like Figure 2As shown in Figure 1, PGFM maps the features of stage 3 and stage 4 to the same channel space as stage 2 through regular 3×3 convolution. The generated feature maps F3 and F4 are upsampled to the same size as F2 and concatenated. In order to extract global context information from feature maps at different levels and prevent the loss of spatial information in the upsampling process, dilated convolutions with dilation rates of r=1, r=2, and r=4 are superimposed to expand the receptive field and compensate for the loss of correlation. Considering that the above operations will increase the model parameters and affect the calculation speed of the network, depthwise separable convolution is performed on the concatenated feature maps before performing dilated convolution. Finally, ordinary convolution is used to obtain the final feature map. The output of the PGFM module is shown in formula (1):

[0051]

[0052] Where Pk represents the output of the PGFM inserted into the kth stage, F k represents the feature map encoder of the kth stage, Indicates a rate of 2 i-k Upsampling operation, Conv 3×3 Represents 3×3 convolution, Conv ds Denotes depth-wise separable convolution, Conv dc R2 i-k Indicates an expansion rate of 2 i-k Cat represents the Concat operation, and m represents the number of stages involved in feature guidance.

[0053] (2) Use a dual-branch upsampling module (DBUM) to reduce the loss of detail information during the upsampling process. In step 1, the dual-branch upsampling module (DBUM) proposed by the present invention uses bicubic interpolation (BiC) and pixel shuffle (PS) for parallel upsampling in the decoder upsampling stage to reduce the loss of image detail information during the upsampling process. This captures the features required by the network in both space and channels, allowing the network to generate high-resolution feature maps with rich details and semantic information.

[0054] Figure 3 The figure schematically shows a schematic diagram of a dual-branch sampling module according to an embodiment of the present invention.

[0055] like Figure 3 As shown, bicubic interpolation (BiC) and pixel shuffle (PS) are used for parallel upsampling. Figure 3 PS represents the pixel reconstruction upsampling method, which first obtains r by convolution. 2The feature map of each channel, r is the image magnification. Then, the r of each pixel in the low-resolution image is filtered periodically. 2 The channels are expanded into r×r large pixels and reassembled together, so the number of channels is r 2 C is reduced to C, and the image size H×W is expanded to rH×rW. In general, DBUM is to perform bicubic interpolation upsampling on the input space to obtain the feature map F s , perform pixel reconstruction upsampling on the channel to obtain F c , F s and F c The output is obtained by adding and fusion. The output of the dual-branch sampling module is as follows:

[0056] Output=Bicubic(Input)+Conv(PS(Input)) (2)

[0057] Where Conv represents convolution operation, BiCubic represents bicubic interpolation, PS represents pixel reconstruction (PixelShuffle), Input represents input image, and Output represents output image.

[0058] Compared with bilinear interpolation, bicubic interpolation not only considers the pixel values ​​of the four directly adjacent points, but also uses the pixel values ​​of the 16 adjacent points around the sampling point for cubic interpolation.

[0059] Figure 4 The figure schematically shows a bicubic interpolation method according to an embodiment of the present invention.

[0060] like Figure 4 Point P is the coordinate point of the source image corresponding to the target image B (x, y) after magnification. The pixel value of P is obtained by calculating the coefficients of the 16 points around point P and weighting them. Taking point A in the upper left corner as an example, its distance from point P is (1+u, 1+v). Substituting it into the most commonly used bicubic interpolation basis function in formula (3):

[0061]

[0062] Then the corresponding coefficient of point A is k 00 =f(1+u)*f(1+v). Similarly, the coefficients for the remaining 15 adjacent points can be calculated. The pixel values ​​of each of the 16 adjacent points are multiplied by the corresponding coefficients and then added together to obtain the pixel value at point P. Because bicubic interpolation considers the impact of changes in pixel values ​​at each adjacent point on the image, this method produces a high-resolution image with richer detail.

[0063] Step 2: Read the endoscopic image and perform image preprocessing by cropping and color space conversion. The dataset used in this article is a self-built dataset consisting of white light endoscopic images, endoscopic images stained with Lugol's solution, and NBI endoscopic images, totaling 3503 images, including 783 white light images, 791 NBI images, and 1929 iodine-stained images. It was provided by the 404 Hospital of Mianyang, Sichuan Province, and all cases were confirmed by histology. The data were collected by doctors when patients underwent gastroscopy and are random, general, and authentic. The data were first marked by the hospital's gastroenterology experts according to the endoscopic report to determine the lesion area, and then the data was finely labeled using Lableme software. The image was cropped to 224×224 and then preprocessed. In order to reduce the impact of reflection and low contrast on the model, speed up the model training efficiency, and enhance the generalization ability of the model, the following preprocessing was performed based on the characteristics of the endoscopic image itself:

[0064] (1) Using random horizontal flipping and random cropping, early esophageal cancer is made to appear in different locations, reducing the model's dependence on the location of the lesion;

[0065] (2) Convert the RGB image into an HSV image;

[0066] (3) To reduce the impact of reflections and glare on endoscopic images, the brightness and contrast of the image processed in step (2) were randomly adjusted between 0.8 and 1.2 to reduce the model's sensitivity to highlights and low contrast.

[0067] (4) Use the standard normal distribution method to standardize data and accelerate model convergence.

[0068] Step 3: Use AMFF-DUNet to perform accurate semantic segmentation of esophageal endoscopic images. The experiment relies on the Pytorch platform to build a neural network, version 1.8.0, and Python version 3.6.5. The training strategy is as follows: the dataset is divided into training set, validation set, and test set according to the ratio of 7:2:1. SGD is used as the optimizer during training, and the initial learning rate is set to 0.5×10 3 , and use warm-up learning in the first 10 training rounds to accelerate model convergence. Save the network model once per iteration, and iterate 300 rounds in total. Save the model with the best test results. Due to the imbalanced distribution of positive and negative samples in esophageal early cancer data, Focalloss is selected as the loss function, which increases the weight of difficult-to-separate positive samples in the loss function and improves the segmentation accuracy of the overall positive samples. Focal loss is shown in formula (4), where Pt reflects the degree of closeness to the true value. The larger Pt is, the closer it is to the true value, that is, the more accurate the classification is. γ is an adjustable factor with a value between 0 and 1. L f1 Represents focal loss.

[0069] Lf1 =-(1-P t ) γ log(P t ) (4)

[0070] Tables 1 and 2 are the ablation test results of PGFM and DBUM on the self-built dataset, respectively, with DeepLabV3+ as the benchmark network. From the ablation experiment in Table 1, we can see that the model with two PGFMs added, that is, the PGFM in the table 2 On a self-constructed dataset, the Mean Intersection over Union (MIoU), Se, and Sp metrics improved from 79.10%, 85.40%, and 89.53% to 79.86%, 87.89%, and 90.26%, respectively. Taking the best model in Table 1 as the base model, Table 2 shows the improvement in accuracy after combining the base model with transposed convolution (TC), bicubic interpolation (BiC), and pixel reconstruction (PS). Table 2 shows that TC produces a "checkerboard effect" due to its zero-padding operation during upsampling, and optimizing its parameters requires extensive trial and error. Therefore, the combination of bicubic interpolation and pixel reconstruction (PS) in Experiment 7 achieved the best results, improving MIoU, Se, and Sp metrics to 80.25%, 88.95%, and 92.02%, respectively. Ablation experiments demonstrate that all proposed modules improve the accuracy of early esophageal cancer segmentation, with the combined approach achieving even better results.

[0071] Table 1 Ablation experiment results of CBAM and PGFM modules

[0072]

[0073]

[0074] Table 2 Ablation experiment results of different types of upsampling methods in DBUM module

[0075]

[0076] Step 4: Compare and analyze the experimental results with the current advanced esophageal early cancer lesion segmentation methods.

[0077] According to an embodiment of the present invention, the pyramid-guided feature fusion module (PGFM), dual-branch dual-sampling module (DBUM) and spatial channel dual-attention module proposed in the present invention are combined to form a feature fusion and dual-branch upsampling network (AMFF-DUNet) combined with attention mechanism, which improves the accuracy of lesion area segmentation in endoscopic examination of early esophageal cancer.

[0078] We conducted experiments on a self-built dataset using mainstream medical image segmentation methods published in recent years. Table 3 shows the experimental results of different segmentation methods. Our method performs best in terms of MIoU, Se, and F1-Score.

[0079] Table 3 Experimental results of different segmentation methods in self-built dataset

[0080]

[0081] In addition to the quantitative experimental data, Grad-CAM visualization results were also used for qualitative analysis.

[0082] Figure 5 A heat map comparison diagram of the esophageal early cancer lesion segmentation method based on attention dual-branch feature fusion according to an embodiment of the present invention and a method of the related art is schematically shown.

[0083] exist Figure 5 It is clearly shown in the figure that the lesion area focused on by the AMFF-DUNet method of the present invention covers the target lesion better than other methods, indicating that the proposed model can better complete the lesion area segmentation task.

[0084] Figure 6 A comparison diagram of segmentation images of an esophageal early cancer lesion segmentation method based on attention dual-branch feature fusion according to an embodiment of the present invention and a method of related art is schematically shown.

[0085] Figure 6 The comparison results between the segmentation model of the method of the present invention and the "Deep Learning Assisted Early Esophageal Cancer Diagnosis Model Sharing Platform" of West China Hospital are shown. From the five randomly selected pictures, it can be seen that the segmentation results of the method of the present invention are closer to the ground truth.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present invention.

[0087] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is defined by the appended claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which are intended to fall within the scope of the present invention.

Claims

1. A method for early esophageal cancer lesion segmentation based on dual-branch attention feature fusion, comprising: Preprocessing the esophageal endoscopic image to obtain a target endoscopic image; Processing the target endoscopic image using an encoder network included in the deep learning model to obtain a first feature image; Processing the first feature image using a dilated spatial pyramid pooling module included in the deep learning model to obtain a second feature image; For each of the plurality of pyramid-guided fusion modules included in the deep learning model, using the pyramid-guided fusion module included in the deep learning model to fuse respective output features of a plurality of first target stage networks in the plurality of stage networks included in the encoder network to obtain a third feature image; Processing the first feature image and the plurality of third feature images using a decoder network included in the deep learning model to obtain a semantic segmentation image of the esophageal endoscopic image includes: For each stage network of the multiple stage networks included in the decoder network, when the stage network is a third target stage network, concatenating the output features of the stage network and a third feature image corresponding to the third target stage network to obtain a first decoded feature image, and using the first decoded feature image as an input feature of a next stage network of the stage network; In a case where a dual-branch on-sampling module is connected to an output end of the stage network, the dual-branch on-sampling module is used to process the output features of the stage network to obtain a second decoded feature image, and the second decoded feature image is used as an input feature of a next stage network of the stage network; When the stage network is the last stage network, obtaining the semantic segmentation image based on output features of the stage network; The decoder network includes a dual-branch upsampling module; the dual-branch upsampling module uses bicubic interpolation and pixel reconstruction for parallel upsampling in the upsampling stage of the decoder, and the multiple network stages include multiple third target stage networks.

2. The method according to claim 1, wherein The method of using the pyramid guided fusion module included in the deep learning model to fuse the output features of each of the multiple first target stage networks in the multiple stage networks included in the encoder network to obtain a third feature image includes: determining a second target phase network from the plurality of first target phase networks; Based on the size of the output feature of the second target stage network, sampling the output features of each of the plurality of first target stage networks respectively to obtain a plurality of fifth encoded feature images; splicing the plurality of fifth coding characteristic images to obtain a sixth coding characteristic image; Processing the sixth coding feature image using a plurality of dilated convolutional layers respectively to obtain a plurality of seventh coding feature images, wherein the dilated convolutional layers have different dilation rates; and The plurality of seventh coded characteristic images are spliced ​​together to obtain the third characteristic image.

3. The method according to claim 2, further comprising: Based on the sizes of the output features of the plurality of second target stage networks, a plurality of third target stage networks are determined from the plurality of network stages included in the decoder network.

4. The method according to claim 3, wherein: The step of processing the output features of the stage network using the dual-branch upsampling module to obtain a second decoded feature image includes: Performing bicubic interpolation processing on the output features of the stage network to obtain a third decoded feature image; Performing pixel reconstruction upsampling processing on the output features of the stage network to obtain a fourth decoded feature image; and The third decoding feature image and the fourth decoding feature image are spliced ​​together to obtain the second decoding feature image.

5. The method according to claim 1, wherein The preprocessing of the esophageal endoscopic image to obtain a target endoscopic image includes: performing color space transformation on the esophageal endoscopic image to obtain an intermediate endoscopic image; and The intermediate endoscopic image is data normalized using a standard normal distribution to obtain the target endoscopic image.

6. The method according to claim 1, wherein The encoder network includes multiple stage networks including a starting stage network, multiple intermediate stage networks and an ending stage network, and the encoder network includes a spatial channel dual attention module; The step of processing the target endoscopic image using an encoder network included in the deep learning model to obtain a first feature image includes: Inputting the target endoscopic image into the starting phase network to obtain a first encoding feature image; Processing the first coding feature image using a spatial channel dual attention module to obtain a second coding feature image; processing the second encoding feature image using the plurality of intermediate stage networks to obtain a third encoding feature image; Processing the third coded feature image using the spatial channel dual attention module to obtain a fourth coded feature image; and The fourth encoded feature image is input into the end-stage network to obtain the first feature image.

7. The method according to claim 1, wherein The backbone network of the deep learning model is ResNet101.

Images