Semantic segmentation method and device based on nested double-U-shaped attention network

Abstract

The invention provides a semantic segmentation method and device based on a nested double-U-shaped attention network, and the method comprises the steps: inputting the image data of an endoscope image into a first encoder, and obtaining first coding features outputted by M first coding units of the first encoder; performing weighted feature fusion on the first coding features output by the M first coding units by using N second coding units included in a second encoder to obtain second coding features output by the N second coding units; processing second coding features output by the N second coding units by using a first decoder to obtain first decoding features; a second decoder is used for processing first coding features output by the M first coding units and second coding features output by the N second coding units, and second decoding features are obtained; and splicing the first decoding feature and the second decoding feature in the channel dimension to obtain a semantic segmentation result of the endoscopic image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and medical image processing technology, and specifically to a semantic segmentation method and device based on a nested double U-type attention network. Background Art

[0002] With the development of artificial intelligence technology, artificial neural networks are increasingly used in various fields of production and life. For example, in the field of medical imaging, models based on artificial neural networks can be used to assist medical staff in completing the analysis of medical images.

[0003] Medical images include endoscopic images taken with an endoscope. Endoscopic images generally have complex backgrounds, and the lesion areas of early mild diseases are not obvious in endoscopic images. The methods of related technologies cannot effectively select and reuse image features in endoscopic images, resulting in low image segmentation accuracy. Summary of the invention

[0004] In view of this, the present disclosure provides a semantic segmentation method, device, electronic device, readable storage medium and computer program product based on a nested double U-type attention network.

[0005] One aspect of the present disclosure provides a semantic segmentation method based on a nested double U-type attention network, wherein the nested double U-type attention network includes a first encoder, a second encoder, a first decoder and a second decoder, and the method includes: inputting image data of an endoscopic image into the first encoder to obtain first coding features output by each of the M first coding units included in the first encoder, wherein the M is a positive integer; using the N second coding units included in the second encoder to perform weighted feature fusion on the first coding features output by each of the M first coding units to obtain second coding features output by each of the N second coding units, wherein the N is a positive integer; using the first decoder to process the second coding features output by each of the N second coding units to obtain a first decoding feature; using the second decoder to process the first coding features output by each of the M first coding units and the second coding features output by each of the N second coding units to obtain a second decoding feature; and splicing the first decoding feature and the second decoding feature in the channel dimension to obtain a semantic segmentation result of the endoscopic image.

[0006] Another aspect of the present disclosure provides a semantic segmentation device based on a nested double U-type attention network, wherein the nested double U-type attention network includes a first encoder, a second encoder, a first decoder and a second decoder, and the device includes: an input module, used to input image data of an endoscopic image into the first encoder, and obtain a first encoding feature output by each of the M first encoding units included in the first encoder, wherein the M is a positive integer; a feature fusion module, used to use the N second encoding units included in the second encoder to perform weighted feature fusion on the first encoding features output by each of the M first encoding units, and obtain a second encoding feature output by each of the N second encoding units, wherein the N is a positive integer; a first processing module, used to use the first decoder to process the second encoding features output by each of the N second encoding units, and obtain a first decoding feature; a second processing module, used to use the second decoder to process the first encoding features output by each of the M first encoding units and the second encoding features output by each of the N second encoding units, and obtain a second decoding feature; and a splicing module, used to splice the first decoding feature and the second decoding feature in the channel dimension to obtain a semantic segmentation result of the endoscopic image.

[0007] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described above.

[0008] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0009] Another aspect of the present disclosure provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when executed.

[0010] According to an embodiment of the present disclosure, a nested double U-type attention network is constructed based on UNet using a first encoder, a second encoder, a first decoder, and a second decoder. The first encoder can extract first coded features of different scales from the image data of the endoscopic image. The second encoder can weightedly fuse the first coded features of different scales so that the second coded features output by the second encoder can take into account information at different levels and enhance the recognition ability of feature information related to the segmentation target. The first decoder can restore the resolution by introducing a lower-level high-resolution feature, namely the second coded feature, through a skip connection to obtain a first decoded feature. The second decoder can introduce global information to obtain a second decoded feature, and use the second decoded feature to supplement and enhance the first decoded feature to supplement the information lost during the interpolation upsampling process. Through the above technical means, the accuracy of the semantic segmentation of endoscopic images can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0012] Figure 1 A flowchart of a semantic segmentation method based on a nested double U-type attention network according to an embodiment of the present disclosure is schematically shown.

[0013] Figure 2 The structural diagram of the second encoder according to the embodiment of the present disclosure is schematically shown.

[0014] Figure 3 The structural diagram of the first decoder according to the embodiment of the present disclosure is schematically shown.

[0015] Figure 4 The structural diagram of the attention gate according to an embodiment of the present disclosure is schematically shown.

[0016] Figure 5 The structural diagram of the second decoder according to the embodiment of the present disclosure is schematically shown.

[0017] Figure 6 The structural diagram of the semantic guidance unit according to the embodiment of the present disclosure is schematically shown.

[0018] Figure 7 The schematic diagram shows the structure of a nested double U-type attention network according to an embodiment of the present disclosure.

[0019] Figure 8 A block diagram of a semantic segmentation device based on a nested double U-type attention network according to an embodiment of the present disclosure is schematically shown.

[0020] Fig. 9A block diagram of an electronic device suitable for implementing a semantic segmentation method based on a nested double U-type attention network according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0023] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0024] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] In related technologies, semantic segmentation of medical images is mostly achieved using networks such as FCN (Fully Convolutional Network), UNet, DeepLab and their derived variant networks.

[0026] Take UNet as an example. Inspired by the architecture and encoder-decoder model of FCN, UNet is a network tailored for practical applications in medical image analysis. The UNet network consists of two parts. The first part is the encoder, which uses a downsampling module consisting of several convolutional blocks to extract semantic and contextual features. The second part is the decoder, which applies a set of convolutional blocks with upsampling operations to gradually increase the spatial resolution of the feature map, usually by a factor of two, while reducing the feature dimension to produce pixel-level classification scores.

[0027] The most important part of UNet is the skip connection, which copies the output of each stage in the encoder to the corresponding stage in the decoder. This design propagates essential high-resolution contextual information along the network, encouraging the network to reuse low-level representations and high-level contextual representations for accurate localization.

[0028] However, the background of medical images is relatively complex, and the lesion areas of early mild diseases are not obvious. The networks in related technologies cannot effectively select and reuse image features. The UNet decoder traditionally uses interpolation or transposed convolution methods to restore resolution based on a limited receptive field, which limits the segmentation performance of the UNet network. The network that combines visual transformers (ViT) or uses atrous convolutions for feature extraction is highly complex and computationally intensive, requiring a large amount of image data for training, and is limited by model parameters and computing resources.

[0029] In view of this, the embodiments of the present disclosure provide a semantic segmentation method and device based on a nested double U-type attention network. In the encoding stage, the feature maps extracted from each layer of the backbone network are weighted fused to fully consider the differences in the contributions of feature maps of different scales in feature learning; in the decoding stage, a decoding enhancement score is constructed to supplement the information lost in the interpolation upsampling process, thereby improving the accuracy of semantic segmentation.

[0030] Specifically, the nested double U-type attention network includes a first encoder, a second encoder, a first decoder and a second decoder. The semantic segmentation method based on the nested double U-type attention network includes: inputting the image data of the endoscopic image into the first encoder to obtain the first coding features output by each of the M first coding units included in the first encoder, wherein M is a positive integer; using the N second coding units included in the second encoder to perform weighted feature fusion on the first coding features output by each of the M first coding units, to obtain the second coding features output by each of the N second coding units, wherein N is a positive integer; using the first decoder to process the second coding features output by each of the N second coding units, to obtain the first decoding features; using the second decoder to process the first coding features output by each of the M first coding units and the second coding features output by each of the N second coding units, to obtain the second decoding features; and splicing the first decoding features and the second decoding features in the channel dimension to obtain the semantic segmentation result of the endoscopic image.

[0031] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security, and national security.

[0032] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0033] Figure 1 A flowchart of a semantic segmentation method based on a nested double U-type attention network according to an embodiment of the present disclosure is schematically shown.

[0034] like Figure 1 As shown, the method includes operations S101 to S105.

[0035] In operation S101, image data of an endoscopic image is input into a first encoder to obtain first encoding features output by respective M first encoding units included in the first encoder.

[0036] In operation S102, weighted feature fusion is performed on first coding features outputted by each of the M first coding units using the N second coding units included in the second encoder to obtain second coding features outputted by each of the N second coding units.

[0037] In operation S103, the second encoding features outputted by each of the N second encoding units are processed by a first decoder to obtain a first decoding feature.

[0038] In operation S104, a second decoder is used to process the first encoding features outputted by each of the M first encoding units and the second encoding features outputted by each of the N second encoding units to obtain a second decoded feature.

[0039] In operation S105 , the first decoding feature and the second decoding feature are concatenated in the channel dimension to obtain a semantic segmentation result of the endoscopic image.

[0040] According to an embodiment of the present disclosure, both M and N may be positive integers, and N may be less than or equal to M, that is, the number of second coding units included in the second encoder may be less than or equal to the number of first coding units included in the first encoder.

[0041] According to an embodiment of the present disclosure, the first encoding unit can be represented as a convolution block, which is used to downsample the input features of the first encoding unit to obtain the output features of the first encoding unit.

[0042] According to an embodiment of the present disclosure, the second encoding unit may perform weighted fusion processing on the first encoding features of different layers. For example, the second encoding unit located in the first layer may perform weighted fusion on the first encoding features output by the first encoding unit located in the first layer and the first encoding features output by the first encoding unit located in the second layer, and process the fused features. The second encoding unit located in the first layer may refer to the first second encoding unit of the second encoder, and similarly, the first encoding unit located in the second layer may refer to the second first encoding unit of the first encoder.

[0043] According to an embodiment of the present disclosure, the first decoder uses an interpolation or transposed convolution method to perform upsampling processing based on the second encoding features output by each of the N second encoding units to obtain a first decoding feature.

[0044] According to an embodiment of the present disclosure, the second decoder can use the high-level low-resolution abstract semantics to guide the construction of a shallow high-resolution feature map based on the correlation between the features output by the first encoding unit or the second encoding unit at different layers to obtain the second decoding feature. By splicing the first decoding feature and the second decoding feature, resolution recovery based on global information can be achieved.

[0045] According to an embodiment of the present disclosure, a nested double U-type attention network is constructed based on UNet using a first encoder, a second encoder, a first decoder, and a second decoder. The first encoder can extract first coded features of different scales from the image data of the endoscopic image. The second encoder can weightedly fuse the first coded features of different scales so that the second coded features output by the second encoder can take into account information at different levels and enhance the recognition ability of feature information related to the segmentation target. The first decoder can restore the resolution by introducing a lower-level high-resolution feature, namely the second coded feature, through a skip connection to obtain a first decoded feature. The second decoder can introduce global information to obtain a second decoded feature, and use the second decoded feature to supplement and enhance the first decoded feature to supplement the information lost during the interpolation upsampling process. Through the above technical means, the accuracy of the semantic segmentation of endoscopic images can be effectively improved.

[0046] Reference below Figure 2 to Figure 7 , combined with specific embodiments Figure 1 The method shown is further explained.

[0047] According to an embodiment of the present disclosure, the image data of the endoscopic image can be directly used as the input of the nested double U-type attention network to achieve semantic segmentation of the endoscopic image. Alternatively, the endoscopic image can be preprocessed and the image data of the preprocessed endoscopic image can be used as the input of the network to complete the semantic segmentation of the endoscopic image.

[0048] According to an embodiment of the present disclosure, specifically, the endoscopic image can be cropped and scaled to obtain a target endoscopic image. The image data of the endoscopic image is input into a first encoder to obtain the first coding features output by each of the M first coding units included in the first encoder, which may include the following operations:

[0049] The image data of the target endoscopic image is input into the first encoder to obtain the first encoding features output by each of the M first encoding units.

[0050] According to an embodiment of the present disclosure, the preprocessing operation of the endoscopic image may also include operations other than image cropping and scale transformation, such as color space transformation of the image, regularization of image data, etc., which are not limited here.

[0051] According to an embodiment of the present disclosure, the first encoder may include M first encoding units at M levels respectively. The first encoding unit may perform downsampling processing on the input features. Specifically, inputting the image data of the endoscopic image into the first encoder, and obtaining the first encoding features output by each of the M first encoding units included in the first encoder may include the following operations:

[0052] For the mth first encoding unit, the mth first encoding unit is used to downsample the first encoding feature output by the m-1th first encoding unit to obtain the first encoding feature output by the mth first encoding unit.

[0053] According to an embodiment of the present disclosure, m is a positive integer less than or equal to M. When m is equal to 1, the first coding feature output by the (m-1)th first coding unit may be image data of the endoscopic image.

[0054] According to an embodiment of the present disclosure, the second encoder may include N second encoding units, and the second encoding units may perform weighted fusion processing on first encoding features of different layers. Specifically, the first encoding unit and the second encoding unit located in the same layer may be connected using an attention gate, and the attention gate may guide the network to focus on more critical information for the endoscopic image semantic segmentation task during training, thereby improving the learning efficiency and segmentation accuracy of the network.

[0055] According to an embodiment of the present disclosure, using the N second encoding units included in the second encoder to perform weighted feature fusion on the first encoding features output by each of the M first encoding units to obtain the second encoding features output by each of the N second encoding units may include the following operations:

[0056] For the nth second coding unit, the second coding feature output by the n-1th second coding unit is downsampled to obtain the first downsampled feature; the first coding feature output by the n+1th first coding unit is upsampled to obtain the first upsampled feature; the first coding feature and the first upsampled feature output by the nth first coding unit are processed using the attention gate to obtain the first attention feature; and the first downsampled feature, the first upsampled feature and the first attention feature are weightedly fused to obtain the second coding feature output by the nth second coding unit.

[0057] According to an embodiment of the present disclosure, n is a positive integer less than or equal to N. When n is equal to 1, the first down-sampling feature is a zero matrix.

[0058] Figure 2 The structural diagram of the second encoder according to the embodiment of the present disclosure is schematically shown.

[0059] like Figure 2 As shown, the first encoder may include, for example, a first encoding unit EN1_1, a first encoding unit EN1_2, and a first encoding unit EN1_3, and the second encoder may include, for example, a second encoding unit EN2_1 and a second encoding unit EN2_2. An attention gate GAT1 may be included between the first encoding unit EN1_1 and the second encoding unit EN2_1, and an attention gate GAT2 may be included between the first encoding unit EN1_2 and the second encoding unit EN2_2.

[0060] According to an embodiment of the present disclosure, the image data of the endoscopic image can be input from the first encoding unit EN1_1, and the first encoding unit EN1_1 can downsample the image data of the endoscopic image to obtain the first encoding feature F1_1. Similarly, the first encoding unit EN1_2 can downsample the first encoding feature F1_1 to obtain the first encoding feature F1_2; the first encoding unit EN1_3 can downsample the first encoding feature F1_2 to obtain the first encoding feature F1_3.

[0061] According to an embodiment of the present disclosure, the first encoding feature F1_2 output by the first encoding unit EN1_2 may be upsampled to obtain a first upsampled feature Fup1. The first upsampled feature Fup1 may be used as a gating signal, and the first encoding feature F11 output by the first encoding unit EN1_1 may be processed by the attention gate GAT1 to obtain a first attention feature Fgat1, as shown in formula (1):

[0062] Fgat1=AG(F1_1,Fup1)=AG(F1_1,UP(F1_2)) (1)

[0063] In formula (1), AG(·) can represent the processing operation of the attention gate, and UP(·) can represent the upsampling operation.

[0064] According to an embodiment of the present disclosure, the second encoding unit EN2_1 may perform weighted summation on the first attention feature Fgat1 and the first up-sampled feature Fup1 to obtain a second encoding feature F2_1 output by the second encoding unit EN2_1, as shown in formula (2):

[0065]

[0066] In formula (2), P2 and P3 can be expressed as weight coefficients, and the values ​​of the weight coefficients can be adjusted during the training process of the nested double U-type attention network. ε can be expressed as a small constant, such as 0.00001, which can be used to prevent the occurrence of unstable feature values ​​during feature fusion.

[0067] According to an embodiment of the present disclosure, similarly, the first encoding feature F13 output by the first encoding unit EN1_3 can be upsampled to obtain the first upsampled feature Fup2. The first upsampled feature Fup2 can be used as a gating signal, and the first encoding feature F1_2 output by the first encoding unit EN1_2 can be processed by the attention gate GAT2 to obtain the first attention feature Fgat2, as shown in formula (3):

[0068] Fgat2=AG(F1_2,Fup2)=AG(F1_2,UP(F1_3)) (3)

[0069] In formula (3), AG(·) can represent the processing operation of the attention gate, and UP(·) can represent the upsampling operation.

[0070] According to an embodiment of the present disclosure, before the second encoding feature F2_1 output by the second encoding unit EN2_1 is passed to the second encoding unit EN2_2, the second encoding feature F2_1 may be downsampled to obtain a first downsampled feature Fdown1, as shown in formula (4):

[0071] Fdown1=DOWN(F2_1) (4)

[0072] In formula (4), DOWN(·) may represent a downsampling operation.

[0073] According to an embodiment of the present disclosure, the second encoding unit EN2_2 may perform weighted summation on the first down-sampling feature Fdown1, the first attention feature Fgat2, and the first up-sampling feature Fup2 to obtain a second encoding feature F2_2 output by the second encoding unit EN2_2, as shown in formula (5):

[0074]

[0075] In formula (2), P1, P2 and P3 can be expressed as weight coefficients, and the values ​​of the weight coefficients can be adjusted during the training process of the nested double U-type attention network. ε can be expressed as a small constant, such as 0.00001, which can be used to prevent the occurrence of unstable feature values ​​during feature fusion.

[0076] According to an embodiment of the present disclosure, by utilizing a second encoder and an attention gate, while achieving normalized weighted feature fusion of the output of the first encoder, the depth of the decoder can also be reduced, thereby reducing the semantic gap between the encoder and the decoder, and helping the decoder to restore the resolution more accurately.

[0077] According to an embodiment of the present disclosure, the first decoder may include K decoding units, where K is a positive integer. The decoding unit may be represented as a convolution block, which is used to upsample the input features of the decoding unit to obtain the output features of the decoding unit. The second encoding unit may be jump-connected to the decoding unit at the same layer to copy the first encoding features output by the second encoding unit to the decoding unit at the same layer.

[0078] According to an embodiment of the present disclosure, the second encoding unit and the decoding unit in the same layer can be jump-connected through an attention gate. Using the first decoder to process the second encoding features output by each of the N second encoding units to obtain the first decoding features may include the following operations:

[0079] For the kth decoding unit, the first output feature output by the k-1th decoding unit is upsampled to obtain a second upsampled feature; the second encoding feature and the second upsampled feature output by the Nkth second encoding unit are processed using an attention gate to obtain a second attention feature; and based on the second upsampled feature and the second attention feature, the first output feature output by the kth decoding unit is obtained.

[0080] According to an embodiment of the present disclosure, k is a positive integer less than or equal to K. When k is equal to 1, the first output feature output by the k-1th decoding unit is the second encoding feature output by the Nth second encoding unit.

[0081] According to an embodiment of the present disclosure, when k is equal to K, the first output feature output by the kth decoding unit is the first decoding feature.

[0082] Figure 3 The structural diagram of the first decoder according to the embodiment of the present disclosure is schematically shown.

[0083] like Figure 3 As shown, the first decoder may include a decoding unit DE1 and a decoding unit DE2, and correspondingly, the second encoder may include a second encoding unit EN2_1, a second encoding unit EN2_2, and a second encoding unit EN2_3. The decoding unit DE1 may be jump-connected to the second encoding unit EN2_2 located at the second layer through an attention gate GAT3, and the decoding unit DE2 may be jump-connected to the second encoding unit EN2_1 located at the first layer through an attention gate GAT4.

[0084] According to an embodiment of the present disclosure, the second encoding feature F2_3 output by the second encoding unit EN2_3 may be upsampled to obtain a second upsampled feature Fup3. The second upsampled feature Fup3 may be used as a gating signal to process the second encoding feature F2_2 output by the second encoding unit EN2_2 using an attention gate GAT3 to obtain a second attention feature Fgat3.

[0085] According to an embodiment of the present disclosure, the decoding unit DE1 may concatenate the second up-sampled feature Fup3 and the second attention feature Fgat3 in the channel dimension to obtain the first output feature Fout1 output by the decoding unit DE1.

[0086] According to an embodiment of the present disclosure, similarly, the first output feature Fout1 output by the decoding unit DE1 can be upsampled to obtain a second upsampled feature Fup4. The second upsampled feature Fup4 can be used as a gating signal to process the second encoding feature F2_1 output by the second encoding unit EN2_1 using the attention gate GAT4 to obtain a second attention feature Fgat4.

[0087] According to an embodiment of the present disclosure, the decoding unit DE2 may concatenate the second up-sampled feature Fup4 and the second attention feature Fgat4 in the channel dimension to obtain the first output feature Fout2 output by the decoding unit DE2, that is, the first decoding feature.

[0088] According to the embodiments of the present disclosure, by normalizing and weighting the multi-scale feature maps from different layers and fusing them into the decoder through skip connections, it is possible to achieve multi-scale feature fusion while improving the efficiency of the network in learning features of different layers, thereby improving the network's expressive power.

[0089] Figure 4 The structural diagram of the attention gate according to an embodiment of the present disclosure is schematically shown.

[0090] like Figure 4As shown, the attention gate may include a first network module 401, a second network module 402, a third network module 403, a first activation function module 404, and a second activation function module 405. The first network module 401, the second network module 402, and the third network module 403 may each be composed of a 1×1 convolution layer and a functional unit that may be used for batch normalization processing. The first activation function module 404 and the second activation function module 405 may use an activation function to process the input of the module.

[0091] According to an embodiment of the present disclosure, the activation function used by the first activation function module 404 may be, for example, a ReLU activation function, and the activation function used by the second activation function module 405 may be, for example, a Sigmoid activation function. The selection of the activation functions used in the first activation function module 404 and the second activation function module 405 may be determined according to a specific application scenario, for example, the first activation function module 404 and the second activation function module 405 may both use a tanh activation function.

[0092] According to an embodiment of the present disclosure, using an attention gate to process the first encoding feature and the first up-sampled feature output by the nth first encoding unit to obtain the first attention feature may include the following operations:

[0093] The first encoding feature output by the nth first encoding unit is convolved and batch normalized to obtain a first feature map; the first up-sampled feature is convolved and batch normalized to obtain a second feature map; based on the first feature map and the second feature map, a third feature map is obtained; the third feature map is convolved and batch normalized to obtain a fourth feature map; the fourth feature map is mapped to obtain a fifth feature map; and based on the fifth feature map and the first feature map, a first attention feature is obtained.

[0094] According to an embodiment of the present disclosure, specifically, the first encoding feature output by the nth first encoding unit can be used as an input feature of the attention gate, and the first up-sampled feature can be used as a gating signal of the attention gate. The first encoding feature output by the nth first encoding unit can be input into the first network module 401 to perform convolution and batch normalization on the first encoding feature output by the nth first encoding unit to obtain a first feature map. The first up-sampled feature can be input into the second network module 402 to perform convolution and batch normalization on the first up-sampled feature to obtain a second feature map. The first feature map and the second feature map can be added pixel by pixel to obtain a first fused feature, and the first fused feature can be processed by the first activation function module 404 to obtain a third feature map. The third feature map can be input into the third network module 403 to perform convolution and batch normalization on the third feature map to obtain a fourth feature map. The fourth feature map can be processed by the second activation function module 405 to perform mapping processing on the fourth feature map to obtain a fifth feature map. The fifth feature map and the first feature map can be multiplied pixel by pixel to obtain the first attention feature.

[0095] According to an embodiment of the present disclosure, similarly, the second encoding feature and the second up-sampled feature output by the Nkth second encoding unit are processed by the attention gate to obtain the second attention feature, which may include the following features:

[0096] The second encoding feature output by the Nk-th second encoding unit is convolved and batch normalized to obtain a sixth feature map; the second up-sampled feature is convolved and batch normalized to obtain a seventh feature map; based on the sixth feature map and the seventh feature map, an eighth feature map is obtained; the eighth feature map is convolved and batch normalized to obtain a ninth feature map; the ninth feature map is mapped to obtain a tenth feature map; and based on the tenth feature map and the sixth feature map, a second attention feature is obtained.

[0097] According to an embodiment of the present disclosure, specifically, the second encoding feature output by the Nkth second encoding unit can be used as an input feature of the attention gate, and the second up-sampled feature can be used as a gating signal of the attention gate. The second encoding feature output by the Nkth second encoding unit can be input into the first network module 401 to perform convolution and batch normalization on the second encoding feature output by the Nkth second encoding unit to obtain a sixth feature map. The second up-sampled feature can be input into the second network module 402 to perform convolution and batch normalization on the second up-sampled feature to obtain a seventh feature map. The sixth feature map and the seventh feature map can be added pixel by pixel to obtain a second fused feature, and the second fused feature can be processed by the first activation function module 404 to obtain an eighth feature map. The eighth feature map can be input into the third network module 403 to perform convolution and batch normalization on the eighth feature map to obtain a ninth feature map. The ninth feature map can be processed by the second activation function module 405 to perform mapping processing on the ninth feature map to obtain a tenth feature map. The tenth feature map and the sixth feature map can be multiplied pixel by pixel to obtain the second attention feature.

[0098] According to an embodiment of the present disclosure, irrelevant information in the feature map output by the encoder can be suppressed through the attention gate, and information related to the semantic segmentation target can be enhanced, thereby improving the accuracy of semantic segmentation.

[0099] According to an embodiment of the present disclosure, the second decoder can be used as a reinforcement branch of the first decoder and can be constructed based on an SGB (Semantically Guided Block) module. Specifically, the second decoder includes P semantically guided blocks, where P is a positive integer. Each semantically guided block can be represented as an SGB module.

[0100] According to an embodiment of the present disclosure, using a second decoder to process the first coding features output by each of the M first coding units and the second coding features output by each of the N second coding units to obtain a second decoding feature may include the following operations:

[0101] For the p-th semantic guidance unit, the p-th semantic guidance unit is used to process the second output feature output by the p-1-th semantic guidance unit and the second encoding feature output by the N-p+1-th second encoding unit to obtain the second output feature output by the p-th semantic guidance unit.

[0102] According to an embodiment of the present disclosure, p is a positive integer less than or equal to P. When p is equal to 1, the second output feature output by the p-1th semantic guidance unit is the first encoding feature output by the Mth first encoding unit; when p is equal to P, the second output feature output by the pth semantic guidance unit is the second decoding feature.

[0103] Figure 5 The structural diagram of the second decoder according to the embodiment of the present disclosure is schematically shown.

[0104] like Figure 5 As shown, the second decoder may include a semantic guide unit SGB1 and a semantic guide unit SGB2, the first encoder may include a first encoding unit EN1_1, a first encoding unit EN1_2, and a first encoding unit EN1_3, and the second encoder may include, for example, a second encoding unit EN2_1 and a second encoding unit EN2_2. The first encoding unit EN1_1, the second encoding unit EN2_1, and the semantic guide unit SGB2 may be located at the same layer, and the first encoding unit EN1_2, the second encoding unit EN2_2, and the semantic guide unit SGB1 may be located at the same layer.

[0105] According to an embodiment of the present disclosure, the semantic guiding unit SGB1 can use the first encoding feature F1_3 output by the first encoding unit EN1_3 as a low-resolution input feature (Low Resolution Ratio, LR), and use the second encoding feature F2_2 output by the second encoding unit EN2_2 as a high-resolution input feature (High Resolution Ratio, HR) for processing to obtain the second output feature Fout3 of the semantic guiding unit SGB1.

[0106] According to an embodiment of the present disclosure, the semantic guiding unit SGB2 can use the second output feature Fout3 of the semantic guiding unit SGB1 as a low-resolution input feature, and use the second encoding feature F2_1 output by the second encoding unit EN2_1 as a high-resolution input feature for processing to obtain the second output feature Fout4 of the semantic guiding unit SGB2, that is, the second decoding feature.

[0107] According to an embodiment of the present disclosure, using the pth semantic guidance unit to process the second output feature output by the p-1th semantic guidance unit and the second encoding feature output by the N-p+1th second encoding unit to obtain the second output feature output by the pth semantic guidance unit may include the following operations:

[0108] The first convolution layer and the second convolution layer are used to process the second output feature output by the p-1th semantic guidance unit respectively to obtain weight features and semantic information features; based on the weight features and the semantic information features, semantic information combination features are obtained; the third convolution layer is used to process the second encoding feature output by the N-p+1th second encoding unit to obtain the third output feature; based on the semantic information feature and the third output feature, the fourth output feature is obtained; based on the semantic information combination feature and the fourth output feature, the fifth output feature is obtained; the fifth output feature and the third output feature are concatenated in the channel dimension to obtain the sixth output feature; and the fourth convolution layer is used to process the sixth output feature to obtain the second output feature output by the pth semantic guidance unit.

[0109] Figure 6 The structural diagram of the semantic guidance unit according to the embodiment of the present disclosure is schematically shown.

[0110] like Figure 6 As shown, the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer can all be represented as a group of 1×1 convolutions.

[0111] According to an embodiment of the present disclosure, for the low-resolution input feature LR, two sets of independent 1×1 convolutions can be used to process the low-resolution input feature LR to generate a weight feature and semantic information features The weight feature A and the semantic information feature B can be processed by dot product to obtain the semantic information combination feature

[0112] According to an embodiment of the present disclosure, for a high-resolution input feature HR, a 1×1 convolution can be used to adjust the number of channels of the high-resolution input feature HR to obtain a third output feature The semantic information feature B can be processed by global average pooling to obtain the semantic information pooling feature The third output feature D1 and the semantic information pooling feature D2 can be added pixel by pixel, and the number of channels can be adjusted using 1×1 convolution to obtain the fourth output feature

[0113] According to an embodiment of the present disclosure, a dot product process may be performed on the semantic information combination feature C and the fourth output feature D, that is, the fifth output feature

[0114] According to an embodiment of the present disclosure, the fifth output feature E and the high-resolution input feature HR adjusted by the number of channels, that is, the third output feature D1, can be concatenated in the channel dimension to obtain the sixth output feature Then use 1×1 convolution to adjust the number of channels of the sixth output feature F to obtain the second output feature output by the semantic guidance unit

[0115] According to an embodiment of the present disclosure, by applying a semantic guidance unit, high-level semantic information can be encoded and then combined according to weights to form a description of low-level features, thereby generating a high-resolution feature map with stronger semantic information. Global information can be effectively extracted, and the global information can be used to supplement the first decoding features of the output of the first decoder, thereby improving the accuracy of semantic segmentation.

[0116] Figure 7 The schematic diagram shows the structure of a nested double U-type attention network according to an embodiment of the present disclosure.

[0117] like Figure 7 As shown, in the nested double U-type attention network, the first encoder may include 5 first encoding units, namely, the first encoding unit EN1_1, the first encoding unit EN1_2, the first encoding unit EN1_3, the first encoding unit EN1_4 and the first encoding unit EN1_5; the second encoder may include 4 second encoding units, namely, the second encoding unit EN2_1, the second encoding unit EN2_2, the second encoding unit EN2_3 and the second encoding unit EN2_4; the first decoder may include 3 decoding units, namely, decoding unit DE1, decoding unit DE2 and decoding unit DE3; the second decoder may include 2 semantic guide units, namely, semantic guide unit SGB1 and semantic guide unit SGB2. The first encoding unit EN1_1 and the second encoding unit EN2_1 can be connected through an attention gate GAT1, the first encoding unit EN1_2 and the second encoding unit EN2_2 can be connected through an attention gate GAT2, the first encoding unit EN1_3 and the second encoding unit EN2_3 can be connected through an attention gate GAT3, and the first encoding unit EN1_4 and the second encoding unit EN2_4 can be connected through an attention gate GAT4; the second encoding unit EN2_3 can be jump-connected to the decoding unit DE1 through an attention gate GAT5, the second encoding unit EN2_2 can be jump-connected to the decoding unit DE2 through an attention gate GAT6, and the second encoding unit EN2_1 can be jump-connected to the decoding unit DE3 through an attention gate GAT7.

[0118] According to an embodiment of the present disclosure, the description of the processing flow of the image data of the endoscopic image in the nested double U-type attention network can be referred to the description of the processing flow of the image features by the first encoder, the second encoder, the first decoder and the second decoder as described above, and will not be repeated here.

[0119] According to an embodiment of the present disclosure, when training a nested double U-type attention network, the data set can be divided into a training set, a validation set, and a test set, and the training is performed using an Adam optimizer. The training parameters can be set as follows: the initial learning rate is set to 1e-4, and without using a pre-trained model, the network model training is set to iterate for a total of 250 rounds, and a model evaluation is performed every 5 rounds, and the model with the best test result is saved as the desired nested double U-type attention network.

[0120] According to an embodiment of the present disclosure, the test results of the model in the related art and the nested double U-type attention network (NDAU-Net) on the LITS2017 liver tumor segmentation challenge dataset can be shown in Table 1. The model in the related art may include Unet++, TransUnet and Deeplabv3+. The model in the related art and the nested double U-type attention network can be trained using the same training strategy. For example, the 3D images included in the LITS2017 dataset can be divided into a training set, a validation set and a test set in a ratio of 8:1:1 to ensure the independence of the data between the sets, and then divided along the Z axis to obtain a 2D dataset for network training and verification. Referring to Table 1, it can be found that the values ​​of the three test indicators of MioU, Recall and Dice of the nested double U-type attention network are 96.20%, 97.68% and 98.04% respectively, which are better than the test results of the model in the related art.

[0121] Table 1

[0122]

[0123] Figure 8 A block diagram of a semantic segmentation device based on a nested double U-type attention network according to an embodiment of the present disclosure is schematically shown.

[0124] like Figure 8 As shown, the semantic segmentation device 800 based on the nested double U-type attention network includes an input module 810, a feature fusion module 820, a first processing module 830, a second processing module 840 and a splicing module 850. The nested double U-type attention network includes a first encoder, a second encoder, a first decoder and a second decoder.

[0125] The input module 810 is used to input the image data of the endoscopic image into the first encoder to obtain the first encoding features output by each of the M first encoding units included in the first encoder, where M is a positive integer.

[0126] The feature fusion module 820 is used to use the N second encoding units included in the second encoder to perform weighted feature fusion on the first encoding features output by each of the M first encoding units to obtain the second encoding features output by each of the N second encoding units, where N is a positive integer.

[0127] The first processing module 830 is used to use the first decoder to process the second encoding features output by each of the N second encoding units to obtain a first decoding feature.

[0128] The second processing module 840 is used to use a second decoder to process the first coding features output by each of the M first coding units and the second coding features output by each of the N second coding units to obtain a second decoding feature.

[0129] The splicing module 850 is used to splice the first decoding feature and the second decoding feature in the channel dimension to obtain a semantic segmentation result of the endoscopic image.

[0130] According to an embodiment of the present disclosure, the feature fusion module 820 may include a first feature fusion unit, a second feature fusion unit, a third feature fusion unit, and a fourth feature fusion unit.

[0131] The first feature fusion unit is used to downsample the second coding feature output by the n-1th second coding unit for the nth second coding unit to obtain a first downsampled feature, wherein n is a positive integer less than or equal to N, and when n is equal to 1, the first downsampled feature is a zero matrix.

[0132] The second feature fusion unit is used to upsample the first coding feature output by the (n+1)th first coding unit to obtain a first upsampled feature.

[0133] The third feature fusion unit is used to use the attention gate to process the first encoding feature and the first up-sampling feature output by the nth first encoding unit to obtain a first attention feature.

[0134] The fourth feature fusion unit is used to perform weighted feature fusion on the first down-sampling feature, the first up-sampling feature and the first attention feature to obtain the second encoding feature output by the nth second encoding unit.

[0135] According to an embodiment of the present disclosure, the third feature fusion unit includes a first feature fusion subunit, a second feature fusion subunit, a third feature fusion subunit, a fourth feature fusion subunit, a fifth feature fusion subunit and a sixth feature fusion subunit.

[0136] The first feature fusion subunit is used to perform convolution and batch normalization processing on the first coding feature output by the nth first coding unit to obtain a first feature map.

[0137] The second feature fusion subunit is used to perform convolution and batch normalization processing on the first up-sampled features to obtain a second feature map.

[0138] The third feature fusion subunit is used to obtain a third feature map based on the first feature map and the second feature map.

[0139] The fourth feature fusion subunit is used to perform convolution and batch normalization on the third feature map to obtain a fourth feature map.

[0140] The fifth feature fusion subunit is used to perform mapping processing on the fourth feature map to obtain a fifth feature map.

[0141] The sixth feature fusion subunit is used to obtain a first attention feature based on the fifth feature map and the first feature map.

[0142] According to an embodiment of the present disclosure, the first decoder includes K decoding units, where K is a positive integer.

[0143] According to an embodiment of the present disclosure, the first processing module 830 includes a first processing unit, a second processing unit, and a third processing unit.

[0144] The first processing unit is used to upsample the first output feature output by the k-1th decoding unit for the k-th decoding unit to obtain a second upsampled feature, wherein k is a positive integer less than or equal to K, and when k is equal to 1, the first output feature output by the k-1th decoding unit is the second encoded feature output by the N-th second encoding unit.

[0145] The second processing unit is used to use the attention gate to process the second encoding feature and the second up-sampling feature output by the Nk-th second encoding unit to obtain a second attention feature.

[0146] The third processing unit is used to obtain a first output feature output by the kth decoding unit based on the second upsampling feature and the second attention feature, wherein when k is equal to K, the first output feature output by the kth decoding unit is the first decoding feature.

[0147] According to an embodiment of the present disclosure, the second processing unit includes a first processing sub-unit, a second processing sub-unit, a third processing sub-unit, a fourth processing sub-unit, a fifth processing sub-unit and a sixth processing sub-unit.

[0148] The first processing subunit is used to perform convolution and batch normalization processing on the second encoding features output by the Nk-th second encoding unit to obtain a sixth feature map.

[0149] The second processing subunit is used to perform convolution and batch normalization processing on the second up-sampled features to obtain a seventh feature map.

[0150] The third processing subunit is used to obtain an eighth feature map based on the sixth feature map and the seventh feature map.

[0151] The fourth processing subunit is used to perform convolution and batch normalization on the eighth feature map to obtain a ninth feature map.

[0152] The fifth processing subunit is used to perform mapping processing on the ninth feature map to obtain a tenth feature map.

[0153] The sixth processing subunit is used to obtain a second attention feature based on the tenth feature map and the sixth feature map.

[0154] According to an embodiment of the present disclosure, the second decoder includes P semantic guidance units, where P is a positive integer.

[0155] According to an embodiment of the present disclosure, the second processing module 840 includes a fourth processing unit.

[0156] The fourth processing unit is used to use the p-th semantic guidance unit to process the second output feature output by the p-1-th semantic guidance unit and the second encoding feature output by the N-p+1-th second encoding unit to obtain the second output feature output by the p-th semantic guidance unit.

[0157] According to an embodiment of the present disclosure, p is a positive integer less than or equal to P. When p is equal to 1, the second output feature output by the p-1th semantic guidance unit is the first encoding feature output by the Mth first encoding unit. When p is equal to P, the second output feature output by the pth semantic guidance unit is the second decoding feature.

[0158] According to an embodiment of the present disclosure, the fourth processing unit includes a seventh processing sub-unit, an eighth processing sub-unit, a ninth processing sub-unit, a tenth processing sub-unit, an eleventh processing sub-unit, a twelfth processing sub-unit, and a thirteenth processing sub-unit.

[0159] The seventh processing subunit is used to use the first convolution layer and the second convolution layer to process the second output feature output by the p-1th semantic guidance unit respectively to obtain a weight feature and a semantic information feature.

[0160] The eighth processing subunit is used to obtain a semantic information combination feature based on the weight feature and the semantic information feature.

[0161] The ninth processing subunit is used to use the third convolutional layer to process the second encoding feature output by the N-p+1th second encoding unit to obtain a third output feature.

[0162] The tenth processing subunit is used to obtain a fourth output feature based on the semantic information feature and the third output feature.

[0163] The eleventh processing subunit is used to obtain the fifth output feature based on the semantic information combination feature and the fourth output feature.

[0164] The twelfth processing subunit is used to concatenate the fifth output feature and the third output feature in the channel dimension to obtain a sixth output feature.

[0165] The thirteenth processing subunit is used to use the fourth convolutional layer to process the sixth output feature to obtain the second output feature output by the p-th semantic guidance unit.

[0166] According to an embodiment of the present disclosure, the input module 810 includes a first input unit.

[0167] The first input unit is used for downsampling the first coding feature output by the m-1th first coding unit using the m-th first coding unit to obtain the first coding feature output by the m-1th first coding unit, wherein m is a positive integer less than or equal to M, and when m is equal to 1, the first coding feature output by the m-1th first coding unit is the image data of the endoscopic image.

[0168] According to an embodiment of the present disclosure, the semantic segmentation device 800 based on the nested double U-type attention network also includes a preprocessing module.

[0169] The preprocessing module is used to perform image cropping and scale transformation on the endoscopic image to obtain a target endoscopic image.

[0170] According to an embodiment of the present disclosure, the input module 810 includes a second input unit.

[0171] The second input unit is used to input the image data of the target endoscopic image into the first encoder to obtain the first encoding features output by each of the M first encoding units.

[0172] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0173] For example, any of the input module 810, the feature fusion module 820, the first processing module 830, the second processing module 840 and the splicing module 850 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the input module 810, the feature fusion module 820, the first processing module 830, the second processing module 840 and the splicing module 850 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the input module 810, the feature fusion module 820, the first processing module 830, the second processing module 840 and the stitching module 850 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be performed.

[0174] It should be noted that the semantic segmentation device part based on the nested double U-type attention network in the embodiments of the present disclosure corresponds to the semantic segmentation method part based on the nested double U-type attention network in the embodiments of the present disclosure. The description of the semantic segmentation device part based on the nested double U-type attention network specifically refers to the semantic segmentation method part based on the nested double U-type attention network, which will not be repeated here.

[0175] Fig. 9 A block diagram of an electronic device suitable for implementing a semantic segmentation method based on a nested double U-type attention network according to an embodiment of the present disclosure is schematically shown. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0176] like Fig. 9 As shown, the computer electronic device 900 according to the embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0177] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0178] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0179] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0180] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0181] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0182] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903 .

[0183] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the semantic segmentation method based on the nested double U-type attention network provided by the embodiment of the present disclosure.

[0184] When the computer program is executed by the processor 901, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0185] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0186] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0188] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A semantic segmentation method based on a nested double U-shaped attention network, where: The nested double U-type attention network includes a first encoder, a second encoder, a first decoder and a second decoder, and the method includes: Inputting the image data of the endoscopic image into the first encoder to obtain first encoding features output by each of M first encoding units included in the first encoder, wherein M is a positive integer; Using the N second encoding units included in the second encoder to perform weighted feature fusion on the first encoding features output by each of the M first encoding units, to obtain the second encoding features output by each of the N second encoding units, wherein N is a positive integer; Using the first decoder to process the second encoding features output by each of the N second encoding units, to obtain a first decoding feature; Processing the first coding features outputted by each of the M first coding units and the second coding features outputted by each of the N second coding units using the second decoder to obtain a second decoding feature; and The first decoding feature and the second decoding feature are concatenated in the channel dimension to obtain a semantic segmentation result of the endoscopic image.

2. The method according to claim 1, wherein: The method of using the N second encoding units included in the second encoder to perform weighted feature fusion on the first encoding features outputted by the M first encoding units to obtain the second encoding features outputted by the N second encoding units includes: For the nth second encoding unit, down-sampling the second encoding feature output by the n-1th second encoding unit to obtain a first down-sampling feature, wherein n is a positive integer less than or equal to N, and when n is equal to 1, the first down-sampling feature is a zero matrix; Performing upsampling processing on the first coding feature output by the (n+1)th first coding unit to obtain a first upsampling feature; Processing the first encoding feature output by the nth first encoding unit and the first up-sampled feature using an attention gate to obtain a first attention feature; and The first down-sampling feature, the first up-sampling feature and the first attention feature are weightedly fused to obtain a second encoding feature output by the nth second encoding unit.

3. The method according to claim 2, wherein: The using of the attention gate to process the first encoding feature output by the nth first encoding unit and the first up-sampled feature to obtain the first attention feature includes: Performing convolution and batch normalization processing on the first encoding feature output by the nth first encoding unit to obtain a first feature map; Performing convolution and batch normalization processing on the first upsampled features to obtain a second feature map; Based on the first characteristic map and the second characteristic map, obtaining a third characteristic map; Performing convolution and batch normalization processing on the third feature map to obtain a fourth feature map; Performing mapping processing on the fourth feature map to obtain a fifth feature map; and Based on the fifth feature map and the first feature map, the first attention feature is obtained.

4. The method according to claim 1, wherein: The first decoder includes K decoding units, where K is a positive integer; The step of using the first decoder to process the second encoding features output by each of the N second encoding units to obtain the first decoding features includes: For the kth decoding unit, up-sample the first output feature output by the k-1th decoding unit to obtain a second up-sampled feature, wherein k is a positive integer less than or equal to K, and when k is equal to 1, the first output feature output by the k-1th decoding unit is the second encoding feature output by the Nth second encoding unit; Processing the second encoding feature output by the Nk-th second encoding unit and the second up-sampled feature using an attention gate to obtain a second attention feature; and Based on the second upsampling feature and the second attention feature, a first output feature output by the kth decoding unit is obtained, wherein when k is equal to K, the first output feature output by the kth decoding unit is the first decoding feature.

5. The method according to claim 4, wherein: The using of the attention gate to process the second encoding feature output by the Nk-th second encoding unit and the second up-sampled feature to obtain the second attention feature includes: Performing convolution and batch normalization processing on the second encoding features output by the Nk-th second encoding unit to obtain a sixth feature map; Performing convolution and batch normalization processing on the second up-sampled features to obtain a seventh feature map; Based on the sixth characteristic map and the seventh characteristic map, an eighth characteristic map is obtained; Performing convolution and batch normalization processing on the eighth feature map to obtain a ninth feature map; Performing mapping processing on the ninth feature map to obtain a tenth feature map; and Based on the tenth feature map and the sixth feature map, the second attention feature is obtained.

6. The method according to claim 1, wherein: The second decoder includes P semantic guidance units, where P is a positive integer; The step of using the second decoder to process the first coding features outputted by each of the M first coding units and the second coding features outputted by each of the N second coding units to obtain the second decoding features includes: For the p-th semantic guidance unit, using the p-th semantic guidance unit to process the second output feature output by the p-1-th semantic guidance unit and the second encoding feature output by the N-p+1-th second encoding unit, to obtain the second output feature output by the p-th semantic guidance unit; Among them, p is a positive integer less than or equal to P. When p is equal to 1, the second output feature output by the p-1th semantic guidance unit is the first encoding feature output by the Mth first encoding unit; when p is equal to P, the second output feature output by the pth semantic guidance unit is the second decoding feature.

7. The method according to claim 6, wherein: The method of using the pth semantic guidance unit to process the second output feature output by the p-1th semantic guidance unit and the second encoding feature output by the N-p+1th second encoding unit to obtain the second output feature output by the pth semantic guidance unit includes: Using the first convolution layer and the second convolution layer respectively to process the second output feature output by the p-1th semantic guidance unit to obtain a weight feature and a semantic information feature; Based on the weight feature and the semantic information feature, a semantic information combination feature is obtained; Using a third convolutional layer to process the second encoding feature output by the N-p+1th second encoding unit to obtain a third output feature; Based on the semantic information feature and the third output feature, obtain a fourth output feature; Obtaining a fifth output feature based on the semantic information combination feature and the fourth output feature; Concatenate the fifth output feature and the third output feature in the channel dimension to obtain a sixth output feature; and The sixth output feature is processed using the fourth convolutional layer to obtain a second output feature output by the p-th semantic guidance unit.

8. The method according to claim 1, wherein: The step of inputting the image data of the endoscopic image into the first encoder to obtain first encoding features outputted by each of the M first encoding units included in the first encoder includes: For the mth first encoding unit, the first encoding feature output by the m-1th first encoding unit is downsampled using the mth first encoding unit to obtain the first encoding feature output by the mth first encoding unit, wherein m is a positive integer less than or equal to M, and when m is equal to 1, the first encoding feature output by the m-1th first encoding unit is the image data of the endoscopic image.

9. The method according to claim 1, further comprising: Performing image cropping and scale transformation on the endoscopic image to obtain a target endoscopic image; The step of inputting the image data of the endoscopic image into the first encoder to obtain the first encoding features output by each of the M first encoding units included in the first encoder includes: The image data of the target endoscopic image is input into the first encoder to obtain the first encoding features output by each of the M first encoding units.

10. A semantic segmentation device based on a nested double U-shaped attention network, wherein: The nested double U-type attention network includes a first encoder, a second encoder, a first decoder and a second decoder, and the device includes: An input module, used for inputting the image data of the endoscopic image into the first encoder, and obtaining a first encoding feature outputted by each of the M first encoding units included in the first encoder, wherein M is a positive integer; A feature fusion module, used to perform weighted feature fusion on the first coding features outputted by the M first coding units respectively using the N second coding units included in the second encoder, so as to obtain the second coding features outputted by the N second coding units respectively, wherein N is a positive integer; A first processing module, configured to use the first decoder to process the second encoding features outputted by each of the N second encoding units to obtain a first decoding feature; A second processing module, configured to use the second decoder to process the first coding features outputted by each of the M first coding units and the second coding features outputted by each of the N second coding units to obtain a second decoding feature; and A splicing module is used to splice the first decoding feature and the second decoding feature in a channel dimension to obtain a semantic segmentation result of the endoscopic image.

Images