A lung image segmentation method based on an attention mechanism parallel U-Net network

Abstract

The application discloses a lung image segmentation method based on a parallel U-Net network of an attention mechanism, and comprises the following steps: constructing a parallel U-NetA network and a U-NetB network, wherein the U-NetA network is a double-encoding path-single-decoding path structure, and the U-NetB network is a single-encoding path-single-decoding path; obtaining PET / CT image features and PET image features, using a two-modal image feature extraction block for the PET / CT image features and the PET image features of each layer, extracting information from the PET image features and the PTE / CT image features containing metabolic information based on an attention door, and obtaining two-modal image features; inputting a CT image into an encoding path of the U-NetB network to obtain CT image features; processing the two-modal image features and the CT image features through a hybrid attention mechanism, then inputting the two-modal image features and the CT image features into decoding paths of the U-NetA network and the U-NetB network respectively, and jointly participating in network upsampling; inputting feature maps of different scales obtained by the U-NetA network and the U-NetB network decoding paths into a multi-scale feature aggregation module to obtain a final image segmentation result.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and more specifically to a lung image segmentation method based on a parallel U-Net network with an attention mechanism. Background Technology

[0002] Currently, early-stage lung cancer often presents as pulmonary nodules, appearing as roundish shadows with a diameter mostly not exceeding 30 millimeters on computed tomography (CT) images. With the rapid advancement of medical imaging technology, the volume of medical image data has increased dramatically, posing a significant challenge to clinicians. The large amount of image data leads to a heavier workload for clinicians; however, because pulmonary nodules occupy a small area in the overall image and have low contrast between the lesion and the background, they are easily missed. Therefore, computer-aided diagnostic systems for automatic pulmonary nodule segmentation have become a current research hotspot.

[0003] The remarkable achievements of deep learning methods in computer semantic segmentation have led to their widespread application in medical image segmentation. A typical model is U-Net, a classic encoder-decoder structure capable of semantic-level segmentation of the entire input image, demonstrating good performance even with limited medical data. Khanna addressed the declining performance of lung CT image lesion segmentation by adding residual blocks to the U-Net network, improving segmentation performance. Liu processed the input CT image through residual blocks with different dimensions and receptive fields to extract global and local features and obtain contextual information. Gridach addressed the spatial information loss caused by U-Net pooling operations by proposing a pyramid dilation network, which integrates multiple dilated convolutions with different dilation rates to capture subtle image details. Wang incorporated channel attention mechanisms into the skip connections of U-Net for lung parenchyma segmentation, using a hybrid dilated attention convolutional layer in the final layer. Some researchers have also fully utilized target slices and their contiguous slices to provide sequence information about lesions, thereby improving segmentation accuracy. Cao proposed a dual-path residual network for 3D segmentation of lung nodules. The network inputs the target slice and two adjacent slices into an encoder with an improved residual structure, and uses a weighted sampling strategy to address training label imbalance. Lee proposed the Mu-Net network for denoising 3D images. This network first downsamples the image to generate images at different scales, and then inputs different U-Nets for images of different scales to extract image features at different scales. However, most current segmentation methods only utilize one modality of medical images, ignoring the complementarity of multimodal medical images for lesions, and the original U-Net network cannot focus on the lesion part and important feature channels in the feature map.

[0004] Therefore, the use of parallel U-Net networks based on attention mechanisms for image segmentation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a lung image segmentation method based on a parallel U-Net network with an attention mechanism. Two parallel U-Net networks are used to extract lesion features of different modalities. Sub-network UNet_A extracts CT image features, while sub-network UNet_B consists of two encoders that extract PET / CT and CT image features. Then, a multimodal feature extraction block is used in the UNet_B network to extract two-modal features from the PET / CT and PET images. Next, a hybrid attention mechanism is added to the encoding paths of the two sub-networks. Finally, a multi-scale feature aggregation block is used to extract feature maps of different scales from the decoding path features of the two sub-networks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A lung image segmentation method based on a parallel U-Net network with an attention mechanism includes the following steps:

[0008] S1. Construct parallel U-NetA and U-NetB networks, where U-NetA network is a dual-encoding-single-decoding path structure and U-NetB network is a single-encoding-single-decoding path.

[0009] S2. Input the PET / CT image into the first encoding path of the U-NetA network to obtain PET / CT image features. Input the PET image into the second encoding path of the U-NetA network to obtain PET image features. Use a two-modal image feature extraction block on the PET / CT image features of each layer of the first encoding path and the PET image features of each layer of the second encoding path. The image feature extraction block extracts information from the PET image features and PTE / CT image features containing metabolic information based on the attention gate to obtain two-modal image features.

[0010] S3. Input the CT image into the encoding path of the U-NetB network to obtain the CT image features;

[0011] S4. The two-modal image features and CT image features are processed through a hybrid attention mechanism and input into the decoding paths of the U-NetA network and U-NetB network respectively through skip connections, so that they can jointly participate in the sampling on the network.

[0012] S5. Input the feature maps of different scales obtained from the decoding paths of U-NetA and U-NetB networks into the multi-scale feature aggregation module to obtain the final image segmentation result.

[0013] Preferably, step S1 specifically includes:

[0014] In the U-NetA network, the first and second encoding paths have the same parameter configuration, with a total of 5 layers. Each layer includes two convolutional blocks, each consisting of a 3×3 convolution, batch normalization, and the activation function EeLU. The first layer has 16 convolutions, and each subsequent layer has twice the number of convolutions. The number of convolutional kernels in the decoder path is 16, 32, 64, 128, and 256. After each convolutional layer, a downsampling operation is connected. The downsampling operation is a 2×2 max pooling. After each downsampling, the image size is reduced to half of its original size.

[0015] In the U-NetB network, the encoding path has 5 layers, each layer includes 2 convolutional blocks. Each convolutional block consists of 3×3 convolutions, batch normalization, and the activation function EeLU. The first layer has 32 convolutions, and each subsequent layer has twice the number of convolutions as the previous layer. The decoding path has 4 layers.

[0016] Preferably, step S2 specifically includes:

[0017] PET / CT images are input into the first encoding path of the U-NetA network to obtain PET / CT image features. PET images are then input into the second encoding path of the U-NetA network to obtain PET image features. A two-modal image feature extraction block is used on the PET / CT image features of each layer in the first encoding path and the PET image features of each layer in the second encoding path. This image feature extraction block extracts information from the PET image features and PTE / CT image features containing metabolic information based on an attention gate, resulting in the feature map of the l-th layer of the PET / CT image. and the feature map of the PET layer l The features are concatenated, and then a 1×1 convolution and ReLU are applied to the concatenated features. A 1×1 convolution and sigmoid activation function are applied to the feature map, compressing the weights of the feature map to between 0 and 1. The output after the sigmoid function is multiplied with the original features to obtain the two-modal image features.

[0018]

[0019] Among them, Conv 1×1 () represents a 1×1 convolution and ReLU, Conv 3×3 () represents a 3×3 convolution, and σ represents the sigmoid activation function. This represents the two-modal image features of the current layer l.

[0020] Preferably, step S3 specifically includes:

[0021] The CT image features of each layer in the coding path of the U-NetB network are extracted to obtain the feature map of the l-th CT layer. Image features are

[0022] Preferably, step S4 specifically includes:

[0023] S41. Input the PET / CT feature map and the CT feature map into the spatial attention mechanism. Superimpose the two types of input feature maps, perform average pooling and max pooling on them respectively, and then concatenate the feature maps after the two pooling operations. Apply a 3×3 convolution operation to the concatenated feature map, and use sigmoid to compress the feature values ​​to between 0 and 1. Multiply the superimposed feature map with the weight values ​​after sigmoid, and then superimpose the multiplied feature map with the initial feature map.

[0024]

[0025] Where σ represents the sigmoid operation, and + represents the concatenation of feature maps from the PET / CT and CT channels. Overlaying PET / CT and CT channel feature maps. For multiplication, SA l (F) represents the spatial attention mechanism operation. and These are the PET / CT and CT feature maps of the current layer l, respectively.

[0026] S42. Convert the feature maps of the two modal images. With CT feature map The feature map χ is obtained by splicing the two parts. l The concatenated feature maps were processed using average pooling (avgPool) and max pooling (maxPool) to obtain two 1×1×C weight values. A multilayer perceptron (MLP) consisting of two fully connected layers and ReLU was then used. The first fully connected layer had C / 3 neurons connected to the ReLU, and the second fully connected layer had C neurons. The two results from the MLP were summed, and the result was passed through a sigmoid function to obtain α. α was then multiplied by χ. l After multiplication and superposition, the channel attention mechanism can be represented as:

[0027]

[0028] Wherein, + indicates channel splicing. For channel overlay, The features of the two-modal image are, Features of CT images.

[0029] Preferably, step S5 specifically includes:

[0030] The decoding paths of U-NetA and U-NetB networks are concatenated, and bilinear interpolation is used to enlarge the feature maps of different scales to 50 pixels × 50 pixels. 1×1 convolutions are used to compress the features of the four scales to 16 pixels each. The four scale features output from the decoder are concatenated, and the initial concatenated feature is denoted as F. Average pooling and MLP are then performed to obtain the channel coefficients α′, as shown in Equation 4.

[0031] α′=σ(MLP(avgPool(F))) (4)

[0032] After multiplying the channel coefficients by the concatenated feature map, a 3×3 convolution, ReLU, and 1×1 convolution, followed by a sigmoid function, are performed to obtain the coefficient β, as shown in Equation 5:

[0033] β=σ(Conv 1×1 (Conv 3×3 ,ReLU(F·α′))) (5)

[0034] Features are connected using residual joins, as shown in Equation 6:

[0035] F MFA =F·α′·β+F·α′+F (6).

[0036] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a lung image segmentation method based on an attention mechanism and a parallel U-Net network. It uses two parallel U-Net networks to extract lesion features of different modalities. The sub-network UNet_A extracts CT image features, and the sub-network UNet_B consists of two encoders to extract PET / CT and CT image features. Then, a multimodal feature extraction block is used in the UNet_B network to extract two modal features from PET / CT and PET images. Next, a hybrid attention mechanism is added to the encoding path of the two sub-networks. Finally, a multi-scale feature aggregation block is used to extract feature maps of different scales from the decoding path features of the two sub-networks. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 The attached figure is a schematic diagram of the APU-Net network structure provided by the present invention.

[0039] Figure 2The attached figure is a schematic diagram of the UNet_A network structure provided by the present invention.

[0040] Figure 3 The attached figure is a schematic diagram of the UNet_B network structure provided by the present invention.

[0041] Figure 4 The attached figure is a schematic diagram of the two-modal medical image feature extraction block structure provided by the present invention.

[0042] Figure 5 The attached figure is a schematic diagram of the hybrid attention mechanism provided by the present invention.

[0043] Figure 6 The attached figure is a schematic diagram of the multi-scale feature aggregation block structure provided by the present invention.

[0044] Figure 7a The attached figure is a lung CT image provided by the present invention.

[0045] Figure 7b The attached figure shows a PET / CT image of the lungs provided by the present invention.

[0046] Figure 8 The attached figure is a schematic diagram of the original network segmentation results provided by the present invention, wherein (a) CT image, (b) 3D grayscale value of CT image, (c) PET / CT image, (d) PET image, (e) label, (f) U-Net segmentation result, (g) MEU-Net segmentation result with three encoders, and (h) PU-Net segmentation result.

[0047] Figure 9 The attached figure is a schematic diagram of the original segmentation index radar provided by the present invention.

[0048] Figure 10 The attached figure is a schematic diagram of the network segmentation results under the attention mechanism provided by the present invention, wherein (a) CT image, (b) 3D grayscale value of CT image, (c) PET / CT image, (d) PET image, (e) label, (f) DU-Net segmentation result, (g) DSAU-Net segmentation result, (h) DCAU-Net segmentation result, (i) DMFU-Net segmentation result, and (j) APU-Net segmentation result.

[0049] Figure 11 A schematic diagram of the segmentation index radar under the attention mechanism provided by the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention discloses a lung image segmentation method based on an attention mechanism using a parallel U-Net network, comprising the following steps:

[0052] S1. Construct parallel U-NetA and U-NetB networks, where U-NetA network is a dual-encoding-single-decoding path structure and U-NetB network is a single-encoding-single-decoding path.

[0053] S2. Input the PET / CT image into the first encoding path of the U-NetA network to obtain PET / CT image features. Input the PET image into the second encoding path of the U-NetA network to obtain PET image features. Use a two-modal image feature extraction block on the PET / CT image features of each layer of the first encoding path and the PET image features of each layer of the second encoding path. The image feature extraction block extracts information from the PET image features and PTE / CT image features containing metabolic information based on the attention gate to obtain two-modal image features.

[0054] S3. Input the CT image into the encoding path of the U-NetB network to obtain the CT image features;

[0055] S4. The two-modal image features and CT image features are processed through a hybrid attention mechanism and input into the decoding paths of the U-NetA network and U-NetB network respectively through skip connections, so that they can jointly participate in the sampling on the network.

[0056] S5. Input the feature maps of different scales obtained from the decoding paths of U-NetA and U-NetB networks into the multi-scale feature aggregation module to obtain the final image segmentation result.

[0057] To further optimize the above technical solution, step S1 specifically includes:

[0058] In the U-NetA network, the first and second encoding paths have the same parameter configuration, with a total of 5 layers. Each layer includes two convolutional blocks, each consisting of a 3×3 convolution, batch normalization, and the activation function EeLU. The first layer has 16 convolutions, and each subsequent layer has twice the number of convolutions. The number of convolutional kernels in the decoder path is 16, 32, 64, 128, and 256. After each convolutional layer, a downsampling operation is connected. The downsampling operation is a 2×2 max pooling. After each downsampling, the image size is reduced to half of its original size.

[0059] In the U-NetB network, the encoding path has 5 layers, each layer includes 2 convolutional blocks. Each convolutional block consists of 3×3 convolutions, batch normalization, and the activation function EeLU. The first layer has 32 convolutions, and each subsequent layer has twice the number of convolutions as the previous layer. The decoding path has 4 layers.

[0060] To further optimize the above technical solution, step S2 specifically includes:

[0061] PET / CT images are input into the first encoding path of the U-NetA network to obtain PET / CT image features. PET images are then input into the second encoding path of the U-NetA network to obtain PET image features. A two-modal image feature extraction block is used on the PET / CT image features of each layer in the first encoding path and the PET image features of each layer in the second encoding path. This image feature extraction block extracts information from the PET image features and PTE / CT image features containing metabolic information based on an attention gate, resulting in the feature map of the l-th layer of the PET / CT image. and the feature map of the PET layer l The features are concatenated, and then a 1×1 convolution and ReLU are applied to the concatenated features. A 1×1 convolution and sigmoid activation function are applied to the feature map, compressing the weights of the feature map to between 0 and 1. The output after the sigmoid function is multiplied with the original features to obtain the two-modal image features.

[0062]

[0063] Among them, Conv 1×1 () represents a 1×1 convolution and ReLU, Conv 3×3 () represents a 3×3 convolution, and σ represents the sigmoid activation function. This represents the two-modal image features of the current layer l.

[0064] To further optimize the above technical solution, step S3 specifically includes:

[0065] The CT image features of each layer in the coding path of the U-NetB network are extracted to obtain the feature map of the l-th CT layer. Image features are

[0066] To further optimize the above technical solution, step S4 specifically includes:

[0067] S41. Input the PET / CT feature map and the CT feature map into the spatial attention mechanism. Superimpose the two types of input feature maps, perform average pooling and max pooling on them respectively, and then concatenate the feature maps after the two pooling operations. Apply a 3×3 convolution operation to the concatenated feature map, and use sigmoid to compress the feature values ​​to between 0 and 1. Multiply the superimposed feature map with the weight values ​​after sigmoid, and then superimpose the multiplied feature map with the initial feature map.

[0068]

[0069] Where σ represents the sigmoid operation, and + represents the concatenation of feature maps from the PET / CT and CT channels. Overlaying PET / CT and CT channel feature maps. For multiplication, SA l (F) represents the spatial attention mechanism operation. and These are the PET / CT and CT feature maps of the current layer l, respectively.

[0070] S42. Convert the feature maps of the two modal images. With CT feature map The feature map χ is obtained by splicing the two parts. l The concatenated feature maps were processed using average pooling (avgPool) and max pooling (maxPool) to obtain two 1×1×C weight values. A multilayer perceptron (MLP) consisting of two fully connected layers and ReLU was then used. The first fully connected layer had C / 3 neurons connected to the ReLU, and the second fully connected layer had C neurons. The two results from the MLP were summed, and the result was passed through a sigmoid function to obtain α. α was then multiplied by χ. l After multiplication and superposition, the channel attention mechanism can be represented as:

[0071]

[0072] Wherein, + indicates channel splicing. For channel overlay, The features of the two-modal image are, Features of CT images.

[0073] To further optimize the above technical solution, step S5 specifically includes:

[0074] The decoding paths of U-NetA and U-NetB networks are concatenated, and bilinear interpolation is used to enlarge the feature maps of different scales to 50 pixels × 50 pixels. 1×1 convolutions are used to compress the features of the four scales to 16 pixels each. The four scale features output from the decoder are concatenated, and the initial concatenated feature is denoted as F. Average pooling and MLP are then performed to obtain the channel coefficients α′, as shown in Equation 4.

[0075] α′=σ(MLP(avgPool(F))) (4)

[0076] After multiplying the channel coefficients by the concatenated feature map, a 3×3 convolution, ReLU, and 1×1 convolution, followed by a sigmoid function, are performed to obtain the coefficient β, as shown in Equation 5:

[0077] β=σ(Conv 1×1 (Conv 3×3 ,ReLU(F·α′))) (5)

[0078] Features are connected using residual joins, as shown in Equation 6:

[0079] F MFA =F·α′·β+F·α′+F (6).

[0080] This application uses 909 PET / CT, CT, and PET images as the training set and 117 PET / CT, CT, and PET images as the test set. Image labels were manually drawn by clinicians. CT is one of the tools for diagnosing tumors, reflecting the anatomical information of lesions. However, CT has a low detection rate for small tumor lesions and primary tumors, easily leading to missed diagnoses. Compared to normal tissue, tumor tissue has characteristics such as rapid growth and vigorous metabolism. PET utilizes these characteristics to inject tracers into the body, causing the markers to accumulate within the lesion. PET / CT combines functional imaging PET and anatomical imaging CT, and the integrated image reflects the anatomical and functional information of the lesion on the same image, obtaining the tumor's location, shape, density, metabolic information, and size, such as... Figure 8 As shown, the lesions circled in the CT image are difficult to discern with the naked eye due to density differences. However, in PET / CT images, the lesions show vigorous metabolism and are highlighted, allowing for better localization of the lesions and identification of their edges, thereby improving image segmentation accuracy.

[0081] To address the issue of insufficient pixel count for lung nodules in the original image, this paper employs Hough transform-based ROI extraction to crop the original image to a 50-pixel × 50-pixel size. To address the low contrast between lung nodules and the background, an exposure fusion-based image enhancement method is used to improve their contrast. Both sub-networks utilize the cross-entropy loss function and the Adam optimizer. The training iterations are 150, the learning rate is initialized to 0.005, and the batch size is 8. Dropout is incorporated into the network layers to prevent overfitting.

[0082] To evaluate the performance of MEAU-Net, DSC, Recall, Volumetric Overlap Error (VOE), and Relative Volume Difference (RVD) are used to assess the model's segmentation performance. The specific formulas are as follows. This paper considers positive values ​​for both the Volumetric Overlap Error and Relative Volume Difference metrics, and the smaller the values ​​of both metrics, the better.

[0083]

[0084]

[0085]

[0086]

[0087] In this model, a correctly segmented lesion area is defined as a true positive (TP), a normal tissue area segmented into a lesion area is defined as a false positive (FP), a correctly segmented normal area is defined as a true negative (TN), and a lesion area segmented into a normal area is defined as a false negative (FN). P represents the target pixel predicted by the model, and G represents the target pixel in the ground truth label.

[0088] The superiority of the parallel U-Net network was verified through three sets of experiments. Experiment 1 used the U-Net network model, with 909 CT images as the training set and 117 CT images as the test set. Experiment 2 used the Multi-Encoder U-Net (MEU-Net) network model from Chapter 3, with 909 PET / CT, CT, and PET images as the training set and 117 PET / CT, CT, and PET images as the test set. Experiment 3 used the parallel U-Net (PU-Net) network model, with 909 PET / CT, CT, and PET images as the training set and 117 PET / CT, CT, and PET images as the test set. Sub-network UNet_A had two encoders that took PET / CT and PET images as inputs, while sub-network UNet_B took CT images as input. The encoding path features of the two sub-networks were concatenated and transmitted to the corresponding decoders of the two sub-networks through skip connections. The final segmentation feature maps of the two sub-networks were then concatenated to output the lesion segmentation result. The segmentation metrics results for the parallel U-Net network architecture are shown in Table 1. Figure 8 For network segmentation results, Figure 9 This is a radar chart of the coefficients for each indicator.

[0089] Table 1. Model Segmentation Index Results

[0090]

[0091] Figure 8 The image shows the segmentation results of U-Net, MEU-Net, and PU-Net. As can be seen from the figure, PU-Net is more accurate in segmenting lesions than U-Net and MEU-Net. (See the third row of the image). Although none of the three networks could segment the fine contours of the lesions, PU-Net's ability to delineate the shape of the lesions is superior to the other two networks compared to the labels. Furthermore, Table 1 and... Figure 9 The results show that PU-Net outperforms U-Net and MEU-Net in all segmentation coefficients, with DSC being 0.32% and 0.28% higher, Recall being 0.92% and 0.78% higher, VOE being 0.19% and 0.13% lower, and RVD being 0.07% and 0.05% lower. This demonstrates that using parallel U-Net for feature extraction from multimodal medical images is effective, and that different image decoders improve the network's segmentation performance.

[0092] This study evaluated the impact of attention mechanisms and multi-scale feature aggregation blocks on network segmentation performance using four sets of experiments. All experiments were based on a parallel U-Net network (PU-Net), using 909 PET / CT, CT, and PET images each as the training set, and 117 PET / CT, CT, and PET images each as the test set. Experiment 1 used a PU-Net model without attention mechanisms and multi-scale feature aggregation blocks. Experiment 2, based on the parallel U-Net network architecture, used a Spatial Attention Parallel U-Net (SAPU-Net) model. Features from the UNet_A and UNet_B encoding paths were concatenated and input into the skip-connection spatial attention mechanism, which was then transmitted to the decoding path. Experiment 3, based on the parallel U-Net network architecture, uses a channel attention-based parallel U-Net (CAPU-Net) model. Features from the UNet_A and UNet_B encoding paths are concatenated and input into the skip connection's channel attention mechanism before being transmitted to the decoding path. Experiment 4 uses a multi-scale feature parallel U-Net (MFPU-Net) model. Based on the parallel U-Net network architecture, features from the UNet_A and UNet_B decoding paths are concatenated and input into a multi-scale feature aggregation block. The segmentation metrics for different models are shown in Table 2. Figure 10 Add different model segmentation results to the network. Figure 11 This is a radar chart of the coefficients for each indicator.

[0093] Table 2 Segmentation Index Results

[0094]

[0095] from Figure 10 It can be seen that adding spatial attention, channel attention, and multi-scale feature aggregation blocks to the network improves segmentation performance. DSAU-Net, DCAU-Net, and DMFU-Net outperform DU-Net in segmenting lesions with complex shapes. DSC, Recall, VOE, and RVD accuracy improve from 95.48%, 95.91%, 7.28%, and 7.19% to 95.81%, 96.92%, 7.1%, and 7.06%, respectively. As shown in rows 2 and 4 of the figure, DU-Net exhibits undersegmentation, while APU-Net demonstrates better segmentation accuracy for complex lesions than the network with a single added module. (See Table 2 and...) Figure 11As can be seen, the overall differences in DSC, Recall, VOE, and RVD metrics among DSAU-Net, DCAU-Net, and DMFU-Net are not significant. However, DMFU-Net has the highest evaluation metric coefficients among the three mechanisms, at 95.81%, 96.92%, 7.1%, and 7.06%, respectively. Since the multi-scale feature aggregation block aggregates the decoding path features of two sub-networks to learn feature mappings at different scales, this demonstrates the importance of network decoding path features for input images. In summary, this experiment shows that the attention module and multi-scale feature aggregation used in the model are effective in improving the segmentation performance of weak targets.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0097] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lung image segmentation method based on a parallel U-Net network with an attention mechanism, characterized in that, Includes the following steps: S1. Construct parallel U-NetA and U-NetB networks, where U-NetA network is a dual-encoding-single-decoding path structure and U-NetB network is a single-encoding-single-decoding path. S2. Input the PET / CT image into the first encoding path of the U-NetA network to obtain PET / CT image features. Input the PET image into the second encoding path of the U-NetA network to obtain PET image features. Use a two-modal image feature extraction block on the PET / CT image features of each layer of the first encoding path and the PET image features of each layer of the second encoding path. The image feature extraction block extracts information from the PET image features and PTE / CT image features containing metabolic information based on an attention gate to obtain the feature map of the l-th layer of PET / CT. and the feature map of the PET layer l The features are concatenated, and then a 1×1 convolution and ReLU are applied to the concatenated features. A 1×1 convolution and sigmoid activation function are then applied to the feature map, compressing the weights of the feature map to between 0 and 1. The output after the sigmoid function is multiplied by the original features to obtain the two-modal image features. （1） in, Represents 1×1 convolution and ReLU. This represents a 3×3 convolution. This represents the sigmoid activation function. This represents the two-modal image features of the current layer l; S3. Input the CT image into the encoding path of the U-NetB network to obtain the CT image features; S4. The two-modal image features and CT image features are processed through a hybrid attention mechanism and input into the decoding paths of the U-NetA network and U-NetB network respectively through skip connections, so that they can jointly participate in the sampling on the network. S5. Input the feature maps of different scales obtained from the decoding paths of U-NetA and U-NetB networks into the multi-scale feature aggregation module to obtain the final image segmentation result.

2. The lung image segmentation method based on a parallel U-Net network with an attention mechanism according to claim 1, characterized in that, Step S1 specifically includes: In the U-NetA network, the first and second encoding paths have the same parameter configuration, with a total of 5 layers. Each layer includes two convolutional blocks, and each convolutional block is composed of... The system consists of convolution, batch normalization, and the activation function EeLU. The first layer has 16 convolutions, and each subsequent layer has twice the number of convolutions. The number of convolution kernels in the decoder path is 16, 32, 64, 128, and 256. After each convolutional layer, a downsampling operation is connected. The downsampling operation is a 2×2 max pooling operation. After each downsampling, the image size is reduced to half of its original size. In the U-NetB network, the encoding path consists of 5 layers, each layer including 2 convolutional blocks, and each convolutional block is composed of... It consists of convolution, batch normalization, and the activation function EeLU. The first layer has 32 convolutions, and each subsequent layer has twice the number of convolutions as the previous layer, for a total of 4 layers in the decoding path.

3. The lung image segmentation method based on a parallel U-Net network with an attention mechanism according to claim 1, characterized in that, Step S3 specifically includes: The CT image features of each layer in the coding path of the U-NetB network are extracted to obtain the feature map of the l-th CT layer. Image features are .

4. The lung image segmentation method based on a parallel U-Net network with an attention mechanism according to claim 1, characterized in that, Step S4 specifically includes: S41. Input the PET / CT feature map and the CT feature map into the spatial attention mechanism. Superimpose the two types of input feature maps, perform average pooling and max pooling on them respectively, and then concatenate the feature maps after the two pooling operations. Apply a 3×3 convolution operation to the concatenated feature map, and use sigmoid to compress the feature values ​​to between 0 and 1. Multiply the superimposed feature map with the weight values ​​after sigmoid, and then superimpose the multiplied feature map with the initial feature map. （2）; Among them, This is a sigmoid operation; the plus sign indicates concatenation of feature maps from both the PET / CT and CT channels. Overlaying PET / CT and CT channel feature maps. For multiplication, For spatial attention mechanism operation, and These are the PET / CT and CT feature maps of the current layer l, respectively. S42. Convert the feature maps of the two modal images. With CT feature map Feature map obtained by splicing The concatenated feature maps were processed using average pooling (avgPool) and max pooling (maxPool) to obtain two 1×1×C weight values. A multilayer perceptron (MLP) consisting of two fully connected layers and ReLU was then used. The first fully connected layer had C / 3 neurons connected to the ReLU, and the second fully connected layer had C neurons. The two results from the MLP were summed and then passed through a sigmoid function to obtain the final weight. ,Will and After multiplication and superposition, the channel attention mechanism can be represented as: （3） in, For channel splicing, For channel overlay, The features of the two-modal image are, Features of CT images.

5. The lung image segmentation method based on a parallel U-Net network with an attention mechanism according to claim 1, characterized in that, Step S5 specifically includes: The decoding paths of U-NetA and U-NetB networks are concatenated, and bilinear interpolation is used to enlarge the feature maps of different scales to 50 pixels × 50 pixels. 1×1 convolutions are then used to compress the features at each of the four scales to 16 pixels. The four scale features output from the decoder are then concatenated, with the initial concatenated feature denoted as F. Average pooling and MLP are then performed to obtain the channel coefficients. As shown in Equation 4: （4） The channel coefficients are multiplied by the concatenated feature map, then subjected to 3×3 convolution, ReLU and 1×1 convolution, and Sigmoid to obtain the coefficients. As shown in Equation 5: （5） Features are connected using residual joins, as shown in Equation 6: （6）。

Images