Improved U-Net network structure for contour prediction

By adopting the SENet-154 model and SCSE attention mechanism module in the U-Net network, combined with the global pooling layer, the problems of long training time and low efficiency of traditional U-Net networks are solved, and more accurate lesion area extraction and image are achieved. segmentation.

CN118918137APending Publication Date: 2024-11-08TAIYUAN UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410641446.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-22
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional U-Net networks have problems of long training time and low efficiency in image segmentation, and it is difficult to take into account positioning accuracy and obtain context information.

Method used

Adopting an improved U-Net network structure, including replacing the encoder with the SENet-154 model, adding the SCSE attention mechanism module in the decoder, and adding a global pooling layer after the fifth encoder to highlight useful information and extract detailed features. and get global information.

Benefits of technology

More accurate image segmentation is achieved, and the lesion area can be extracted more effectively, improving the accuracy and efficiency of treatment planning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918137A_ABST
    Figure CN118918137A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, in particular to an improved U-Net network structure for contour prediction, and solves the technical problems that a traditional U-Net network structure is very slow in processing process and low in processing efficiency. The decoder comprises a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder, a fifth decoder, a central block and a global pooling layer. According to the improvement of the structure, encoders are replaced, SCSE attention mechanism modules are added in the decoders, a global pooling layer is added behind a fifth encoder, and output results of the decoders are connected. And the fused feature map, the branch segmentation content of the first prediction segmentation module and the branch segmentation contour of the second prediction segmentation module are processed in parallel through the first prediction segmentation module and the second prediction segmentation module. Image segmentation is carried out through the network structure, segmentation details are more accurate, and meanwhile excessive segmentation is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an improved U-Net network structure for contour prediction. Background Art

[0002] Using two-dimensional magnetic resonance imaging (MRI) images to segment rectal cancer regions is a key step in radiotherapy. For this task, the traditional U-Net network is used for image segmentation in existing image processing.

[0003] The traditional U-Net network structure is shown in Figure (1). The traditional U-Net network structure has two obvious disadvantages: (1) It is very slow because the network must train each patch, and because there is a lot of redundancy in the overlap between patches, the same feature will be trained multiple times, resulting in a waste of resources, longer training time and reduced efficiency. (2) Positioning accuracy and obtaining context information cannot be achieved at the same time. Large patches require more max-pooling, which will reduce positioning accuracy because max pooling will lose the spatial relationship between the target pixel and the surrounding pixels, while small patches can only see very small local information and do not contain enough background information. Summary of the invention

[0004] In order to overcome the technical defects of the traditional U-Net network structure, such as slow processing and low processing efficiency, the present invention provides an improved U-Net network structure for contour prediction, so that the big data of medical images can be processed by the U-Net network structure of the present invention to more accurately extract the lesion area, thereby completing the image segmentation task more efficiently.

[0005] The invention discloses an improved U-Net network structure for contour prediction, comprising a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder, a fifth decoder, a central block, a global pooling layer, a first prediction segmentation module and a second prediction segmentation module, wherein the first encoder, the second encoder, the third encoder, the fourth encoder and the fifth encoder are respectively connected to the first decoder, the second decoder, the third decoder, the fourth decoder and the fifth decoder correspondingly, after the original image is input into the network, it is firstly downsampled by the first encoder, the second encoder, the third encoder, the fourth encoder and the fifth encoder in sequence, so that the image resolution is reduced to 1 / 32 of the resolution of the original image, the central block and the global pooling layer are respectively connected to the output end of the fifth encoder, the output of the central block is connected to the fifth decoder, and then the image is upsampled by the fifth decoder, the fourth decoder, the third decoder, the second decoder and the first decoder in sequence, so that the image is restored to the resolution of the original image, the output of the global pooling layer is sequentially spliced ​​with the output results of the fifth decoder, the fourth decoder, the third decoder, the second decoder and the first decoder, and finally the image is obtained. The fused feature map is transmitted in two ways to the first prediction segmentation module for segmenting the content and the second prediction segmentation module for segmenting the contour; the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder have the same structure and are all constructed by adding the SE attention mechanism module on the basis of the ResNeX model; the first decoder, the second decoder, the third decoder, the fourth decoder, and the fifth decoder have the same structure and respectively include two bottleneck modules and an SCSE attention mechanism module connected in sequence. The bottleneck module uses the ResNet structure. The bottleneck module includes three first 3×3 convolutional layers connected in sequence, and each first 3×3 convolutional layer is connected with the first BatchNorm layer. After the original input is processed by the three first 3×3 convolutional layers and the three first BatchNorm layers, the output result is added to the original input, and the final addition result is processed by the first ReLU layer; the SCSE attention mechanism module consists of a cSE module and an sSE module; the central block includes two second 3×3 convolutional layers and a 2×2 maximum pooling operation layer, wherein each second 3×3 convolutional layer is connected with a second BatchNorm layer and a second ReLU layer in sequence.

[0006] The U-Net network structure described in the present invention is improved on the basis of the traditional U-Net network structure. 1. Replace the encoder. This application replaces the encoder in the traditional U-Net network structure with the SENet-154 model. The SENet-154 model is constructed by adding the SE attention mechanism module on the basis of the ResNeX model. The SE attention mechanism module highlights useful information and suppresses invalid information by weighting the channels. The SE attention mechanism module includes two operations: squeezing and excitation. 2. An SCSE attention mechanism module is added to the decoder. This is to further focus the network's attention on the shape of rectal cancer and extract more detailed features of the shape of rectal cancer. The SCSE attention mechanism module consists of a cSE module and an sSE module. The cSE module has the same structure as the sSE module. The sSE module introduces a channel squeezing and spatial excitation block, which squeezes the feature map U along the channel and excites in space. Wherein U_SCSE is the output of the SCSE attention mechanism module, and U_cSE is the output of the cSE module. This operation recalibrates the input U spatially and channel-wise at the same time. For a position (i, j, c), if it gets high importance from channel rescaling and spatial rescaling, it is given a high activation. This recalibration encourages the network to learn more meaningful feature maps related to space and channels. The bottleneck module uses the ResNet structure to make information transmission smoother. 3. Add a global pooling layer after the fifth encoder. The overall information can be better concentrated through global features, so it is necessary to introduce global features. In the traditional U-Net network, the output feature map of the fifth encoder has the largest receptive field, and global features can be obtained more conveniently. Therefore, in the improved U-Net network structure of this application, a global pooling layer is added after the fifth encoder to obtain global information. Then, the global features are further extracted through a convolutional layer, and the final extracted global features are named Center_global. 4. The output results of each decoder are connected. In order to further improve the robustness of the model at multiple scales and better combine the multi-scale feature combinations of the harvester, the output of the decoder is cascaded. The five decoder outputs in the channel dimension are connected to obtain features. The first prediction segmentation module and the second prediction segmentation module process the fused feature map in parallel, the branch of the first prediction segmentation module segments the content, the branch of the second prediction segmentation module segments the contour, and the first prediction segmentation module and the second prediction segmentation module share the network weights of the improved U-Net network structure, thereby training the network to learn content segmentation and contour segmentation. While segmenting the content, the improved U-Net network structure of the present invention directly focuses on contour segmentation. Contour information also supervises and constrains content segmentation, which makes it easier for the improved U-Net network structure of the present invention to classify pixels outside the contour as background and pixels in the contour as foreground. Therefore, the segmentation details are more accurate while avoiding over-segmentation.

[0007] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: the improved U-Net network structure described in the present invention can more accurately segment the image and extract the lesion area, thereby helping doctors to make more reasonable and effective treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0009] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 It is a structural diagram of a traditional U-Net network in the background technology; Figure 2 It is a structural diagram of the improved U-Net network structure in the present invention; Figure 3 It is a structural schematic diagram of the global pooling layer in the present invention further extracting the global feature Center_global through the convolution layer; Figure 4 is a schematic diagram of the structure of the first decoder, the second decoder, the third decoder, the fourth decoder or the fifth decoder in the present invention; Figure 5 It is a structural schematic diagram of the bottleneck module described in the present invention. DETAILED DESCRIPTION

[0011] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0012] In the description, it should be noted that the terms "first" and "second" are only used for descriptive purposes and should not be understood as indicating or implying relative importance. It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0013] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0014] The following is combined with Figure 2-Figure 5 The specific embodiments of the present invention are described in detail.

[0015] In one embodiment, Figure 2As shown, an improved U-Net network structure for contour prediction includes a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder, a fifth decoder, a central block, a global pooling layer, a first prediction segmentation module and a second prediction segmentation module, the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder are respectively connected to the first decoder, the second decoder, the third decoder, the fourth decoder, and the fifth decoder. After the original image is input into the network, it is first down-sampled by the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder in sequence to reduce the image resolution to 1 / 32 of the resolution of the original image. The central block and the global pooling layer are respectively connected to the output end of the fifth encoder, and the output of the central block is connected to the fifth decoder. Subsequently, the image is up-sampled by the fifth decoder, the fourth decoder, the third decoder, the second decoder, and the first decoder in sequence to restore the image to the resolution of the original image. The output of the global pooling layer is sequentially spliced ​​with the output results of the fifth decoder, the fourth decoder, the third decoder, the second decoder, and the first decoder to obtain a fusion image. The fused feature map is transmitted in two ways to the first prediction segmentation module for segmenting the content and the second prediction segmentation module for segmenting the contour; the first encoder, the second encoder, the third encoder, the fourth encoder and the fifth encoder have the same structure and are constructed by adding the SE attention mechanism module on the basis of the ResNeX model; the first decoder, the second decoder, the third decoder, the fourth decoder and the fifth decoder have the same structure and respectively include two bottleneck modules and the SCSE attention mechanism module connected in sequence, the bottleneck module uses the ResNet structure, the bottleneck module includes three first 3×3 convolutional layers connected in sequence, each of which is connected to the first BatchNorm layer, the original input is processed by the three first 3×3 convolutional layers and the three first BatchNorm layers, the output result is added to the original input, and the final addition result is processed by the first ReLU layer; the SCSE attention mechanism module consists of a cSE module and an sSE module; the central block includes two second 3×3 convolutional layers and a 2×2 maximum pooling operation layer, wherein each second 3×3 convolutional layer is connected to a second BatchNorm layer and a second ReLU layer in sequence.

[0016] The U-Net network structure described in the present invention is improved on the basis of the traditional U-Net network structure. 1. Replace the encoder. This application replaces the encoder in the traditional U-Net network structure with the SENet-154 model. The SENet-154 model is constructed by adding the SE attention mechanism module on the basis of the ResNeX model. The SE attention mechanism module highlights useful information and suppresses invalid information by weighting the channels. The SE attention mechanism module includes two operations: squeezing and excitation. 2. An SCSE attention mechanism module is added to the decoder. This is to further focus the network's attention on the shape of rectal cancer and extract more detailed features of the shape of rectal cancer. The SCSE attention mechanism module consists of a cSE module and an sSE module. The cSE module has the same structure as the sSE module. The sSE module introduces a channel squeezing and spatial excitation block, which squeezes the feature map U along the channel and excites in space. Wherein U_SCSE is the output of the SCSE attention mechanism module, and U_cSE is the output of the cSE module. This operation recalibrates the input U spatially and channel-wise at the same time. For position (i, j, c), if it gets high importance from channel rescaling and spatial rescaling, it is given a high activation. This recalibration encourages the network to learn more meaningful feature maps related to space and channels. The bottleneck module uses the ResNet structure to make information transmission smoother. 3. Add a global pooling layer after the fifth encoder. The overall information can be better concentrated through global features, so it is necessary to introduce global features. In the traditional U-Net network, the output feature map of the fifth encoder has the largest receptive field, and global features can be obtained more conveniently. Therefore, in the improved U-Net network structure of this application, a global pooling layer is added after the fifth encoder to obtain global information. Then, the global features are further extracted through a convolutional layer, and the final extracted global features are named Center_global. 4. The output results of each decoder are connected. In order to further improve the robustness of the model at multiple scales and better combine the multi-scale feature combinations of the harvester, the output of the decoder is cascaded. The five decoder outputs in the channel dimension are connected to obtain features. The first prediction segmentation module and the second prediction segmentation module process the fused feature map in parallel, the branch of the first prediction segmentation module segments the content, the branch of the second prediction segmentation module segments the contour, and the first prediction segmentation module and the second prediction segmentation module share the network weights of the improved U-Net network structure, thereby training the network to learn content segmentation and contour segmentation. While segmenting the content, the improved U-Net network structure of the present invention directly focuses on contour segmentation. Contour information also supervises and constrains content segmentation, which makes it easier for the improved U-Net network structure of the present invention to classify pixels outside the contour as background and pixels in the contour as foreground. Therefore, the segmentation details are more accurate while avoiding over-segmentation.

[0017] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. An improved U-Net network structure for contour prediction, characterized in that: The invention comprises a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, a first decoder, a second decoder, a third decoder, a fourth decoder, a fifth decoder, a central block, a global pooling layer, a first prediction segmentation module and a second prediction segmentation module. The first encoder, the second encoder, the third encoder, the fourth encoder and the fifth encoder are respectively connected to the first decoder, the second decoder, the third decoder, the fourth decoder and the fifth decoder. After the original image is input into the network, it is firstly downsampled by the first encoder, the second encoder, the third encoder, the fourth encoder and the fifth encoder in sequence to reduce the image resolution to 1 / 32 of the resolution of the original image. The central block and the global pooling layer are respectively connected to the output end of the fifth encoder. The output of the central block is connected to the fifth decoder. Subsequently, the image is upsampled by the fifth decoder, the fourth decoder, the third decoder, the second decoder and the first decoder in sequence to restore the image to the resolution of the original image. The output of the global pooling layer is sequentially spliced ​​with the output results of the fifth decoder, the fourth decoder, the third decoder, the second decoder and the first decoder. Finally, a fused feature map is obtained. The fused feature map The graph is divided into two paths and transmitted to the first prediction segmentation module for segmenting the content and the second prediction segmentation module for segmenting the contour respectively; the first encoder, the second encoder, the third encoder, the fourth encoder, and the fifth encoder have the same structure and are all constructed by adding the SE attention mechanism module on the basis of the ResNeX model; the first decoder, the second decoder, the third decoder, the fourth decoder, and the fifth decoder have the same structure and respectively include two bottleneck modules and an SCSE attention mechanism module connected in sequence. The bottleneck module uses the ResNet structure. The bottleneck module includes three first 3×3 convolutional layers connected in sequence. Each first 3×3 convolutional layer is connected with the first BatchNorm layer. After the original input is processed by the three first 3×3 convolutional layers and the three first BatchNorm layers, the output result is added to the original input, and the final addition result is processed by the first ReLU layer; the SCSE attention mechanism module consists of a cSE module and an sSE module; the central block includes two second 3×3 convolutional layers and a 2×2 maximum pooling operation layer, wherein each second 3×3 convolutional layer is connected with a second BatchNorm layer and a second ReLU layer in sequence.