Deep learning-based sentinel lymph node ultrasound image semantic segmentation method
Through the deep learning-based RA-U-Net++ network, combined with the residual cavity pyramid module and deep supervision mechanism, the problem of poor manual intervention and segmentation results in semantic segmentation of sentinel lymph node ultrasonic images is solved, and high-precision automatic segmentation and Transfer and improve over IoU.
Patent Information
- Application Number
- CN202411071788.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-06
AI Technical Summary
The prior art requires a lot of manual intervention in semantic segmentation of sentinel lymph node ultrasound images. Manual segmentation is time-consuming and labor-intensive and has poor segmentation results. The existing methods such as FCN and U-Net have insufficient accuracy and redundancy, especially global information extraction capabilities. insufficient.
Using the RA-U-Net++ network based on deep learning, the network consisting of residual cavity pyramid module and deep supervision mechanism is constructed to perform semantic segmentation of sentinel lymph node ultrasound images, and the network parameters are tuned using the RA-U-Net++ network, combined with The output feature map of the decoding node is deeply supervised and trained, and the segmentation loss function is optimized to achieve high-precision segmentation.
Automatic segmentation of sentinel lymph node ultrasound images is realized, getting rid of the cumbersomeness of manual segmentation, and achieving a higher cross-sectional ratio IoU, improving the segmentation effect.
Smart Images

Figure CN118967704A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a sentinel lymph node ultrasound image semantic segmentation method based on deep learning. Background Art
[0002] During the treatment of breast cancer, it is crucial to understand whether the cancer has spread to the lymphatic system. Sentinel lymph nodes (SLN), as the "first stop" for cancer cell metastasis, are the key to assessing whether breast cancer has spread. Two-dimensional ultrasound and contrast ultrasound dual-modality imaging technology plays a vital role in the detection of sentinel lymph node (SLN) cancer. Although two-dimensional ultrasound and contrast ultrasound dual-modality imaging technology play a key role in SLN cancer detection, its inherent limitations and reliance on manual review of traditional diagnostic methods urgently need to be improved by introducing new technical means.
[0003] The paper "Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation [C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, 3431-440" proposed the Fully Convolutional Networks (FCN). The core contribution of FCN is to transform the traditional convolutional neural network (CNN) into a fully convolutional form, so that it can accept input images of any size and output dense prediction maps of the corresponding size. However, the segmentation results of this method are not fine enough and the cost is relatively high. At the same time, the paper "Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation [C]. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III18, 2015, 234-241" proposed the U-Net network, whose biggest feature is the unique "U"-shaped structure. This structure enables the network to achieve high-precision segmentation with less data by effectively using the contextual information of the image and precise positioning information. Although the segmentation accuracy of U-Net is improved compared to FCN, it also has the disadvantages of large redundancy and insufficient global information extraction ability. Summary of the invention
[0004] The purpose of the present invention is to provide a deep learning-based sentinel lymph node ultrasound image semantic segmentation method to address the deficiencies in the prior art, so as to achieve semantic segmentation of dual-modality sentinel lymph node ultrasound images, so as to solve the problem that the current stage of identifying the sentinel lymph node location in ultrasound images requires a lot of manual intervention, manual segmentation is time-consuming and labor-intensive, and the segmentation results are poor.
[0005] The technical solution adopted by the present invention is:
[0006] A sentinel lymph node ultrasound image semantic segmentation method based on deep learning, the method comprises the following steps:
[0007] Step 1, obtaining a sentinel lymph node ultrasound image dataset, and preprocessing the dataset to obtain a training set;
[0008] Step 2: construct a RA-U-Net++ network for image semantic segmentation, and tune the network parameters of the RA-U-Net++ network based on the training set to obtain a medical image semantic segmentation model for the target lesion;
[0009] Among them, the RA-U-Net++ network adopts a U-shaped structure, which includes L (L≥2) layers, each layer is provided with an encoding node, the encoding node adopts a residual hole pyramid module, and the output feature map of the encoding node of the current layer is input into the encoding node of the next layer after downsampling; from the first layer to the L-1 layer of the RA-U-Net++ network, decoding nodes are provided, the first layer includes L-1 decoding nodes, and one decoding node is reduced layer by layer from the first layer, in each layer, the decoding node is located after the encoding node, and the input of each decoding node includes: the output feature map of each node before the current decoding node in the same layer after jump connection, and the output feature map of the previous adjacent node in the next layer after upsampling; the final segmentation result is obtained based on the output feature map of the last decoding node of the first layer. Preferably, the output feature map of the last decoding node of the first layer can be sent to a 1×1 convolutional layer with an output channel of 2 to obtain the final segmentation result.
[0010] Furthermore, in step 2, when tuning the network parameters of the RA-U-Net++ network, deep supervised training is performed in combination with the output feature maps of all decoding nodes in the first layer. The output feature maps of each decoding node are sent to a 1×1 convolutional layer with an output channel of 2, and the segmentation loss of the output results of each convolution branch is calculated and fused to obtain the comprehensive segmentation loss.
[0011] Furthermore, the loss function used to calculate the segmentation loss is:
[0012]
[0013] Where N is the number of samples, y i represents the true label of the i-th sample, p i It represents the probability that the network predicts that the i-th sample is a positive class, and σ() represents the Sigmoid function.
[0014] Furthermore, the decoding node includes: a concatenation layer and two convolution modules in sequence, wherein the convolution module includes a convolution layer with a convolution kernel of 3×3, an activation function and a batch normalization layer in sequence.
[0015] Furthermore, preprocessing includes image screening, image classification, image cropping, image enhancement processing and image annotation.
[0016] Furthermore, the residual atrous pyramid module includes a residual connected convolutional module and an atrous spatial pyramid pooling structure;
[0017] Among them, the convolution module with residual connection includes a main branch and a shortcut connection branch, wherein the main branch includes two 3×3 convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function; the shortcut connection branch includes a 1×1 convolutional layer and a batch normalization layer in sequence;
[0018] The input of the dilated spatial pyramid pooling structure is the output feature map of the residual-connected convolution module, which includes a global average pooling branch, a dilated convolution branch, and a convolution branch.
[0019] Among them, the global average pooling branch includes 1×1 global average pooling layer, 1×1 convolution layer, batch normalization layer, ReLu activation function and upsampling in sequence;
[0020] The dilated convolution branch includes three parallel dilated convolution modules with different dilated rates. Each dilated convolution module includes a dilated convolution layer, a batch normalization layer, and a ReLu activation function in sequence.
[0021] The convolution branch includes a 1×1 convolution layer, a batch normalization layer, and a ReLu activation function in sequence;
[0022] The output feature maps of the global average pooling branch, the dilated convolution branch, and the convolution branch of the dilated spatial pyramid pooling structure are concatenated, and then the output feature map of the residual dilated pyramid module is obtained through the convolution module, where the convolution module includes a 1×1 convolution layer, a batch normalization layer, and a ReLu activation function in sequence.
[0023] The technical solution provided by the present invention brings at least the following beneficial effects:
[0024] The present invention realizes automatic segmentation of sentinel lymph node ultrasound images, and gets rid of tedious, time-consuming and labor-intensive manual segmentation; and the present invention has good segmentation effect and can obtain a higher intersection-over-union ratio (IoU). BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0026] Figure 1 A schematic diagram of the implementation process of the deep learning-based sentinel lymph node ultrasound image semantic segmentation method provided in an embodiment of the present invention;
[0027] Figure 2 A schematic diagram of the network model structure used in an embodiment of the present invention;
[0028] Figure 3 This is a schematic diagram of the standard directory format of the labelme image annotation tool used in an embodiment of the present invention;
[0029] Figure 4 It is a schematic diagram of the structure of the residual hole pyramid module used in an embodiment of the present invention. DETAILED DESCRIPTION
[0030] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.
[0031] An embodiment of the present invention provides a semantic segmentation method for sentinel lymph node ultrasound images based on deep learning, which first preprocesses the sentinel lymph node data to establish a data set to obtain a labeled data set, and then trains the constructed semantic segmentation network (RA-U-Net++ network) based on the data set to obtain a network model (medical image semantic segmentation model) for sentinel lymph node segmentation to achieve semantic segmentation of the target ultrasound image. That is, the preprocessed image data is used as the input of the RA-U-Net++ network, and the network parameters are adjusted through back propagation to obtain the optimal network structure. The artificially set gold standard and the segmentation results of the model can then be evaluated to quantify the segmentation effect of the medical image semantic segmentation model of the present invention. When applying the semantic segmentation method for sentinel lymph node ultrasound images based on deep learning provided by an embodiment of the present invention, such as Figure 1 As shown, the main process includes:
[0032] (1) Obtaining the original data set and performing pre-processing (such as image screening, image classification, preprocessing, image annotation, etc.) on it to obtain the data sets SLN2DUS and SLNCEUS that can be used for image semantic segmentation. In the embodiment of the present invention, the sentinel lymph node multimodal ultrasound images used include conventional two-dimensional ultrasound and ultrasound contrast imaging images.
[0033] In this embodiment, in order to implement the python script to achieve format conversion, the annotated json file is converted to the semantic segmentation label format of PASCAL VOC. The json files in the SLN2D_Annotated and SLNCE_Annotated folders are converted into two folders, SLN2DUS and SLNCEUS, through Python script processing. The file directory of each folder includes JPEGImages, SegmentationClass, class_name.txtSegmentationClassBinary, and SegmentationClassVisualization, where JPEGImages stores SLN contrast ultrasound images; SegmentationClass stores the annotation information corresponding to the original image, which is usually a color image, where different colors represent different categories; SegmentationClassBinary stores the binary annotated image, where each pixel has only two possible values (0 or 255); SegmentationClassVisualization stores the visualized version of the annotated image; class_name.txt records the name list of all target categories involved in the segmentation, and each line represents a category.
[0034] (2) Perform data augmentation on the datasets SLN2DUS and SLNCEUS to enrich the sample data. For example, use the Albumentations library to perform data augmentation, and the data augmentation methods used include but are not limited to: RandomRotate90 (random rotation of 90 degrees), Flip (random horizontal or vertical flip of the image), OneOf (random adjustment of image hue, brightness, contrast, saturation), etc.
[0035] The processed data set is divided into a training set Train and a test set Test. The training set Train is used for parameter tuning of the network model, which includes M original medical images and M gold standards; the test set Test is used to evaluate the segmentation performance of the trained model, which includes N original medical images Test. M , and N gold standard Test G .
[0036] (3) Based on the processed data set, the RA-U-Net++ network constructed by the embodiment of the present invention is trained through multiple data iterations to obtain an optimal model, namely, a medical image semantic segmentation model;
[0037] (4) The performance of the medical image semantic segmentation model is tested based on the test set Test, and the evaluation results are obtained based on the specified evaluation indicators. That is, each sentinel lymph node ultrasound image is used as the input of the medical image semantic segmentation model, and the model gives the segmentation result of each voxel. The segmentation result of the model is then evaluated with the gold standard annotated by the doctor. The evaluation criteria include but are not limited to: IoU (Intersection over Union), mIoU (Mean Intersection over Union), fwIoU (Frequency Weighted Intersection over Union), Acc (Accuracy), mAc (mean pixel-wise accuracy).
[0038] In the embodiment of the present invention, the constructed RA-U-Net++ network adopts a symmetrical U-shaped structure, such as Figure 2 As shown in the figure, it includes several layers (node layers), for example, 5 layers. The overall architecture of the RA-U-Net++ network can be divided into three parts: encoder-nested skip connection-decoder, and has a deep supervision mechanism. In addition, by incorporating the ASPP structure, RA-U-Net++ expands the network's receptive field and enhances the processing capabilities of semantic expression and multi-scale features. At the same time, the addition of residual connections effectively alleviates the problem of gradient disappearance and further improves the accuracy of segmentation. Figure 2 In the figure, the green arrows represent downsampling, which is used to gradually reduce the spatial resolution of the image while increasing the depth of the feature channel, allowing the network to capture more global features; the blue arrows represent upsampling, which is used to gradually restore the resolution of the feature map, reconstruct the details and structure of the image, and enable the network to make accurate pixel-level predictions; the red dotted arrows are jump connections (i.e., shortcut connections), which specifically represent splicing operations, which are used for the feature maps of the current layer nodes and the upsampled feature maps of the next layer, which can reduce the semantic gap between the feature maps of the encoder and the decoder.
[0039] The encoder of the RA-U-Net++ network is the backbone network part, which is node X 0,0 , X 1,0 , X 2,0 , X 3,0 , X 4,0 and a downsampling path composed of downsampling; at the same time, a composite module combining a residual connection and a hole space pyramid pooling structure is adopted in the encoder (the embodiment of the present invention is named as a residual hole pyramid module (ResidualASPPBlock)), Figure 2 The red nodes are used to improve the feature transmission capability of the RA-U-Net++ network and its ability in feature fusion.
[0040] The decoder of the RA-U-Net++ network is an upsampling path consisting of upsampling, skip connections, and convolution modules. The feature map from the same level is concatenated with the upsampled feature map from the previous level and input into the convolution network. Each convolution network contains two identical convolution modules. Each convolution module consists of a convolution layer with a convolution kernel of 3×3, an activation function (usually ReLU), and a batch normalization layer (BN). The feature map output by the convolution network is then upsampled and concatenated with other feature maps to be input into the next decoder node until X is reached. 4,0 The feature map is decoded and sent to a 1×1 convolutional layer with an output channel of 2 to obtain a segmented image of size h×w×2, where h and w are the same size as the original image.
[0041] During network training, the RA-U-Net++ network implements deep supervision by 1,0 , X 2,0 , X 3,0 , X 4,0 After the output of , a 1×1 convolution layer with an output channel of 2 is added. After the convolution calculation, each branch can obtain a feature map of the same size, and since the size of the feature map and the original image is also the same, it can be used to directly predict the final segmentation map. In this embodiment of the present invention, the loss function is shown in the following formula:
[0042]
[0043] Where N is the number of samples, y i represents the true label of the i-th sample, p i It represents the probability that the model predicts that the i-th sample is a positive class, and σ() represents the Sigmoid function, which is used to map the original output to the (0,1) interval.
[0044] Example
[0045] Environment construction and language selection: The operating system is Ubuntu 20.04, the CPU model is AMD EPYC754232-Core, the GPU is RTX 3090 (24GB), the memory size is 64GB, this embodiment is written in Python, the Python version is Python 3.8, the Pytorch version is PyTorch 1.11.0, and the CUDA version is Cuda 11.3. In addition, opencv-python 4.9.0.80, albumentations 1.3.1 and other dependent libraries are also installed.
[0046] 1. Make a data set:
[0047] In this example, the sentinel lymph node ultrasound image data comes from the relevant departments of Sichuan Provincial People's Hospital. A team of medical experts collected multimodal ultrasound images of the sentinel lymph nodes in the affected armpit of 763 breast cancer patients who visited the hospital from June 2017 to May 2022, including conventional two-dimensional ultrasound and ultrasound angiography images.
[0048] 1-1) Image screening. Since some of the images collected from the hospital contain unusable data, this embodiment manually selects and deletes them.
[0049] 1-2) Image classification. The images derived from the ultrasound instrument include SLN contrast ultrasound and SLN two-dimensional ultrasound. Due to the significant differences in color and features of these two images, this embodiment distinguishes them so that they can be accurately labeled and trained using appropriate model parameters.
[0050] 1-3) Preprocessing. Preprocessing includes two parts: ① Convert the image from JPEG format to PNG format, ② Resize the image and cut off the useless part. The image format exported from the ultrasound instrument is JPEG, but in the subsequent binarization process, due to the characteristics of image compression caused by the JPEG format, some pixels cannot be correctly binarized. Therefore, in order to avoid this situation, this embodiment adopts the lossless format PNG. In addition, the image exported from the ultrasound instrument is the original image, and many areas in the image are not required for training (such as the machine name, hospital logo). These additional areas will increase the image size and reduce the training speed. Therefore, this embodiment cuts off the useless information and only retains the central part of the image. For SLN contrast ultrasound images, the size after ReSize is 448×480, and for SLN two-dimensional ultrasound images, the size after ReSize is 736×496.
[0051] 1-4) Construct the labelme standardized directory format. In order to ensure the efficiency and orderliness of the labeling process, this embodiment constructs the following Figure 3 The standardized directory structure shown in Figure 1. The SLN2D_img and SLNCE_img directories store SLN two-dimensional angiography images and SLN angiography ultrasound images, respectively. Correspondingly, the SLN2D_Annotated and SLNCE_Annotated directories are used to store the annotated json label files.
[0052] 1-5) Put the preprocessed SLN contrast ultrasound image and SLN two-dimensional ultrasound image into the SLNCE_img and SLN2D_img folders respectively.
[0053] 1-6) Use labelme to label the image. First, click the CreatePolygons button to start drawing polygons, then use the mouse to draw points one by one to mark the target boundary (place the mouse on the first point and click it to automatically close the boundary). After labeling, a selection box for selecting a category will pop up, and you can select the corresponding category.
[0054] 1-7) The annotation results are json label files. SLN2D_Annotated and SLNCE_Annotated store the annotation results of SLN two-dimensional ultrasound and SLN contrast ultrasound respectively.
[0055] 1-8) Submit the labeling results to professional doctors for review on LabelMe to ensure that the sentinel lymph nodes in each image are segmented correctly.
[0056] 2. Construction of RA-U-Net++ network:
[0057] According to the network structure, use python language and pytorch framework to write and run the code.
[0058] 2-1) Encoder module implementation. The RA-U-Net++ network constructed in this embodiment is as follows Figure 2 As shown in , the network layer includes 5 layers, and the number of channels in each layer is set to 32, 64, 128, 256, and 512 respectively. The encoder uses a residual hole pyramid module. Figure 2 Indicated by the red nodes in the figure, this module integrates residual connections and ASPP (Atrousspatial pyramid pooling) on the basis of standard convolutional layers, and uses dilated convolutions with different dilation rates to expand the receptive field of the model, enabling it to capture a wider range of contextual information. At the same time, the introduction of residual connections can help gradients propagate more effectively through the network, alleviate the gradient vanishing problem, and ensure that deep features are effectively learned. In the choice of downsampling, the RA-U-Net++ network uses the classic Max Pooling.
[0059] Implementation of a convolutional network structure with residual connections (convolutional modules with residual connections). Figure 4, the structure contains two 3×3 convolutional layers with a stride of 1 and an output channel number of O (customizable) on the main line, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function. On the shortcut connection branch, there is a 1×1 convolutional layer with a stride of 1 and an output channel number of O, followed by a batch normalization layer. The 1×1 convolution is mainly used to adjust the number of channels of the input data xi so that its size is equal to the output of the main line. Subsequently, the feature map on the main line is added to the feature map on the shortcut connection branch, and then the final feature map is output through the ReLu activation function.
[0060] Implementation of the void space pyramid pooling structure. Figure 4 As shown in the figure, its structure can be divided into three parts: ① global average pooling branch; ② three dilated convolution branches with different dilation rates; ③ 1x1 convolution branch; among them, the global average pooling branch contains 1×1 global average pooling layer, 1×1 convolution layer, batch normalization layer, ReLu activation function and upsampling. The global average pooling branch is used to capture the global context information of the image. This branch first performs global average pooling on the input feature map to obtain a single feature vector, then adjusts the number of channels through 1x1 convolution, then performs batch normalization and introduces nonlinearity through ReLu function, and finally restores it to the same size as the original feature map through upsampling, which ensures that the model can utilize the global information of the entire image; the dilated convolution branch with different dilation rates consists of three parallel dilated convolutions with different dilation rates. Each module consists of a dilated convolution layer, a batch normalization layer and a ReLu activation function, among which the dilation rates are set to 6, 12, and 18 respectively. The introduction of three dilated convolution branches with different dilation rates expands the receptive field of the convolution layer, allowing the model to observe a wider area in the input image; at the same time, it can capture multi-scale contextual information in the image and improve the accuracy and robustness of segmentation or recognition; in addition, ASPP also includes a 1×1 convolution branch, which is used to capture feature information at the original scale.
[0061] The implementation of the residual void pyramid module, such as Figure 4 As shown in the figure, the residual dilute pyramid module includes a convolution module with residual connection and a dilute spatial pyramid pooling structure. The upper part is a convolution module with residual connection, and the lower part is a dilute spatial pyramid pooling structure, that is, the output feature map of the three parts of the dilute spatial pyramid pooling structure is concatenated, and then the output feature map of the residual dilute pyramid module is obtained by passing through a 1×1 convolution module (including a 1×1 convolution layer, a batch normalization layer, and a ReLu activation function in sequence).
[0062] 2-2)Decoder module implementation.
[0063] See also Figure 2In this embodiment, there are 5 node layers in total, and the backbone network is node X in the figure. n,0 (n=0,…,4), i.e., encoding nodes; each layer corresponds to a number of decoding nodes X i,j , i is used to identify the node layer identifier, j is used to identify the node identifier of each node layer, the first layer includes 4 decoding nodes, the decoding nodes decrease layer by layer, and only one encoding node is set in the last layer. That is, each decoding node includes a concatenation layer and a convolutional network including two identical convolutional modules in sequence. An upsampling layer is set between the decoding nodes of adjacent node layers to realize the splicing of the feature map from the same layer and the upsampled feature map from the previous layer by the current decoding node, and the output feature map of the last decoding node of the first layer is sent to the 1×1 convolutional layer with an output channel of 2 to obtain the final segmentation result.
[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0065] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A semantic segmentation method of sentinel lymph node ultrasound images based on deep learning, It is characterized in that The following steps are involved: Step 1, obtaining a sentinel lymph node ultrasound image dataset, and preprocessing the dataset to obtain a training set; Step 2: construct a RA-U-Net++ network for image semantic segmentation, and tune the network parameters of the RA-U-Net++ network based on the training set to obtain a medical image semantic segmentation model for the target lesion; Among them, the RA-U-Net++ network adopts a U-shaped structure, which includes L levels, where L≥2; each level is set with a coding node, and the coding node adopts a residual void pyramid module. The output feature map of the coding node of the current level is downsampled and input into the coding node of the next level; From the first layer to the L-1 layer of the RA-U-Net++ network, decoding nodes are set. The first layer includes L-1 decoding nodes, and one decoding node is reduced layer by layer starting from the first layer. In each layer, the decoding node is located after the encoding node. The input of each decoding node includes: the output feature map of each node before the current decoding node in the same layer after jump connection, and the output feature map of the previous adjacent node in the next layer after upsampling; The final segmentation result is obtained based on the output feature map of the last decoding node in the first layer.
2. The method according to claim 1, It is characterized in that The output feature map of the last decoding node of the first layer is sent to a 1×1 convolutional layer with an output channel of 2 to obtain the final segmentation result.
3. The method according to claim 1, It is characterized in that In step 2, when tuning the network parameters of the RA-U-Net++ network, deep supervised training is performed based on the output feature maps of all decoding nodes in the first layer. The output feature maps of each decoding node are sent to a 1×1 convolutional layer with an output channel of 2. The segmentation loss of the output results of each convolution branch is calculated and fused to obtain the comprehensive segmentation loss.
4. The method according to claim 3, It is characterized in that The loss function used to calculate the segmentation loss is: Where N is the number of samples, y i represents the true label of the i-th sample, p i It represents the probability that the network predicts that the i-th sample is a positive class, and σ() represents the Sigmoid function.
5. The method according to claim 1, It is characterized in that The decoding node includes: a concatenation layer and two convolution modules in sequence, wherein the convolution module includes a convolution layer with a convolution kernel of 3×3, an activation function and a batch normalization layer in sequence.
6. The method according to claim 1, It is characterized in that Preprocessing includes image screening, image classification, image cropping, image enhancement and image annotation.
7. The method according to any one of claims 1 to 6, It is characterized in that The residual atrous pyramid module includes a residual connected convolutional module and an atrous spatial pyramid pooling structure; Among them, the convolution module with residual connection includes a main branch and a shortcut connection branch, wherein the main branch includes two 3×3 convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function; the shortcut connection branch includes a 1×1 convolutional layer and a batch normalization layer in sequence; The input of the dilated spatial pyramid pooling structure is the output feature map of the residual-connected convolution module, which includes a global average pooling branch, a dilated convolution branch, and a convolution branch. Among them, the global average pooling branch includes 1×1 global average pooling layer, 1×1 convolution layer, batch normalization layer, ReLu activation function and upsampling in sequence; The dilated convolution branch includes three parallel dilated convolution modules with different dilation rates. Each dilated convolution module includes a dilated convolution layer, a batch normalization layer, and a ReLu activation function in sequence. The convolution branch includes a 1×1 convolution layer, a batch normalization layer, and a ReLu activation function in sequence; The output feature maps of the global average pooling branch, the dilated convolution branch, and the convolution branch of the dilated spatial pyramid pooling structure are concatenated, and then the output feature map of the residual dilated pyramid module is obtained through the convolution module, where the convolution module includes a 1×1 convolution layer, a batch normalization layer, and a ReLu activation function in sequence.
8. The method according to claim 7, It is characterized in that The three dilation rates of the dilated convolution branch are set to 6, 12, and 18 respectively.
9. The method according to claim 1, It is characterized in that The value of level L is set to 5, and the number of channels in each level is set to 32, 64, 128, 256, and 512 respectively.
Citation Information
Patent Citations
Semantic segmentation method based on pyramid cavity convolution network
CN111369563A
Image segmentation method, system and storage medium
CN114066903A
Skin lesion image segmentation method based on lightweight multi-scale UNet
CN116310329A
Medical image segmentation system and universal efficient medical image segmentation method
CN117274279A
Method for automatically segmenting ground-glass pulmonary nodule and computer device
WO2022100495A1
