Image segmentation method and system based on multi-scale residual codec network

Abstract

The present invention discloses an image segmentation method and system based on a multi-scale residual codec network, which pre-processes a skin lesion image; performs data enhancement on the pre-processed image to obtain more image data, i.e., an input feature image; performs feature extraction, encoding feature fusion and decoding feature fusion on the input feature map based on the multi-scale residual codec network; performs maximum pooling, average pooling and soft pooling on the spliced ​​feature F to obtain its spatial attention feature, and then performs a channel attention operation on it to obtain the feature after channel attention, and then performs a sigmoid activation operation on the feature to obtain the final segmentation result; the skin lesion segmentation network Ms RED has a better segmentation effect, especially for lesions with irregular shapes, large scale changes, unclear contrast and blurred boundaries, and has a good segmentation result; the Ms RED network designed by the present invention has great application potential in clinical skin disease diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence or deep learning technology, and specifically relates to an image segmentation method and system based on a multi-scale residual codec network. Background Art

[0002] Skin diseases are among the most common diseases, among which skin cancer has a very high mortality rate. For example, the five-year survival rate of melanoma is less than 15%. As a non-invasive imaging tool, dermatoscopes are widely used to assist in the screening and diagnosis of skin lesions. However, manual screening of skin disease images is a time-consuming, labor-intensive and subjective task. In order to solve the above problems, computer-aided diagnosis technology has been introduced into the daily diagnosis of clinicians to help clinicians screen and diagnose skin lesions efficiently. In the auxiliary diagnosis of skin diseases, a key step is to accurately locate the boundary of the lesion, that is, skin lesion segmentation. The accuracy of lesion segmentation will affect the accuracy of the entire auxiliary diagnosis system. Therefore, researchers have invested a lot of energy in designing automatic skin lesion segmentation algorithms, including classic machine learning algorithms and currently popular deep learning algorithms.

[0003] Classical machine learning algorithms mainly include: clustering algorithms, threshold algorithms, variability contour models, and region growing algorithms. These algorithms rely heavily on manual features and sophisticated post-processing methods, resulting in poor performance in complex application scenarios. On the other hand, deep learning methods can automatically learn high-dimensional features without manual intervention, and their accuracy exceeds that of classical machine learning algorithms, so they have a dominant position in the field of skin disease segmentation. The codec structure is the most popular image segmentation framework. In the codec network architecture, the encoder is usually used to extract image features, while the decoder reconstructs the extracted features to the output image size and outputs the final segmentation result. In the present invention, we compare the Ms RED we designed with a variety of well-known segmentation networks, including: FCN, U-Net, U-Net++, AttU-Net, DeepLabv3+, DenseASPP, CA-Net, DO-Net, CE-Net.

[0004] FCN: It converts the fully connected layers in traditional CNN into convolutional layers, so that the network can adapt to inputs of any size. Most subsequent deep learning segmentation algorithms are built based on FCN.

[0005] U-Net: Combining the FCN architecture and the idea of ​​skip connection, Ronneberger et al. designed the famous U-Net network, which has achieved a lot of success in the field of medical image segmentation.

[0006] U-Net++: Based on U-Net, the different layers of U-Net encoding and decoding are connected so that the network can automatically learn the importance of features at different layers.

[0007] AttU-Net: Introduces the attention mechanism into the U-Net structure to enhance the network’s ability to learn useful features and inhibit the network’s learning of invalid features.

[0008] DeepLabv3+: An encoding and decoding network built based on the ASPP structure. ASPP stands for the atrous spatial pyramid pooling module, which can obtain multi-scale features without increasing the amount of computation.

[0009] DenseASPP: In order to obtain better boundary segmentation results, DenseASPP uses densely connected ASPP modules to effectively fuse features of different scales and retain more detail information during feature transfer.

[0010] CA-Net: It contains three attention mechanism modules, including spatial attention, channel attention and scale attention. The network can provide better segmentation performance with very few parameters.

[0011] DO-Net: Combining ASPP and LSTM modules, it simulates the correlation between multi-scale features and effectively extracts the contextual information of features.

[0012] CE-Net: Combines the Inception-ResNet-V2 module and the ASPP module to generate multi-scale features with a larger receptive field. Existing skin lesion segmentation algorithms based on deep learning have two main disadvantages: (1) The algorithm model has a huge number of parameters and cannot be used when computing resources are limited; (2) The segmentation results are not ideal for lesions with large size changes, irregular shapes, unclear contrast (the contrast between the lesion area and the background is not obvious), and blurred boundaries.

[0013] The reasons for the above two shortcomings are: (a) The feature extractor in the above comparison method fails to mention rich multi-scale features, resulting in poor segmentation effect for lesions with large size changes and irregular shapes; the traditional convolutional layer can only mention features of a fixed size, while the ASPP module usually extracts features of four scales. (b) The above comparison method fails to effectively integrate the multi-scale features between different layers in the encoding and decoding stage, and fails to fully integrate the context information of the features, resulting in poor segmentation effect for lesions with unclear foreground and background contrast and blurred boundaries. Summary of the invention

[0014] In order to solve the problems existing in the prior art, the present invention provides an image segmentation method and system based on a multi-scale residual codec network: a multi-scale residual codec network (Ms RED) is proposed, which has the advantages of small number of parameters and accurate segmentation; the number of parameters of Ms RED is only 3.8M, and it can be used when computing resources are limited. Ms RED has good segmentation effect, especially for lesions with large size changes, irregular shapes, unclear contrast, and blurred boundaries, the segmentation effect is significantly better than similar methods.

[0015] In order to achieve the above object, the technical solution adopted by the present invention is: an image segmentation method based on a multi-scale residual codec network, comprising the following steps: preprocessing the skin lesion image;

[0016] Performing data enhancement on the preprocessed image to obtain more image data, namely, input feature image;

[0017] Based on the multi-scale residual codec network, feature extraction, encoding feature fusion and decoding feature fusion are performed on the input feature image; wherein the multi-scale residual codec network is based on CS2-Net, and a multi-scale feature extraction module M2F2 is used to replace the convolution module in CS2-Net, and a multi-scale residual encoding feature fusion module MsR-EFM is used to integrate the multi-scale context information between different layers in the network encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is used to integrate the multi-scale context information between different layers in the network decoding stage;

[0018] Four branches are set in the multi-scale feature extraction module M2F2, each of which is provided with a convolution layer, a multi-scale feature extraction unit and a feature splitting and splicing unit. Each branch convolves the input feature map and extracts multi-scale features to obtain the corresponding feature map, and then the feature splicing unit splices the feature map to obtain the multi-scale feature map to be extracted; the multi-scale feature maps of different layers in the encoding stage are used as the input of the multi-scale residual coding feature fusion module MsR-EFM;

[0019] The multi-scale residual coding feature fusion module MsR-EFM is provided with a convolution unit, a pooling unit and a weight allocation unit. The convolution unit is used to change the number of channels of the multi-scale feature map, the pooling unit is used to reduce the size of the feature map, and the weight allocation unit is used to assign weights to feature images of different layers respectively.

[0020] The multi-scale residual decoding feature fusion module MsR-DFM is equipped with upsampling units, convolution units, pooling units and multi-layer perceptrons. The upsampling unit is used to unify the feature maps of different layers in the decoding stage to the size required for the final output. The convolution unit is used to capture the local information of the feature map and change the number of channels of the feature map. The pooling unit and the multi-layer perceptron are used to refine the spatial information of the feature map to obtain the spatial attention of the feature map, perform maximum pooling, average pooling and soft pooling on the concatenated feature F to obtain its spatial attention feature, and then perform channel attention operation on it to obtain the feature after channel attention, and then perform sigmoid activation operation on the feature to obtain the final segmentation result.

[0021] The multi-scale feature extraction module M2F2 changes the number of channels of the input feature map to the number of channels required for the output feature map through convolution, and the obtained feature map is recorded as F1; the number of channels of the input feature map is reduced to 1 / 4 through convolution, and then multi-scale features are extracted, and the obtained feature map is recorded as F2; ​​the number of channels of the input feature map is reduced to 1 / 8 through convolution, and then multi-scale features are extracted, and the obtained feature map is recorded as F3; the number of channels of the input feature map is reduced to 1 / 8 through the number of convolution layers, and then multi-scale features are extracted after convolution, and the obtained feature map is recorded as F4; the feature maps F2, F3 and F4 are spliced ​​together and recorded as F 234 , the feature map F 234 The number of channels becomes the same as the output feature, denoted as F 123 ; The final output feature map of M2F2 is F1+F 123 .

[0022] In the multi-scale residual coding feature fusion module MsR-EFM, the feature images of different layers in the network coding stage are first resampled to the same size through convolution units and pooling units to obtain the resampled feature images. At the same time, weights are assigned to each layer respectively to obtain the effectively fused multi-scale coding features.

[0023] In the multi-scale residual decoding feature fusion module MsR-DFM, the feature images of different layers are first resampled to the same size through bilinear difference upsampling, and then convolution processing is performed. The feature maps obtained after convolution processing are spliced ​​along the convolution channel dimension to obtain the spliced ​​feature F.

[0024] When preprocessing the skin lesion image, the acquired original skin lesion image is processed to obtain an image with a preset pixel size as the input of the neural network.

[0025] During data augmentation, the preprocessed images are randomly rotated by -20 degrees to 20 degrees, horizontally flipped, vertically flipped, and randomly cropped to obtain more training data.

[0026] On the other hand, the present invention also provides a skin lesion image segmentation system based on a multi-scale residual codec network, including an image data processing module and a feature codec network module;

[0027] The image data processing module is used to preprocess the image and perform data enhancement to obtain more training data;

[0028] The feature encoding and decoding network module performs feature extraction, encoding feature fusion and decoding feature fusion on the input feature map based on a multi-scale residual encoding and decoding network; wherein the multi-scale residual encoding and decoding network is based on CS2-Net, and a multi-scale feature extraction module M2F2 is used to replace the convolution module in CS2-Net, and a multi-scale residual encoding feature fusion module MsR-EFM is used to integrate the multi-scale context information between different layers in the network encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is used to integrate the multi-scale context information between different layers in the network decoding stage;

[0029] Four branches are set in the multi-scale feature extraction module M2F2, each of which is provided with a convolution layer, a multi-scale feature extraction unit and a feature splitting and splicing unit. Each branch convolves the input feature map and extracts multi-scale features to obtain the corresponding feature map, and then the feature splicing unit splices the feature map to obtain the multi-scale feature map to be extracted; the multi-scale feature maps of different layers in the encoding stage are used as the input of the multi-scale residual coding feature fusion module MsR-EFM;

[0030] The multi-scale residual coding feature fusion module MsR-EFM is provided with a convolution unit, a pooling unit and a weight allocation unit. The convolution unit is used to change the number of channels of the multi-scale feature map, the pooling unit is used to reduce the size of the feature map, and the weight allocation unit is used to assign weights to feature images of different layers respectively.

[0031] The multi-scale residual decoding feature fusion module MsR-DFM is equipped with upsampling units, convolution units, pooling units and multi-layer perceptrons. The upsampling unit is used to unify the feature maps of different layers in the decoding stage to the size required for the final output. The convolution unit is used to capture the local information of the feature map and change the number of channels of the feature map. The pooling unit and the multi-layer perceptron are used to refine the spatial information of the feature map to obtain the spatial attention of the feature map, perform maximum pooling, average pooling and soft pooling on the concatenated feature F to obtain its spatial attention feature, and then perform channel attention operation on it to obtain the feature after channel attention, and then perform sigmoid activation operation on the feature to obtain the final segmentation result.

[0032] In addition, the present invention provides a computer device, including a processor and a memory, the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and when the processor executes the computer executable program, it can implement the image segmentation method based on the multi-scale residual codec network described in the present invention.

[0033] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image segmentation method based on a multi-scale residual codec network of the present invention can be implemented.

[0034] Compared with the prior art, the present invention has at least the following beneficial effects:

[0035] The skin lesion segmentation network Ms RED designed by the present invention has a small number of parameters, only 3.8M, and can be used when computing resources are limited;

[0036] The skin lesion segmentation network Ms RED designed by the present invention has better segmentation effect, especially for lesions with irregular shapes, large scale changes, unclear contrast, and blurred boundaries.

[0037] The Ms RED network designed by the present invention has great application potential not only in clinical skin disease diagnosis, but also in other image segmentation fields.

[0038] Furthermore, the M2F2 of the present invention can extract features of 13 different scales with much fewer parameters than traditional convolution. Thanks to this advantage of M2F2, the network Ms RED designed based on M2F2 of the present invention has fewer parameters and stronger learning ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is the network structure diagram of the multi-scale residual codec network Ms RED.

[0040] Figure 2 It is the multi-scale feature extraction module M2F2.

[0041] Figure 3 It is the multi-scale residual coding feature fusion module MsR-EFM.

[0042] Figure 4 It is the multi-scale residual decoding feature fusion module MsR-DFM.

[0043] Figure 5 The segmentation effects of different networks. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with the accompanying drawings.

[0045] The present invention provides a skin lesion segmentation network: a multi-scale residual codec network (Ms RED). The present invention is described in detail from five aspects: the data used; the network settings; the evaluation criteria for the segmentation results; the structure of the Ms RED network; and the processing results.

[0046] (1) Data used

[0047] The present invention uses two public skin disease segmentation datasets (ISIC2018, PH2) to verify the designed segmentation network Ms RED. ISIC2018 is a large-scale dermoscopic image published by the International Skin Imaging Collaboration (ISIC) in 2018, which contains 2594 RGB images and has become a benchmark dataset for the evaluation of skin disease segmentation algorithms. In the invention, the original image is resampled to 224×320 size and divided into training set, test set and validation set according to 70%, 20% and 10% respectively.

[0048] PH2 is a small dataset containing only 200 dermoscopic images. The original images were resampled to 224×320 size, and 80 images were used as training set, 20 as validation set, and 100 as test set.

[0049] The training data includes the original data and the corresponding ground truth annotations: 0 represents the background and 1 represents the lesion area.

[0050] (2) Experimental setup

[0051] The present invention is implemented based on the PyTorch library and runs on an NVIDIA 2080Ti graphics card with a video memory of 11G. Adam is used as the optimizer, and the learning rate descent strategy selects CosineAnnealingWarmRestarts (T0=10, Tmult=2). The initial learning rate is set to 0.001, and the weight dcay is set to 0.00005. All networks are trained for 250 epochs, and the model with the highest Jaccard Index on the validation set is selected as the final model, and its performance is tested on the test set; data enhancement strategies are used on both ISIC2018 and PH2 datasets to prevent overfitting. The data enhancement strategy includes four random operations: random rotation from -20 degrees to 20 degrees, horizontal flipping, vertical flipping, and random cropping. In order to obtain reliable performance of each network, a five-fold cross-validation method is used, and the average result of five experiments is taken as the final performance.

[0052] (3) Evaluation indicators

[0053] Four indicators, Jaccard Index (JI), Accuracy (ACC), Recall and Precision, are used to evaluate the performance of different methods.

[0054]

[0055]

[0056]

[0057]

[0058] Among them, TP, TN, FP, and FN are true positive, true negative, false positive, and false negative, respectively.

[0059] (4) Ms RED network structure

[0060] Figure 1 The network structure diagram of a multi-scale residual codec network Ms RED is shown in the figure. The Ms RED network described in the present invention is designed based on CS2-Net. On the basis of CS2-Net, a multi-scale feature extraction module M2F2 is used to replace the traditional convolution operation. A multi-scale residual coding feature fusion module MsR-EFM is designed in the encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is designed in the decoding stage. MsR-EFM and MsR-DFM can adaptively fuse the multi-scale context features of the encoding and decoding stages, retain more detailed information in the process of feature transmission, and provide more accurate segmentation results. Figure 2 and Figure 3 The innovations are as follows: (1) In E2, E3, E4, E5 and D4, D3, D2, D1, a redesigned multi-scale feature extraction module M2F2 is used to replace the traditional convolution module to extract rich multi-scale feature information; (2) a multi-scale residual encoding feature fusion module MsR-EFM is designed in the encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is designed in the decoding stage.

[0061] A multi-scale feature extraction module M2F2 is Figure 2 As shown in the figure, in_c represents the number of channels of the convolution input, out_c represents the number of channels of the convolution output, and kernel size represents the size of the convolution kernel. Conv(rate) represents the filling rate of the dilated convolution. Figure 2As shown in the figure, M2F2 has four branches, which are called branch 1, branch 2, branch 3, and branch 4 from top to bottom. On branch 1, a 1×1 convolution is used to change the number of channels of the input feature map to the number of channels required for the output feature map, and the resulting feature map is recorded as F1; on branch 2, a 1×1 convolution is first used to reduce the number of channels of the input feature map to 1 / 4 to reduce the number of parameters for subsequent operations, and then a SplitB module with a filling rate of 1 is used to extract multi-scale features, and the resulting feature map is recorded as F2. The SplitB module consists of dilated convolution, channel splitting and channel merging techniques, which can extract rich multi-scale features with less computational effort; on branch 3, first use a 1×1 convolution to reduce the number of channels of the input feature map to 1 / 8, then use a 3×3 convolution, and finally use a SplitB module with a filling rate of 3 to extract multi-scale features, and the resulting feature map is denoted as F3; on branch 4, first use a 1×1 convolution to reduce the number of channels of the input feature map to 1 / 8, then use two 3×3 convolutions in series, and finally use a SplitB module with a filling rate of 5 to extract multi-scale features, and the resulting feature map is denoted as F4; the features obtained after branches 2, 3, and 4 pass through the SplitB module are concatenated and denoted as F 234 , after a 1×1 convolution, the number of its channels becomes the same as the number of channels of the output feature, denoted as F 123 ; The final output of M2F2 is F1+F 123 .

[0062] M2F2 has two advantages: one is that it has a small number of parameters; the other is that it can extract rich multi-scale features. Compared with traditional convolution operations, the number of parameters of M2F2 is only one-eighth of that of traditional convolution; compared with existing multi-scale feature extraction modules, such as the Inception module and the ASPP module, M2F2 can extract richer multi-scale features. Generally speaking, the Inception module and the ASPP module with four channels can extract features of four different sizes, while the M2F2 of the present invention can extract features of 13 different scales. Thanks to these two advantages of M2F2, the network Ms RED designed by the present invention based on M2F2 has fewer parameters and stronger learning ability.

[0063] Multi-scale residual coding feature fusion module MsR-EFM Figure 3 As shown, it includes a convolution unit with a convolution kernel size of 3 and a stride of 2, and a pooling unit with a stride of 2. In this module, the convolution unit is used to change the number of channels of the feature map and reduce the size of the feature map by half; the pooling unit is used to reduce the size of the feature map by half.

[0064] The multi-scale residual coding feature fusion module MsR-EFM can adaptively fuse the multi-scale features of different layers in the Ms RED network coding stage and effectively fuse the context information of the network. In order to effectively fuse the features of different layers in the network coding stage, it is first necessary to unify these feature maps to the same size. Figure 3 As shown, the feature map size in the E1 layer is 32×224×320, the feature map size in the E2 layer is 64×112×160, the feature map size in the E3 layer is 128×56×80, and the feature map size in the E4 layer is 256×28×40. The present invention first resamples E1, E2, and E3 to the size of E4, i.e., 256×28×40, through a convolution unit with a step size of 2 and a pooling unit with a step size of 2. Different weights w1, w2, w3, and w4 are given to the resampled E1, E2, E3, and E4, respectively, so that the network can adaptively learn the importance of features at different layers, effectively fuse features at different layers, and provide more valuable feature information for subsequent tasks.

[0065] Multi-scale residual decoding feature fusion module MsR-DFM Figure 4 As shown in the figure, the multi-scale residual decoding feature fusion module MsR-DFM can adaptively fuse the multi-scale features of different layers in the Ms RED network decoding stage, effectively fuse the context information of the network, and provide more accurate segmentation results. The MsR-DFM module includes upsampling units, convolution units, pooling units, and multi-layer perceptrons; the upsampling unit is used to unify the feature maps of different layers in the decoding stage to the size required for the final output, that is, 224×320; the convolution unit is used to capture the local information of the feature map and change the number of channels of the feature map; the pooling unit and the multi-layer perceptron are used to refine the spatial information of the feature map to obtain the spatial attention of the feature map.

[0066] In order to effectively fuse the features of different layers in the network decoding stage, it is first necessary to unify these feature maps to the same size. A multi-scale residual decoding feature fusion module MsR-DFM is shown in Figure 4 As shown in the figure, the feature map size in the D4 layer is 256×28×40, the feature map size in the D3 layer is 128×56×80, the feature map size in the D2 layer is 64×112×160, and the feature map size in the D1 layer is 32×224×320. First, D4, D3, and D2 are resampled to the size of D1, i.e., 32×224×320, by upsampling using bilinear difference. Then, a convolution operation with a convolution kernel size of 3*3 and an output channel number of 4 is used to uniformly transform the resampled D4, D3, D2, and D1 into a 4×224×320 feature map; the feature maps are concatenated along the channel dimension to form a 16×224×320 feature map, denoted as F.

[0067] The concatenated feature map F is subjected to maximum pooling, average pooling, and soft pooling operations to obtain its spatial attention features, and then the channel attention operation is performed to obtain the feature map after channel attention, and then the sigmoid activation operation is performed on the feature map to obtain the final output result. The output result is a picture of size 224×320, which represents the segmentation result. Figure 3 A multi-scale residual coding feature fusion module MsR-EFM is shown, where stride represents the step size in convolution and pooling.

[0068] (5) Experimental results. Tables 1 and 2 show the performance of different models on the ISIC2018 dataset and the PH2 dataset, respectively. It can be seen that the model with the smallest number of parameters is CA-Net, but its segmentation index is far behind that of the designed Ms RED. Except for CA-Net, the model of the present invention has much fewer parameters than the comparison method, and its performance is better than the results obtained by other methods.

[0069] It is worth noting that PH2 is a very small data set with only 200 data examples, and the present invention uses 80 data examples as a training set and 100 data examples as a test set. In this case, Ms RED still performs well.

[0070] Experimental results show that Ms RED is a network with few parameters, good performance and low dependence on training data. Compared with other methods, Ms RED has obvious advantages.

[0071] Table 1 Performance of different models on the ISIC2018 dataset. Bold indicates the best results among the comparison methods.

[0072]

[0073] Table 2 Performance of different models on the PH2 dataset. Bold indicates the best results among the comparison methods.

[0074]

[0075]

[0076] Figure 5 The segmentation effects of different networks on ISIC2018 and PH2 are shown. The first four rows show the segmentation effects of different networks on the ISIC2018 dataset, and the last four rows show the segmentation effects of different networks on the PH2 dataset. It can be seen from the figure that the segmentation results of Ms RED are significantly better than the comparison methods, especially for lesions with irregular shapes, large scale changes, unclear contrast, and blurred boundaries.

[0077] The present invention also provides a skin lesion image segmentation system based on a multi-scale residual codec network, comprising an image data processing module and a feature codec network module;

[0078] The image data processing module is used to preprocess the image and perform data enhancement to obtain more training data;

[0079] The feature encoding and decoding network module performs feature extraction, encoding feature fusion and decoding feature fusion on the input feature image based on a multi-scale residual encoding and decoding network; wherein the multi-scale residual encoding and decoding network is based on CS2-Net, and a multi-scale feature extraction module M2F2 is used to replace the convolution module in CS2-Net, and a multi-scale residual encoding feature fusion module MsR-EFM is used to integrate the multi-scale context information between different layers in the network encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is used to integrate the multi-scale context information between different layers in the network decoding stage;

[0080] Four branches are set in the multi-scale feature extraction module M2F2, each of which is provided with a convolution layer, a multi-scale feature extraction unit and a feature splitting and splicing unit. Each branch convolves the input feature map and extracts multi-scale features to obtain the corresponding feature map, and then the feature splicing unit splices the feature map to obtain the multi-scale feature map to be extracted; the multi-scale feature maps of different layers in the encoding stage are used as the input of the multi-scale residual coding feature fusion module MsR-EFM;

[0081] The multi-scale residual coding feature fusion module MsR-EFM is provided with a convolution unit, a pooling unit and a weight allocation unit. The convolution unit is used to change the number of channels of the multi-scale feature map, the pooling unit is used to reduce the size of the feature map, and the weight allocation unit is used to assign weights to feature images of different layers respectively.

[0082] The multi-scale residual decoding feature fusion module MsR-DFM is equipped with upsampling units, convolution units, pooling units and multi-layer perceptrons. The upsampling unit is used to unify the feature maps of different layers in the decoding stage to the size required for the final output. The convolution unit is used to capture the local information of the feature map and change the number of channels of the feature map. The pooling unit and the multi-layer perceptron are used to refine the spatial information of the feature map to obtain the spatial attention of the feature map, perform maximum pooling, average pooling and soft pooling on the concatenated feature F to obtain its spatial attention feature, and then perform channel attention operation on it to obtain the feature after channel attention, and then perform sigmoid activation operation on the feature to obtain the final segmentation result.

[0083] In addition, the present invention can also provide a computer device, including a processor and a memory, the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and when the processor executes part or all of the computer executable program, it can implement the image segmentation method based on the multi-scale residual codec network described in the present invention.

[0084] On the other hand, the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the image segmentation method based on the multi-scale residual codec network described in the present invention can be implemented.

[0085] The computer device may be a laptop computer, a desktop computer or a workstation.

[0086] The processor may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or an off-the-shelf field programmable gate array (FPGA).

[0087] The memory of the present invention may be an internal storage unit of a laptop computer, a desktop computer or a workstation, such as a memory or a hard disk; or an external storage unit, such as a mobile hard disk or a flash memory card.

[0088] Computer-readable storage media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may include: read-only memory (ROM), random access memory (RAM), solid-state drive (SSD) or optical disk, etc. Among them, random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

Claims

1. An image segmentation method based on a multi-scale residual codec network, characterized in that: Preprocessing of skin lesion images; Performing data enhancement on the preprocessed image to obtain more image data, namely, input feature image; Based on the multi-scale residual codec network, feature extraction, encoding feature fusion and decoding feature fusion are performed on the input feature image; wherein the multi-scale residual codec network is based on CS2-Net, and a multi-scale feature extraction module M2F2 is used to replace the convolution module in CS2-Net, and a multi-scale residual encoding feature fusion module MsR-EFM is used to integrate the multi-scale context information between different layers in the network encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is used to integrate the multi-scale context information between different layers in the network decoding stage; Four branches are set in the multi-scale feature extraction module M2F2, each of which is provided with a convolution layer, a multi-scale feature extraction unit and a feature splitting and splicing unit. Each branch convolves the input feature map and extracts multi-scale features to obtain the corresponding feature map, and then the feature splicing unit splices the feature map to obtain the multi-scale feature map to be extracted; the multi-scale feature maps of different layers in the encoding stage are used as the input of the multi-scale residual coding feature fusion module MsR-EFM; The multi-scale residual coding feature fusion module MsR-EFM is provided with a convolution unit, a pooling unit and a weight allocation unit. The convolution unit is used to change the number of channels of the multi-scale feature map, the pooling unit is used to reduce the size of the feature map, and the weight allocation unit is used to assign weights to feature images of different layers respectively. The multi-scale residual decoding feature fusion module MsR-DFM is equipped with upsampling units, convolution units, pooling units and multi-layer perceptrons. The upsampling unit is used to unify the feature maps of different layers in the decoding stage to the size required for the final output. The convolution unit is used to capture the local information of the feature map and change the number of channels of the feature map. The pooling unit and the multi-layer perceptron are used to refine the spatial information of the feature map to obtain the spatial attention of the feature map, perform maximum pooling, average pooling and soft pooling on the concatenated feature F to obtain its spatial attention feature, and then perform channel attention operation on it to obtain the feature after channel attention, and then perform sigmoid activation operation on the feature to obtain the final segmentation result.

2. The image segmentation method based on a multi-scale residual codec network according to claim 1, characterized in that: The multi-scale feature extraction module M2F2 changes the number of channels of the input feature map to the number of channels required for the output feature map through convolution, and the obtained feature map is recorded as F1; the number of channels of the input feature map is reduced to 1 / 4 through convolution, and then multi-scale features are extracted, and the obtained feature map is recorded as F2; ​​the number of channels of the input feature map is reduced to 1 / 8 through convolution, and then multi-scale features are extracted, and the obtained feature map is recorded as F3; the number of channels of the input feature map is reduced to 1 / 8 through the number of convolution layers, and then multi-scale features are extracted after convolution, and the obtained feature map is recorded as F4; the feature maps F2, F3 and F4 are spliced ​​together and recorded as F 234 , the feature map F 234 The number of channels becomes the same as the output feature, denoted as F 123 ; The final output feature map of M2F2 is F1+F 123 .

3. The image segmentation method based on a multi-scale residual codec network according to claim 1, characterized in that: In the multi-scale residual coding feature fusion module MsR-EFM, the feature images of different layers in the network coding stage are first resampled to the same size through convolution units and pooling units to obtain the resampled feature images. At the same time, weights are assigned to each layer respectively to obtain the effectively fused multi-scale coding features.

4. The image segmentation method based on a multi-scale residual codec network according to claim 1, characterized in that: In the multi-scale residual decoding feature fusion module MsR-DFM, the feature images of different layers are first resampled to the same size through bilinear difference upsampling, and then convolution processing is performed. The feature maps obtained after convolution processing are spliced ​​along the convolution channel dimension to obtain the spliced ​​feature F.

5. The image segmentation method based on a multi-scale residual codec network according to claim 1, characterized in that: When preprocessing the skin lesion image, the acquired original skin lesion image is processed to obtain an image with a preset pixel size as the input of the neural network.

6. The image segmentation method based on a multi-scale residual codec network according to claim 1, characterized in that: During data augmentation, the preprocessed images are randomly rotated by -20 degrees to 20 degrees, horizontally flipped, vertically flipped, and randomly cropped to obtain more training data.

7. A skin lesion image segmentation system based on a multi-scale residual codec network, characterized in that: Including image data processing module and feature encoding and decoding network module; The image data processing module is used to preprocess the image and perform data enhancement to obtain more image data; The feature encoding and decoding network module performs feature extraction, encoding feature fusion and decoding feature fusion on the more image data based on the multi-scale residual encoding and decoding network; wherein the multi-scale residual encoding and decoding network is based on CS2-Net, and a multi-scale feature extraction module M2F2 is used to replace the convolution module in CS2-Net, and a multi-scale residual encoding feature fusion module MsR-EFM is used to integrate the multi-scale context information between different layers in the network encoding stage, and a multi-scale residual decoding feature fusion module MsR-DFM is used to integrate the multi-scale context information between different layers in the network decoding stage; Four branches are set in the multi-scale feature extraction module M2F2, each of which is provided with a convolution layer, a multi-scale feature extraction unit and a feature splitting and splicing unit. After each branch performs convolution on more image data and extracts multi-scale features, the corresponding feature maps are obtained respectively, and then the feature splicing unit splices the feature maps to obtain the multi-scale feature maps to be extracted; the multi-scale feature maps of different layers in the encoding stage are used as the input of the multi-scale residual coding feature fusion module MsR-EFM; The multi-scale residual coding feature fusion module MsR-EFM is provided with a convolution unit, a pooling unit and a weight allocation unit. The convolution unit is used to change the number of channels of the multi-scale feature map, the pooling unit is used to reduce the size of the feature map, and the weight allocation unit is used to assign weights to feature images of different layers respectively. The multi-scale residual decoding feature fusion module MsR-DFM is equipped with upsampling units, convolution units, pooling units and multi-layer perceptrons. The upsampling unit is used to unify the feature maps of different layers in the decoding stage to the size required for the final output. The convolution unit is used to capture the local information of the feature map and change the number of channels of the feature map. The pooling unit and the multi-layer perceptron are used to refine the spatial information of the feature map to obtain the spatial attention of the feature map, perform maximum pooling, average pooling and soft pooling on the concatenated feature F to obtain its spatial attention feature, and then perform channel attention operation on it to obtain the feature after channel attention, and then perform sigmoid activation operation on the feature to obtain the final segmentation result.

8. A computer device, characterized in that: It includes a processor and a memory, the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and when the processor executes the computer executable program, it can implement the image segmentation method based on the multi-scale residual codec network as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored in a computer-readable storage medium. When the computer program is executed by a processor, the image segmentation method based on a multi-scale residual codec network as described in any one of claims 1 to 6 can be implemented.

Images