Skin lesion semantic segmentation two-channel deep learning model
The dual-channel deep learning model for skin lesion segmentation addresses the limitations of single-network architectures by integrating SPResNet and PVT structures with stripe pooling and feature fusion, improving segmentation accuracy through enhanced feature extraction and attention mechanisms.
Patent Information
- Application Number
- CN202410414127.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-08
- Publication Date
- 2024-12-27
AI Technical Summary
Existing deep learning models for semantic segmentation of skin lesions suffer from limited performance due to the use of single network architectures, which result in information loss and ineffective restoration of key pixels, particularly in CNN structures, and miss detailed features in Transformer applications.
A dual-channel deep learning model combining SPResNet and PVT structures, utilizing stripe pooling and feature pyramid mosaicking to retain information, and an aggregate attention module for feature fusion, enhances the network's performance by integrating local and global feature extraction.
The dual-channel model improves the accuracy of skin lesion segmentation by effectively combining CNN and Transformer structures, retaining feature information, and reducing calculation overhead, thereby enhancing the model's learning effect.
Smart Images
Figure CN119206720A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of semantic segmentation and medical image processing, and in particular to a dual-channel deep learning model for semantic segmentation of skin lesions. Background Art
[0002] Melanoma is a malignant skin tumor caused by abnormal growth of melanocytes located in the uppermost layer of the epidermis, which in turn causes cancer. It has a low incidence rate but a high mortality rate. Early diagnosis of melanoma is crucial to the treatment of patients. With the help of non-invasive, effective, convenient and fast dermoscopic imaging, the diagnostic accuracy of clinical dermatologists for melanoma tumors has been greatly improved. Due to factors such as clinical diagnostic experience, working environment and labor intensity, the diagnostic results show that the deviations of the same doctor at different times or different doctors at the same time are large, resulting in high missed diagnosis and misdiagnosis rates. In today's booming artificial intelligence technology, the introduction of deep learning technology to automatically identify and segment target areas can greatly reduce the workload of doctors, focusing on diagnosing patients' conditions and formulating more suitable medical plans for patients. Computer-assisted melanoma tumor diagnosis can effectively alleviate the workload of doctors, reduce the differences between or within diagnosers, and help improve medical standards. The dual-channel deep learning model for semantic segmentation of skin lesions is a key and basic step in the computer-aided diagnosis system.
[0003] The prior art has the following deficiencies:
[0004] Most of the existing deep learning models use a single type of model as the network architecture, with limited performance improvement and limitations inherited by a single network. For example, a single convolutional neural network (CNN) structure or a Transformer structure. The CNN structure can extract local features in a fine manner, has the advantages of local perception and parameter sharing, and can gradually increase the receptive field of the deep network. However, information loss occurs after operations such as pooling and convolution, and the receptive field is limited, so the information obtained has no correlation with the pixel information in the distance. Due to the shortcomings of the CNN structure, the encoder-decoder structure is often used in segmentation tasks to supplement the lost information. Even so, key pixels are often not effectively restored. The Transformer structure was originally used in the field of natural language processing. After being applied to the field of computer vision, it has shown excellent performance. The attention mechanism can associate and extract global features and increase the receptive field of the network. Attention focuses on inputs of different sizes. When focusing on global features, some detailed features may be missed.
[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention
[0006] The purpose of the present invention is to provide a dual-channel deep learning model for semantic segmentation of skin lesions. The dual-channel structure is adopted, one of which uses the SPResNet structure and the other uses the PVT structure, taking into account both detail extraction and global feature association, combining the advantages of CNN and Transformer structures, and using a complementary approach to increase the actual performance of the entire network. SPResNet adds a stripe pooling scheme to the structure with ResNet-50 as the backbone network. Compared with the global average pooling and maximum and minimum pooling, which will cause information loss, the stripe pooling adopts a structure similar to the attention mechanism to obtain more information while reducing the amount of calculation. The pyramid structure performs dimension mosaicking on the feature images of each layer of the two pathways, retains the feature information of each structure, and then realizes the complementary advantages of different structures through convolution mapping. During the training process, the feature images of each layer are deeply supervised to ensure the learning performance of the overall network from the bottom up. The aggregate attention module (AAM) with feature fusion is used for the top-level features to focus on learning key information, and the results are returned to the upsampling pathway, which is fused with the dual-pathway image of the top layer for pixel-level addition to enhance the model learning effect, so as to solve the structural defects of the existing deep learning model proposed in the above-mentioned background technology, improve the accuracy of skin lesion segmentation, and solve the problems in the above-mentioned background technology.
[0007] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a dual-channel deep learning model for semantic segmentation of skin lesions, comprising a preprocessing module, a feature extraction module, a stripe pooling module, a PVT structure module, a DPNet network construction module, a CB module, an internal learning module and a lesion segmentation module;
[0008] The preprocessing module preprocesses the skin lesion data set, and the processing adopts a random preprocessing method;
[0009] Feature extraction module, using ResNet50 as the backbone network of SPResNet for feature extraction;
[0010] The PVT structure module uses the feature pyramid structure as the main feature extraction structure, and uses PVTv2 as the backbone network, using the weights pre-trained on ImageNet;
[0011] The DPNet network building module uses SPResNet and PVT to form two upper and lower feature extraction paths, adjust and optimize network parameters, and deploy DPNet;
[0012] Internal learning module, which loads the original image into Dataloader, sets hyperparameters, and sends it to the network for internal learning;
[0013] The lesion segmentation module uses the trained deep learning model to perform skin lesion segmentation on the test set.
[0014] Preferably, the steps of the SPResNet feature extraction pathway are as follows:
[0015] First, ResNet50 is used as the backbone network of SPResNet, and the input skin lesion image is passed through the feature extraction layer to obtain a series of feature images;
[0016] Then, the feature image is divided into four levels, and the feature map between each level is reduced by 1 / 2. The stripe pooling module is introduced at the end of the residual block of each level to obtain more image information. The stripe pooling operation uses a strip-shaped pooling area instead of the original N×N area.
[0017] Then, the feature images of each layer are dimensionally mosaicked through the feature pyramid structure to retain the feature information of each level;
[0018] Finally, convolutional mapping is used to complement the advantages of feature images at different levels, and the feature images at each level are combined to form a complete SPResNet feature extraction pathway.
[0019] Preferably, the stripe pooling module consists of two pooling parts, namely vertical stripe pooling and horizontal stripe pooling, which are added to obtain long-distance image information. The stripe pooling module formula is as follows: The formula Represents the pooling operation in the horizontal direction, and performs an average pooling operation on the corresponding rows. The h in the equation represents the horizontal direction, i represents the position of the corresponding row, W represents the total number of rows, ∑ 0≤j≤W x i , j Represents the accumulation of the current row of elements, the formula The vertical pooling operation performs an average pooling operation on the columns. The v in the equation represents the vertical direction, j represents the position of the corresponding column, H represents the total number of columns, ∑ 0≤i≤H x i,j Represents the accumulation of the current number of elements in the column.
[0020] Preferably, the steps of the PVT feature extraction pathway are as follows:
[0021] First, the original skin lesion image is embedded in image blocks and position encoded to convert the image into the image block sequence required by the Transformer encoder;
[0022] Then, the image block sequence is processed through a multi-layer Transformer encoder. After each stage, the feature image is downsampled to reduce the image to 1 / 2 of the original. At each stage, the downsampled feature map is embedded with new image blocks and position information to ensure that the new feature image is segmented and positionally encoded.
[0023] A multi-head self-attention mechanism module is used inside the Transformer to perform attention calculation on each input image block to obtain the global feature connection of the entire image;
[0024] Finally, after being processed by a multi-layer Transformer encoder, a high-dimensional feature representation is obtained after passing through the PVT feature extraction pathway.
[0025] Preferably, both SPResNet and PVT networks have four-layer structures, and the output images of each layer have the same size but different dimensions. The output feature maps of the two networks are reduced to the same latitude and then dimensionalized and sent as a whole to the subsequent up-sampled image restoration. After passing through the CB module, the input feature image is reduced in dimensionality to facilitate pixel-scale addition with the feature map of the lower layer. The CB module embeds group normalization, SiLU activation function and residual link to reduce the information loss problem caused by dimensionality reduction upsampling when the information merged by the two networks in the concatenate stage.
[0026] Preferably, it also includes an aggregate attention module with feature fusion;
[0027] The aggregation attention module,in the DPNet network building module, fuses the high-dimensional semantic information of the,top layers of the two networks, and performs the attention mechanism after fusing the,features extracted from each network to obtain the attention points of the two,structures.
[0028] Preferably, the aggregation attention module is written as follows: Here y represents the top-level input of SPResNet. Represents the top-level input of PVT, Comcate represents the fusion of two feature images according to the dimension, Q, K, and V represent the matrices corresponding to the Queery, Key, and Value of the image after the fusion feature, respectively. Represents the reduction factor.
[0029] Preferably, the steps of loading the original image into Dataloader, setting the hyperparameters, and sending it to the network for learning are as follows:
[0030] Data loading:
[0031] First, the original skin lesion dataset is preprocessed to ensure that the format and size of the image data meet the input requirements of the network;
[0032] Next, the preprocessed image dataset is divided into training set, validation set and test set;
[0033] Then, use the Dataset and DataLoader modules in PyTorch to encapsulate the training set and validation set into data loaders for subsequent batch training and validation;
[0034] Set hyperparameters:
[0035] Set hyperparameters and adjust and optimize according to the characteristics of specific tasks and datasets;
[0036] Network initialization:
[0037] Initialize the DPNet network, including two feature extraction pathways of SPResNet and PVT and an integration module;
[0038] Define the loss function and optimizer:
[0039] Define the loss function used to calculate the difference between the model prediction value and the true label, and choose the appropriate loss function based on the task requirements;
[0040] Initialize the optimizer and set the learning rate and other hyperparameters according to the actual situation;
[0041] Train the network:
[0042] The network is trained iteratively. Each iteration cycle includes the following steps:
[0043] Load a batch of training data from the data loader, including input images and corresponding labels;
[0044] Send the input image into the DPNet network for forward propagation to obtain the network output;
[0045] Calculate the loss between the network output and the true label;
[0046] Update network parameters based on back propagation of loss values;
[0047] Repeat the above steps until all training data are trained;
[0048] After each iteration, the network is evaluated using the validation set, and the loss value and performance indicators of the model on the validation set are calculated to monitor the training progress of the network and adjust the hyperparameters.
[0049] In the above technical solution, the technical effects and advantages provided by the present invention are:
[0050] The present invention adopts a dual-channel structure, one of which uses the SPResNet structure and the other uses the PVT structure, taking into account both detail extraction and global feature association, combining the advantages of CNN and Transformer structures, and using a complementary approach to increase the actual performance of the entire network.
[0051] In the present invention, SPResNet adds a stripe pooling scheme to the structure with ResNet-50 as the backbone network. Compared with the global average pooling and maximum and minimum pooling that will cause information loss, the stripe pooling adopts a structure similar to the attention mechanism to obtain more information while reducing the amount of calculation. The feature images of each layer of the two pathways are dimensionally mosaicked through the feature pyramid structure, and the feature information of each structure is retained. Then, the advantages of different structures are complementary through convolution mapping. During the training process, the feature images of each layer are deeply supervised to ensure the bottom-up learning performance of the overall network. The aggregate attention module (AAM) with feature fusion is used for the top-level features to focus on learning key information, and the results are returned to the upsampling pathway, which is fused with the dual-path image of the top layer for pixel-level addition to enhance the model learning effect, so as to solve the structural defects of the existing deep learning model proposed in the above background technology and improve the accuracy of skin lesion segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0053] Figure 1 This is a module schematic diagram of a dual-channel deep learning model for semantic segmentation of skin lesions in the present invention.
[0054] Figure 2 This is a DPNet structure diagram of a dual-channel deep learning model for semantic segmentation of skin lesions in the present invention.
[0055] Figure 3 This is a detailed diagram of the structure of the Aggregate Attention Module (AAM) of a dual-channel deep learning model for semantic segmentation of skin lesions in the present invention.
[0056] Figure 4 This is a diagram of the CB module structure of a dual-channel deep learning model for semantic segmentation of skin lesions in the present invention.
[0057] Figure 5 This is a training flow chart of a dual-channel deep learning model for semantic segmentation of skin lesions in the present invention.
[0058] Figure 6 This is a diagram of the skin lesion segmentation prediction effect of the present invention, where DPNet is the model proposed by the present invention. DETAILED DESCRIPTION
[0059] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.
[0060] The present invention provides Figure 1-6 A dual-channel deep learning model for semantic segmentation of skin lesions shown in the figure includes a preprocessing module, a feature extraction module, a stripe pooling module, a PVT structure module, a DPNet network building module, a CB module, an aggregation attention module, an internal learning module, and a lesion segmentation module;
[0061] The preprocessing module preprocesses the skin lesion data set, and the processing adopts a random preprocessing method;
[0062] Among them, random preprocessing methods include affine transformation of data and color contrast enhancement, which can increase the diversity of data and alleviate the overfitting problem caused by limited data volume;
[0063] Affine transformation of data refers to changing the geometric shape and spatial structure of the original image through operations such as translation, rotation, scaling, and shearing to increase the diversity of the data. Specifically, translation refers to moving the pixels of the image in the horizontal and vertical directions; rotation refers to rotating the image around its center point by a certain angle; scaling refers to adjusting the size of the image proportionally; and shearing refers to cropping a part of the image in the horizontal and vertical directions. Through these affine transformation operations, images with different angles, sizes, and positions can be generated, thereby expanding the data set and increasing the generalization ability of the model.
[0064] Color contrast enhancement refers to changing the color distribution and brightness of an image by adjusting parameters such as image brightness, contrast, and saturation to enhance the contrast and clarity of the image. Specifically, brightness adjustment is to change the overall brightness of the image; contrast adjustment is to increase or decrease the brightness difference between pixels in the image; saturation adjustment is to adjust the color saturation of the image to make it more vivid or soft. Through color contrast enhancement operations, the details of the image can be made more prominent, thereby improving the model's ability to extract image features.
[0065] Feature extraction module, using ResNet50 as the backbone network of SPResNet for feature extraction;
[0066] The steps of the SPResNet feature extraction pipeline are as follows:
[0067] First, ResNet50 is used as the backbone network of SPResNet, and the input skin lesion image is passed through the feature extraction layer to obtain a series of feature images.
[0068] Then, the feature image is divided into four levels, and the feature map between each level is reduced by 1 / 2. The stripe pooling module is introduced at the end of the residual block of each level to obtain more image information. The stripe pooling operation is similar to the self-attention mechanism and non-local network, but the stripe pooling area is used instead of the original N×N area to effectively capture long-distance dependencies and prevent interference from irrelevant areas.
[0069] Then, the feature images of each layer are dimensionally mosaicked through the feature pyramid structure to retain the feature information of each level.
[0070] Finally, convolutional mapping is used to complement the advantages of feature images at different levels, and the feature images at each level are combined to form a complete SPResNet feature extraction pathway.
[0071] The reason why SPResNet is used as the feature extraction channel is that the pyramid structure can obtain different feature information by using a bottom-up structure. The feature image in the low-dimensional space contains a lot of spatial information and object location information, which helps to determine the size, shape and specific location of the segmented object in the image. In the commonly used pyramid structure, the low-dimensional feature map and the upsampled high-dimensional feature map are added at the pixel level to ensure that the detailed features can be effectively restored after layer-by-layer extraction. After multiple feature extractions, the information in high dimensions lacks the perception of object details, but is endowed with sensitivity to object categories. The information contained in the pixel is several times that of the low-dimensional space. The specific category of the object can be judged based on the information contained in the pixel, and irrelevant information can be excluded. In the image segmentation process, the proportion of the segmented area in the image can be effectively judged.
[0072] Traditional global average pooling and maximum and minimum pooling operations can only rely on local windows to obtain image information within the window. This operation can extract information within the window but does not take into account the relationship with neighboring pixels. The operation logic of stripe pooling is similar to the self-attention mechanism and non-local network. The difference is that the strip-shaped pooling area is used instead of the original N×N area, that is, the pooling kernel is replaced with N×1 or 1×N, which can effectively allow the backbone network to effectively capture long-distance dependencies. In addition, since the pooling kernel is narrow and long, it is conducive to capturing contextual relationships and preventing interference from irrelevant areas.
[0073] The stripe pooling module consists of two pooling parts, namely vertical stripe pooling and horizontal stripe pooling, which are added together to obtain long-distance image information. The stripe pooling module formula is as follows: The formula Represents the pooling operation in the horizontal direction, and performs an average pooling operation on the corresponding rows. The h in the equation represents the horizontal direction, i represents the position of the corresponding row, W represents the total number of rows, ∑ 0≤j≤W x i , j Represents the accumulation of the current row's elements. Formula The vertical pooling operation performs an average pooling operation on the columns. The v in the equation represents the vertical direction, j represents the position of the corresponding column, H represents the total number of columns, ∑ 0≤i≤H x i,j Represents the accumulation of the current column number of elements. Pooling operations in different directions effectively avoids interference from irrelevant information.
[0074] The PVT structure module uses the feature pyramid structure as the main feature extraction structure, and uses PVTv2 as the backbone network, using the weights pre-trained on ImageNet;
[0075] The steps of the PVT feature extraction pipeline are as follows:
[0076] First, the original skin lesion image is patch embedded (Patch Embed) and position encoded (PositionEncode) to convert the image into the image patch sequence required by the Transformer encoder.
[0077] Then, the image block sequence is processed through a multi-layer Transformer encoder. After each stage, the feature image is downsampled to reduce the image to 1 / 2 of the original size. At each stage, the downsampled feature map needs to be embedded with new image blocks and position information to ensure that the new feature image is segmented and positionally encoded.
[0078] A multi-head self-attention mechanism module is used inside the Transformer to perform attention calculation on each input image block to obtain the global feature connection of the entire image.
[0079] Finally, after being processed by a multi-layer Transformer encoder, a high-dimensional feature representation is obtained after passing through the PVT feature extraction pathway.
[0080] Compared with other Transformer structures that require the original image to be extracted by CNN to form a feature image, PVT can directly use the original image as input and convert the image into the image block required by the Transformer encoder by performing patch embedding and position encoding on the original image. The overall structure is divided into four parts, each of which has two units, namely, patch embedding and position encoding. Patch embedding divides an entire original image or an extracted feature image into blocks of the same size. Position encoding adds a position code to each image block to indicate the position of the image block in the entire image, which facilitates the network to learn the connection between image blocks. Position encoding uses a learnable position code. By initializing learnable parameters, the position of the image block is learned during the network training process. The encoder part of the Transformer serializes the input image blocks through the standard Transformer encoder structure and performs linear mapping. After the first stage, the original image is scaled to 1 / 8 of the original image, and then reduced by 1 / 2 in each subsequent stage, and reduced to 1 / 32 of the original image in the last layer.
[0081] Since the PVT network downsamples the feature image after each part and scales the image to 1 / 2 of the original, the image block embedding information and position encoding information learned in the previous part are invalid for the new image. Therefore, new image block embedding and position information are required for the downsampled feature map at each stage to ensure the new feature image is divided into blocks and position encoded. The PVT Transformer uses a multi-head self-attention mechanism module to perform attention calculations on each input image block, calculate the correlation between different image blocks, and then obtain the global feature connection of the entire image.
[0082] The DPNet network building module uses SPResNet and PVT to form two upper and lower feature extraction paths, adjust and optimize network parameters, and deploy DPNet;
[0083] SPResNet channel is based on CNN, which enables the network to extract local features. This part can well preserve important information such as the details of the segmented area. PVT can extract features from a global perspective and realize the correlation of global features, which can better learn the overall contour of the object and the position information of the segmented area in the image. In view of the different attention paid by the Transformer structure to objects of different sizes in images of different sizes, multi-scale image training is used, so that objects of different sizes on an image can have a good learning effect. Both networks are four-layer structures. The output image of each layer has the same size but different dimensions. The output feature maps of the two networks are reduced to the same latitude and then spliced. The whole is sent to the subsequent up-sampled image restoration. After the CB module, the input feature image is reduced in dimension so that it can be added to the pixel scale with the feature map of the lower layer. The CB module is embedded with group normalization, SiLU activation function and residual connections, which can reduce the information loss problem caused by the dimensionality reduction upsampling when the information merged by the two backbone networks in the concatenate stage;
[0084] The SiLU (Sigmoid Linear Unit) activation function is a nonlinear activation function, and its mathematical expression is f(x)=x*sigmoid(x), where sigmoid(x) is an S-type function. The SiLU activation function compresses the input value through an S-type function and then multiplies it by the input value, thereby retaining the input feature information and introducing a nonlinear transformation. Compared with other commonly used activation functions such as ReLU, SiLU maintains more smoothness during the activation process, which helps alleviate the gradient vanishing problem and has better performance in deep networks.
[0085] Residual Connections is a network structure design method, which is usually used in deep residual networks. The idea of residual connections is to add cross-layer connections between certain layers of the network, directly adding the output of the previous layer to the input of the subsequent layer, so as to pass the feature information of the previous layer to the subsequent layer without loss. This design helps to alleviate the gradient vanishing problem, simplifies the network training process, and can effectively improve the learning ability and convergence speed of the network. Using residual connections in the CB module helps to retain more feature information, so that the network can better learn the details and semantic information of the input feature map.
[0086] Adjust and optimize the network parameters to ensure that both SPResNet and PVT pathways can effectively extract the feature information of skin lesion images.
[0087] The findings from the experiment:
[0088] High-dimensional semantic information cannot directly obtain the desired results for segmentation tasks through prediction output, because the semantic information of high-dimensional feature images is extremely rich, and the expansion of the receptive field also compresses the image into a very small range, and the position of the pixel block in the original image can only be judged based on the relative position. However, the information brought by the high-dimensional feature image after continuous fusion of the low-dimensional feature image is crucial for the judgment of the segmented area. Therefore, the present invention designs an aggregate attention module (AAM) with feature fusion, which fuses the high-dimensional semantic information of the highest layers of the two networks in the DPNet network construction module, and performs an attention mechanism after the features extracted from each network are fused to obtain two attention points with different structures. The aggregate attention module (AAM) can be written as the following formula: Here y represents the top-level input of SPResNet. Represents the top-level input of PVT, Concate represents the fusion of two feature images according to the dimension, Q, K, and V represent the matrices corresponding to the Queery, Key, and Value of the image after the fusion feature, respectively. Represents the reduction factor.
[0089] Internal learning module, which loads the original image into Dataloader, sets hyperparameters, and sends it to the network for internal learning;
[0090] The steps to load the original image into Dataloader, set the hyperparameters, and send it to the network for learning are as follows:
[0091] Data loading:
[0092] First, the original skin lesion dataset is preprocessed, including reading image files, normalizing, resizing, and cropping the images to ensure that the format and size of the image data meet the input requirements of the network.
[0093] Next, the preprocessed image dataset is divided into training set, validation set and test set, usually in a certain ratio.
[0094] Then, use the Dataset and DataLoader modules in PyTorch to encapsulate the training set and validation set into data loaders for subsequent batch training and validation.
[0095] Set hyperparameters:
[0096] Set hyperparameters, including learning rate, optimizer type, loss function, number of training rounds, batch size, etc. The selection of these hyperparameters is crucial for network training and convergence, and needs to be adjusted and optimized according to the characteristics of specific tasks and datasets.
[0097] Define the learning rate decay strategy and other hyperparameters that may be required during training.
[0098] Network initialization:
[0099] Initialize the DPNet network, including the SPResNet and PVT feature extraction paths and the integration module. You can call the corresponding network structure and load the pre-trained weights as needed.
[0100] Define the loss function and optimizer:
[0101] Define the loss function used to calculate the difference between the model prediction value and the true label. Commonly used loss functions include cross entropy loss function, Dice loss function, etc. Select the appropriate loss function according to the task requirements.
[0102] Initialize the optimizer. Common optimizers include SGD, Adam, etc. Set the learning rate and other hyperparameters according to actual conditions.
[0103] Train the network:
[0104] The network is trained iteratively, and each iteration cycle includes the following steps: load a batch of training data from the data loader, including input images and corresponding labels.
[0105] The input image is sent into the DPNet network for forward propagation to obtain the network output.
[0106] Calculate the loss between the network output and the true label.
[0107] Back propagates according to the loss value and updates the network parameters.
[0108] Repeat the above steps until all training data are trained.
[0109] After each iteration, the network is evaluated using the validation set, and the loss value and performance indicators of the model on the validation set are calculated to monitor the training progress of the network and adjust the hyperparameters.
[0110] The lesion segmentation module uses the trained deep learning model to perform skin lesion segmentation on the test set.
[0111] The present invention adopts a dual-channel structure, one of which uses the SPResNet structure and the other uses the PVT structure, taking into account both detail extraction and global feature association, combining the advantages of CNN and Transformer structures, and using a complementary approach to increase the actual performance of the entire network.
[0112] In the present invention, SPResNet adds a stripe pooling scheme to the structure with ResNet-50 as the backbone network. Compared with the global average pooling and maximum and minimum pooling that will cause information loss, the stripe pooling adopts a structure similar to the attention mechanism to obtain more information while reducing the amount of calculation. The feature images of each layer of the two pathways are dimensionally mosaicked through the feature pyramid structure, and the feature information of each structure is retained. Then, the advantages of different structures are complementary through convolution mapping. During the training process, the feature images of each layer are deeply supervised to ensure the bottom-up learning performance of the overall network. The aggregate attention module (AAM) with feature fusion is used for the top-level features to focus on learning key information, and the results are returned to the upsampling pathway, which is fused with the dual-path image of the top layer for pixel-level addition to enhance the model learning effect, so as to solve the structural defects of the existing deep learning model proposed in the above background technology and improve the accuracy of skin lesion segmentation.
[0113] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. A dual-channel deep learning model for semantic segmentation of skin lesions, characterized by: It includes preprocessing module, feature extraction module, stripe pooling module, PVT structure module, DPNet network construction module, CB module, internal learning module and lesion segmentation module; The preprocessing module preprocesses the skin lesion data set, and the processing adopts a random preprocessing method; Feature extraction module, using ResNet50 as the backbone network of SPResNet for feature extraction; The PVT structure module uses the feature pyramid structure as the main feature extraction structure, and uses PVTv2 as the backbone network, using the weights pre-trained on ImageNet; The DPNet network building module uses SPResNet and PVT to form two upper and lower feature extraction paths, adjust and optimize network parameters, and deploy DPNet; Internal learning module, which loads the original image into Dataloader, sets hyperparameters, and sends it to the network for internal learning; The lesion segmentation module uses the trained deep learning model to perform skin lesion segmentation on the test set.
2. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 1, characterized in that: The steps of the SPResNet feature extraction pipeline are as follows: First, ResNet50 is used as the backbone network of SPResNet, and the input skin lesion image is passed through the feature extraction layer to obtain a series of feature images; Then, the feature image is divided into four levels, and the feature map between each level is reduced by 1 / 2. The stripe pooling module is introduced at the end of the residual block of each level to obtain more image information. The stripe pooling operation uses a strip-shaped pooling area instead of the original N×N area. Then, the feature images of each layer are dimensionally mosaicked through the feature pyramid structure to retain the feature information of each level; Finally, convolutional mapping is used to complement the advantages of feature images at different levels, and the feature images at each level are combined to form a complete SPResNet feature extraction pathway.
3. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 2, characterized in that: The stripe pooling module consists of two pooling parts, namely vertical stripe pooling and horizontal stripe pooling. The two are added together to obtain long-distance image information. The stripe pooling module formula is as follows: The formula Represents the pooling operation in the horizontal direction, and performs an average pooling operation on the corresponding rows. The h in the equation represents the horizontal direction, i represents the position of the corresponding row, W represents the total number of rows, ∑ 0≤j≤W x i , j Represents the accumulation of the current row of elements, the formula The vertical pooling operation performs an average pooling operation on the columns. The v in the equation represents the vertical direction, j represents the position of the corresponding column, H represents the total number of columns, ∑ 0≤i≤H x i,j Represents the accumulation of the current number of elements in the column.
4. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 1, characterized in that: The steps of the PVT feature extraction pipeline are as follows: First, the original skin lesion image is embedded in image blocks and position encoded to convert the image into the image block sequence required by the Transformer encoder; Then, the image block sequence is processed through a multi-layer Transformer encoder. After each stage, the feature image is downsampled to reduce the image to 1 / 2 of the original. At each stage, the downsampled feature map is embedded with new image blocks and position information to ensure that the new feature image is segmented and positionally encoded. A multi-head self-attention mechanism module is used inside the Transformer to perform attention calculation on each input image block to obtain the global feature connection of the entire image; Finally, after being processed by a multi-layer Transformer encoder, a high-dimensional feature representation is obtained after passing through the PVT feature extraction pathway.
5. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 1, characterized in that: Both SPResNet and PVT networks have four-layer structures. The output images of each layer are of the same size but different dimensions. The output feature maps of the two networks are reduced to the same dimension and then concatenated. The whole is sent to the subsequent up-sampled image restoration. After passing through the CB module, the input feature image is reduced in dimension to facilitate pixel-scale addition with the feature map of the lower layer. Group normalization, SiLU activation function and residual link are embedded in the CB module to reduce the information loss problem caused by dimensionality reduction upsampling when the information merged by the two networks in the concatenate stage.
6. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 1, characterized in that: It also includes an aggregate attention module with feature fusion; The aggregation attention module,in the DPNet network building module, fuses the high-dimensional semantic information of the,top layers of the two networks, and performs the attention mechanism after fusing the,features extracted from each network to obtain the attention points of the two,structures.
7. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 6, characterized in that: The aggregation attention module is written as follows: Here y represents the top-level input of SPResNet. Represents the top-level input of PVT, Concate represents the fusion of two feature images according to the dimension, Q, K, and V represent the matrices corresponding to the Queery, Key, and Value of the image after the fusion feature, respectively. Represents the reduction factor.
8. A dual-channel deep learning model for semantic segmentation of skin lesions according to claim 1, characterized in that: The steps to load the original image into Dataloader, set the hyperparameters, and send it to the network for learning are as follows: Data loading: First, the original skin lesion dataset is preprocessed to ensure that the format and size of the image data meet the input requirements of the network; Next, the preprocessed image dataset is divided into training set, validation set and test set; Then, use the Dataset and DataLoader modules in PyTorch to encapsulate the training set and validation set into data loaders for subsequent batch training and validation; Set hyperparameters: Set hyperparameters and adjust and optimize according to the characteristics of specific tasks and datasets; Network initialization: Initialize the DPNet network, including two feature extraction pathways of SPResNet and PVT and an integration module; Define the loss function and optimizer: Define the loss function used to calculate the difference between the model prediction value and the true label, and choose the appropriate loss function based on the task requirements; Initialize the optimizer and set the learning rate and other hyperparameters according to the actual situation; Train the network: The network is trained iteratively. Each iteration cycle includes the following steps: Load a batch of training data from the data loader, including input images and corresponding labels; Send the input image into the DPNet network for forward propagation to obtain the network output; Calculate the loss between the network output and the true label; Update network parameters based on back propagation of loss values; Repeat the above steps until all training data are trained; After each iteration, the network is evaluated using the validation set, and the loss value and performance indicators of the model on the validation set are calculated to monitor the training progress of the network and adjust the hyperparameters.