Dermoscope image classification method and system based on neural network

Abstract

The invention relates to the field of image classification, in particular to a dermatoscope image classification method and system based on a neural network, and the method comprises the steps: obtaining a spatial enhancement feature according to a dermatoscope image; inputting the spatial enhancement features into a deep convolution unit to carry out local feature extraction of a single channel, obtaining a first local feature of each channel, and carrying out linear combination between the channels to obtain a first local feature map; inputting the first local feature map into a deep convolution unit to perform global feature extraction of a single channel, and obtaining a first global feature of each channel; carrying out linear combination between channels to obtain a first global feature map; inputting the spatial enhancement feature, the first local feature map and the first global feature map into a feature fusion module for feature fusion to obtain a fused feature map; and classifying the dermatoscope images according to the fused feature map. According to the dermatoscope image classification method and system, the model parameter quantity is remarkably reduced, and meanwhile, the classification accuracy is improved to a certain degree.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a dermatoscope image classification method and system based on a neural network. Background Art

[0002] Traditional screening methods for skin diseases and skin cancer mainly include dermoscopy and pathological tissue examination; dermatoscope, also known as epidermal light transmission microscope, can provide clear images of fine structures such as living epidermis that are difficult to observe with the naked eye. Compared with the approximately 70% accuracy of naked eye diagnosis, the pathological consistency rate of skin lesions diagnosed by dermatoscope is as high as 95%. By analyzing dermoscopic images, the medical community can detect early signs of skin cancer, thereby improving the cure rate of patients. However, manual analysis of these images is not only time-consuming and laborious, but also requires doctors to have rich clinical experience, which can easily lead to misdiagnosis.

[0003] In recent years, with the rapid development of deep learning technology, neural network-based image classification methods have gradually become the mainstream means of dermoscopic image analysis. These technologies use deep learning models such as convolutional neural networks (CNNs) to extract features and classify dermoscopic images, which is expected to improve the accuracy and efficiency of classification. However, some deficiencies of existing technologies in application are still significant.

[0004] First, many neural network-based dermoscopic image classification methods have the problem of large number of algorithm parameters. Deep learning models are usually composed of multiple layers, and the number of parameters in each layer often reaches millions or even tens of millions. This not only requires a large amount of computing resources for the training and reasoning process of the model, but also limits its application to environments with high-performance hardware, making it difficult to popularize in medical institutions with limited resources.

[0005] Secondly, although some advanced network structures have been proven to be able to extract rich features, the classification accuracy is still not high in the actual application of dermoscopic image classification. This is because the lesion area in the dermoscopic image may be affected by uneven lighting, occlusion by hair or foreign objects, unclear skin texture, etc., resulting in the model being unable to effectively capture key features during feature extraction. In addition, the high similarity between lesioned skin and benign moles will also affect the accuracy of the algorithm. Existing classification models have limitations, and there are relatively few reliable data sets. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problems in the prior art of dermoscopic image classification that the algorithm has a large number of parameters and the classification accuracy is not high, and to provide a dermoscopic image classification method and system based on a neural network, which significantly reduces the number of model parameters and improves the classification accuracy to a certain extent, so as to better play a role in an environment with lower computing resources.

[0007] In a first aspect, in order to solve the above technical problems, the present invention provides a dermatoscopic image classification method based on a neural network, comprising:

[0008] Acquire a pre-trained image classification model; wherein the image classification model includes a coordinate attention processing module and a plurality of feature extraction modules sequentially connected along the forward propagation direction, each of the feature extraction modules includes a multi-head self-attention feature extraction submodule; each of the multi-head self-attention feature extraction submodule includes a local feature extraction block, a global feature extraction block and a feature fusion module; the local feature extraction block and the global feature extraction block both include a depth convolution unit and a point-by-point convolution unit;

[0009] Inputting the dermoscopic image to be classified into the coordinate attention processing module to obtain spatial enhancement features;

[0010] Inputting the spatial enhancement feature into the depth convolution unit of the local feature extraction block of the first feature extraction module to extract the local features of a single channel, and obtaining the first local features of each channel;

[0011] Inputting the first local features of each channel into the point-by-point convolution unit of the local feature extraction block of the first feature extraction module to perform linear combination between channels to obtain a first local feature map;

[0012] Inputting the first local feature map into the deep convolution unit of the global feature extraction block of the first feature extraction module to perform global feature extraction of a single channel to obtain a first global feature of each channel;

[0013] Inputting the first global features of each channel into the point-by-point convolution unit of the global feature extraction block of the first feature extraction module to perform linear combination between channels to obtain a first global feature map;

[0014] Inputting the spatial enhancement feature, the first local feature map and the first global feature map into the feature fusion module for feature fusion to obtain a fused feature map;

[0015] The fused feature map is used as the input of the next feature extraction module; and the dermoscopic image is classified according to the fused feature map output by the last feature extraction module.

[0016] In one embodiment of the present invention, the dermoscopic image to be classified is input into the coordinate attention processing module to obtain spatial enhancement features, including:

[0017] Performing preliminary processing on the dermoscopic image to obtain preliminary features;

[0018] Performing an average pooling operation in the horizontal direction on the preliminary features to extract X-direction features;

[0019] Performing an average pooling operation in the vertical direction on the preliminary features to extract features in the Y direction;

[0020] Performing stitching on the X-direction feature and the Y-direction feature to obtain a stitching feature;

[0021] Segmenting the splicing feature to obtain two parts of segmentation features;

[0022] Performing convolution operations on the two parts of the segmentation features respectively to obtain X-direction weight information and Y-direction weight information; wherein the weight information represents the importance of the corresponding position feature;

[0023] Multiplying the X-direction weight information by the preliminary feature to obtain an X-direction weighted feature; multiplying the Y-direction weight information by the preliminary feature to obtain a Y-direction weighted feature; taking the sum of the X-direction weighted feature and the Y-direction weighted feature as a weighted feature;

[0024] A residual block is introduced, and the spatial enhancement feature is obtained according to the weighted feature and the preliminary feature; wherein the spatial enhancement feature is obtained by calculating in the following manner:

[0025] ;

[0026] Among them, F is the spatial enhancement feature; F1 is the weighted feature; F2 is the preliminary feature; is the weight parameter.

[0027] In one embodiment of the present invention, the weight parameter Acquired through learning, which includes,

[0028] Initialize the weight parameters ; The weight parameter is defined as a vector of the same size as the preliminary feature, each element corresponding to a feature channel;

[0029] Add a network model to the residual block, and train the network model by adjusting hyperparameters to obtain the weight parameters;

[0030] Wherein, training the network model includes:

[0031] Define the hyperparameter space and set candidate values ​​for each hyperparameter that needs to be optimized;

[0032] Creating a model function, the model function is used to receive a set of the hyperparameters and return a model performance indicator;

[0033] Train and validate all hyperparameter combinations, and record the performance indicators under each set of hyperparameters;

[0034] Select the best set of hyperparameters based on the performance on the validation set;

[0035] The weight parameters are obtained according to the optimal set of hyperparameters.

[0036] In one embodiment of the present invention, the dermoscopic image is preliminarily processed to obtain preliminary features, including:

[0037] Performing a channel dimension increase operation on the dermoscopic image, and then normalizing and activating the channel dimension increase operation to obtain a channel dimension increase feature;

[0038] Performing a diffusion separable convolution operation on the channel dimension-raising feature to obtain a convolution feature;

[0039] Normalizing and activating the convolutional features to obtain the preliminary features;

[0040] After obtaining the spatial enhancement feature, the method further includes performing a channel dimensionality reduction operation on the spatial enhancement feature, and inputting the result of the channel dimensionality reduction operation into the feature extraction module.

[0041] In one embodiment of the present invention, the depth convolution unit of the local feature extraction block includes a 5*5 diffusion separable convolution layer and a 7*7 dilated diffusion separable convolution layer; and along the forward propagation direction, the 5*5 diffusion separable convolution layer and the 7*7 dilated diffusion separable convolution layer are arranged in sequence; wherein the dilation rate of the 7*7 dilated diffusion separable convolution layer is configured to be 3.

[0042] In one embodiment of the present invention, a multi-head self-attention mechanism is embedded in the global feature extraction block, and the global feature extraction block also has a point convolution block, which is used to simulate the interactive operation between multiple heads in the multi-head self-attention mechanism.

[0043] In one embodiment of the present invention, the process of performing global feature extraction based on the multi-head self-attention mechanism includes:

[0044] Reduce the dimension of the input feature map based on the diffusion separable convolution operation;

[0045] Map the dimensionally reduced feature map to query Q, key K, and value V respectively through linear transformation;

[0046] Get an attention score based on the query Q and key K;

[0047] Apply the Softmax function to normalize the attention score and obtain the attention weight;

[0048] Multiply the attention weight by the value V to obtain a weighted value;

[0049] Normalizing the weighted values, and performing global feature extraction according to the normalized results;

[0050] The self-attention of the multi-head self-attention mechanism is calculated according to the following formula:

[0051] ;

[0052] Where Q represents the query of the multi-head self-attention mechanism; K represents the key of the multi-head self-attention mechanism; V represents the value of the multi-head self-attention mechanism; d k represents the dimension of K; K T Represents the transpose of K; Conv() represents the diffusion separable convolution operation; Softmax() represents the Softmax function; IN() represents the implementation normalization.

[0053] In one embodiment of the present invention, the training process of the image classification model includes:

[0054] Get an image sample set;

[0055] Constructing an image classification loss function based on the output of the last feature extraction module;

[0056] Iteratively training the coordinate attention processing module and the plurality of feature extraction modules using the image samples in the image sample set until the value of the image classification loss function is minimized, thereby obtaining a trained image classification model;

[0057] Among them, the image classification loss function is:

[0058] ;

[0059] i represents the sample number; n represents the total number of samples; y i Represents the true value probability of the i-th sample; represents the probability that the i-th sample is evaluated as a positive sample; ln represents the natural logarithm function.

[0060] In one embodiment of the present invention, the image classification model also includes a coordinate attention downsampling module; wherein the output of the coordinate attention processing module is connected to the coordinate attention downsampling module; the output of the feature extraction module is connected to the coordinate attention downsampling module; the coordinate attention downsampling module is used to perform downsampling operations on the connected input features.

[0061] In a second aspect, in order to solve the above-mentioned technical problem, the present invention provides a dermoscopic image classification system based on a neural network, which is used to execute the dermoscopic image classification method based on a neural network.

[0062] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0063] In the neural network-based dermoscopic image classification method and system described in the present invention, both the local feature extraction block and the global feature extraction block decompose a complete convolution into deep convolution and point-by-point convolution. First, deep convolution is used to perform convolution operations only in a single channel to extract the features of each channel, and then point-by-point convolution is used to perform linear combinations of the features of each channel between channels. The convolution operation within a single channel greatly reduces the amount of calculation and the number of parameters compared to the inter-channel convolution operation of an ordinary convolutional network. In addition, the significant reduction in the number of parameters will theoretically weaken the accuracy of image classification, but this scheme adds a coordinate attention mechanism to the residual structure of the network, adds sensitivity to position information, and enhances information in specific areas, so that the model focuses more on key features and enhances the network's ability to integrate high-dimensional features, ultimately achieving the purpose of improving network classification performance. A multi-head self-attention mechanism is introduced to process local and global feature information, and by increasing the interaction between multi-head features and optimizing the computational complexity of the algorithm at the same time, the global dependencies in the feature map are captured, and the network's ability to extract global information is enhanced, thereby indirectly improving the efficiency and accuracy of model classification.

[0064] In summary, the neural network-based dermoscopic image classification method and system described in the present invention significantly reduced the model size to 0.83MB, and achieved a classification accuracy of 95.54% for the HAM10000 dataset, which is more suitable for deployment on embedded devices for clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0066] Figure 1 Flow chart of a dermoscopic image classification method based on a neural network in a preferred embodiment of the present invention;

[0067] Figure 2 It is the overall framework diagram of the image classification model in the preferred embodiment of the present invention;

[0068] Figure 3 It is a structural block diagram of the MV2 module in which the coordinate attention mechanism is added in the preferred embodiment of the present invention;

[0069] Figure 4 for Figure 3The structural block diagram of coordinate attention in the MV2 module with the coordinate attention mechanism added is shown;

[0070] Figure 5 It is a structural block diagram of the multi-head self-attention feature extraction submodule in the preferred embodiment of the present invention;

[0071] Figure 6 A flow chart of obtaining spatial enhancement features in a preferred embodiment of the present invention;

[0072] Figure 7 This is the test set accuracy graph;

[0073] Figure 8 is the test set loss graph;

[0074] Fig. 9 This is a diagram of the confusion matrix;

[0075] Description of the reference numerals in the specification: 100 - coordinate attention processing module;

[0076] 200-feature extraction module; 201-5*5 diffusion separable convolution layer; 202-7*7 dilated diffusion separable convolution layer; 210-local feature extraction block; 220-global feature extraction block; 230-feature fusion module;

[0077] 300-coordinate attention downsampling module. DETAILED DESCRIPTION

[0078] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0079] Embodiment 1

[0080] The embodiment of the present invention discloses a dermatoscopic image classification method based on a neural network, comprising:

[0081] Obtain a pre-trained image classification model; this step requires pre-training to obtain an image classification model, the image classification model comprising a coordinate attention processing module 100, a plurality of feature extraction modules 200 sequentially connected along the forward propagation direction, and a coordinate attention downsampling module 300;

[0082] Specifically, refer to Figure 2As shown, the image classification model includes a 3*3 convolution module, a coordinate attention processing module 100, a coordinate attention downsampling module 300, a coordinate attention processing module 100, a coordinate attention downsampling module 300, a feature extraction module 200, a coordinate attention downsampling module 300, a feature extraction module 200, a coordinate attention downsampling module 300, a feature extraction module 200, a coordinate attention downsampling module 300, a feature extraction module 200, a 1*1 convolution module and a global pooling layer, which are sequentially connected along the forward propagation direction. The dermoscopic image to be classified is processed by the above modules in sequence, and finally the classification result is output through the global pooling layer.

[0083] Among them, the coordinate attention processing module 100 is configured as an MV2 module with a coordinate attention mechanism, which is a variant of MobileNetV2 and has a coordinate attention mechanism (Coordinate Attention); the feature extraction module 200 is configured as an Efficient-MobileViT Block with a multi-head self-attention mechanism; the coordinate attention downsampling module 300 is configured as an MV2 module with a coordinate attention mechanism.

[0084] Reference Figure 3 As shown, the MV2 module with the coordinate attention mechanism is performed in sequence along the forward propagation direction:

[0085] The first 1×1 convolutional layer is used to adjust the number of channels;

[0086] Bn: normalization;

[0087] ReLU6: activation function, used to introduce nonlinearity;

[0088] MV2 block: MobileNetV2 block, which contains a depthwise separable convolution. The depthwise separable convolution can reduce the number of parameters when performing feature extraction.

[0089] Coordinate Attention (CA): Coordinate attention mechanism used to enhance the spatial information in feature maps;

[0090] Add: residual connection, adding the input feature map to the processed feature map;

[0091] The second 1×1 convolutional layer: used to adjust the number of channels.

[0092] In summary, the processing flow of the MV2 module with the coordinate attention mechanism includes: the input feature map passes through the first 1×1 convolution layer to adjust the number of channels, and the ReLU activation function introduces nonlinearity; it is processed by the MV2 block, including depth-wise separable convolution and ReLU activation function; it is processed by the coordinate attention mechanism (Coordinate Attention); it is then processed by the second 1×1 convolution layer and the ReLU activation function; and finally it is added to the input feature map through the residual connection (Add) to output the processed feature map.

[0093] Reference Figure 4 As shown, the coordinate attention is carried out in sequence along the positive propagation direction:

[0094] X Avg Pool: Performs horizontal average pooling on the input feature map;

[0095] Y Avg Pool: Performs vertical average pooling on the input feature map;

[0096] Concat: Concatenate the pooling results in the horizontal and vertical directions;

[0097] Split: Split the feature map into two parts;

[0098] Alpha: weighted processing of the segmented feature map;

[0099] Add: Add the weighted feature map to the original input feature map.

[0100] In summary, the processing flow of the coordinate attention mechanism includes: the input feature map undergoes horizontal and vertical average pooling operations to extract horizontal and vertical features; the horizontal and vertical features are spliced ​​to retain more spatial information; it is processed by residual blocks, including multiple convolutional layers and activation functions, for feature extraction and enhancement; the feature map is divided into two parts for weighted processing; and finally, it is added to the input feature map through the residual connection Add, and the processed feature map is output.

[0101] Reference Figure 5 As shown, the multi-head self-attention feature extraction submodules include a local feature extraction block 210, a global feature extraction block 220 and a feature fusion module 230; the local feature extraction block 210 includes multiple convolution units and a coordinate attention mechanism for extracting local features; wherein the convolution unit includes a deep convolution unit and a point-by-point convolution unit; the global feature extraction block 220 includes an efficient multi-head self-attention mechanism EMSA and a convolution unit for extracting global features; wherein the convolution unit includes a deep convolution unit and a point-by-point convolution unit; the feature fusion module 230 includes a splicing operation, a convolution layer and a coordinate attention mechanism for fusing local features with global features.

[0102] The dermoscopic image classification method of the embodiment of the present invention automatically classifies the image classification model obtained through the above training, referring to Figure 1 As shown, the classification method specifically includes the following steps:

[0103] S100, inputting the dermoscopic image to be classified into the coordinate attention processing module to obtain spatial enhancement features;

[0104] S200, inputting the spatial enhancement feature into the deep convolution unit of the local feature extraction block of the first feature extraction module to perform local feature extraction of a single channel to obtain a first local feature of each channel;

[0105] S300, inputting the first local features of each channel into the point-by-point convolution unit of the local feature extraction block of the first feature extraction module to perform linear combination between channels to obtain a first local feature map;

[0106] S400, inputting the first local feature map into the deep convolution unit of the global feature extraction block of the first feature extraction module to perform global feature extraction of a single channel to obtain a first global feature of each channel;

[0107] S500, inputting the first global features of each channel into the point-by-point convolution unit of the global feature extraction block of the first feature extraction module to perform linear combination between channels to obtain a first global feature map;

[0108] S600, inputting the spatial enhancement feature, the first local feature map and the first global feature map into the feature fusion module for feature fusion to obtain a fused feature map;

[0109] S700, using the fused feature map as the input of the next feature extraction module; and classifying the dermoscopic image according to the fused feature map output by the last feature extraction module.

[0110] In the neural network-based dermoscopic image classification method described in the present invention, both the local feature extraction block and the global feature extraction block decompose a complete convolution into deep convolution and point-by-point convolution. First, deep convolution is used to perform convolution operations only in a single channel to extract the features of each channel, and then point-by-point convolution is used to perform linear combinations of the features of each channel between channels. The convolution operation within a single channel greatly reduces the amount of calculation and the number of parameters compared to the inter-channel convolution operation of the ordinary convolutional network. In addition, the significant reduction in the number of parameters will theoretically weaken the accuracy of image classification, but in this scheme, an improved coordinate attention mechanism is added to the residual structure of the network, position information sensitivity is added, information in specific areas is enhanced, and adaptability to various lighting, color and texture transformations is improved, so that the model focuses more on key features and enhances the network's ability to integrate high-dimensional features, ultimately achieving the purpose of improving network classification performance. A multi-head self-attention mechanism is introduced to process local and global feature information, and the computational complexity of the algorithm is optimized while increasing the interaction of multi-head features, capturing global dependencies in the feature map, and enhancing the network's ability to extract global information, thereby indirectly improving the efficiency and accuracy of model classification.

[0111] Specifically, in one embodiment of the present invention, the dermoscopic image to be classified is input into the coordinate attention processing module to obtain the spatial enhancement feature, referring to Figure 6 As shown, it includes the following steps,

[0112] Perform preliminary processing on the dermoscopic image to obtain preliminary features; perform an average pooling operation in the horizontal direction on the preliminary features to extract X-direction features; perform an average pooling operation in the vertical direction on the preliminary features to extract Y-direction features; perform splicing on the X-direction features and the Y-direction features to obtain splicing features; perform segmentation on the splicing features to obtain two-part segmentation features; perform convolution operations on the two-part segmentation features respectively to obtain X-direction weight information and Y-direction weight information; wherein the weight information represents the importance of the corresponding position feature; multiply the X-direction weight information with the preliminary features to obtain X-direction weighted features; multiply the Y-direction weight information with the preliminary features to obtain Y-direction weighted features; take the sum of the X-direction weighted features and the Y-direction weighted features as weighted features; introduce a residual block, and obtain the spatial enhancement features according to the weighted features and the preliminary features; wherein the spatial enhancement features are calculated in the following manner:

[0113] ;

[0114] Among them, F is the spatial enhancement feature; F1 is the weighted feature; F2 is the preliminary feature; is the weight parameter.

[0115] In specific application scenarios, first, features are extracted from the X and Y directions respectively, and through splicing and segmentation operations, useful information in different directions can be captured at the same time, thereby enhancing the diversity of features, so that the model can provide a more comprehensive feature description when facing complex skin lesions. Secondly, the weight information in the X and Y directions is calculated and multiplied with the preliminary features to obtain weighted features. This process is achieved through learning the weight parameters Dynamic adjustment is achieved; this feature enhancement mechanism can adaptively emphasize important features and suppress irrelevant features, thereby improving the representation ability of features and their contribution to classification decisions. Thirdly, by introducing residual blocks, the gradient vanishing problem in deep networks can be effectively alleviated, the transmission of information flow can be promoted, and the full utilization of features in the network can be ensured. This makes the model not only have faster training convergence, but also maintain high performance in deep networks. Finally, through effective feature extraction and weighting mechanisms, the number of parameters that need to be processed is reduced. Compared with traditional deep learning models, it can significantly reduce computational costs while maintaining high classification accuracy.

[0116] Furthermore, unlike the conventional method of setting weight parameters based on expert experience , the weight parameter in the embodiment of the present invention Acquired through learning, which includes,

[0117] Initialize the weight parameters ; The weight parameter is defined as a vector with the same size as the preliminary feature, and each element corresponds to a feature channel; a network model is added to the residual block, and the weight parameter is obtained by adjusting the hyperparameters to train the network model;

[0118] Among them, training the network model includes defining a hyperparameter space and setting candidate values ​​for each hyperparameter that needs to be optimized; creating a model function, which is used to receive a set of hyperparameters and return model performance indicators; training and verifying all hyperparameter combinations, and recording the performance indicators under each set of hyperparameters; selecting the optimal set of hyperparameters based on the performance on the verification set; and obtaining the weight parameters based on the optimal set of hyperparameters.

[0119] In the above embodiment of the present invention, first, the weight parameters are configured. is a vector of the same size as the preliminary feature, with each element corresponding to a feature channel; different weights can be assigned to different channels, so that the model can adjust the importance of features more finely. Secondly, through the learning of adaptive weight parameters and the effective optimization of hyperparameters, not only the accuracy and efficiency of dermoscopic image classification are improved, but also the burden of parameter adjustment is reduced, and the robustness and applicability of the model are improved.

[0120] Furthermore, the dermoscopic image is preliminarily processed to obtain preliminary features, including performing a channel dimensionality increase operation on the dermoscopic image, and normalizing and activating the channel dimensionality increase operation to obtain channel dimensionality increase features; performing a diffuse separable convolution operation on the channel dimensionality increase features to obtain convolution features; normalizing and activating the convolution features to obtain the preliminary features; after obtaining the spatial enhancement features, it also includes performing a channel dimensionality reduction operation on the spatial enhancement features, and inputting the results of the channel dimensionality reduction operation into the feature extraction module.

[0121] On the basis of the above specific embodiments, a channel dimensionality increase operation is performed on the input dermoscopic image, which increases the number of channels of the feature map, so that the model can capture more local information and enhance the feature representation capability; normalization and activation are performed immediately after the channel dimensionality increase, which can accelerate the training process and make the training more stable; normalization helps to alleviate the problem of gradient vanishing or exploding, while the activation function introduces nonlinearity and improves the expressive power of the model; the diffuse separable convolution decomposes the standard convolution into two steps: depth convolution and point-by-point convolution, which significantly reduces the amount of computation and the number of parameters; in addition, despite the reduced amount of computation, the diffuse separable convolution can still effectively capture local features and ensure the quality of feature extraction; finally, the channel dimensionality reduction operation is performed to reduce the number of channels of the feature map, thereby further reducing the size of the model and the number of parameters.

[0122] The local feature extraction block (Local Representation) in the public MobileViT block uses 3*3 convolution and 1*1 convolution for local representation. The receptive field of 3*3 convolution is too small; but if a large kernel convolution is used, although the receptive field is improved, it also increases the amount of calculation and parameters. In order to solve the above contradiction, refer to Figure 5As shown, the depth convolution unit of the local feature extraction block in the embodiment of the present invention includes a 5*5 diffusion separable convolution layer 201 and a 7*7 dilated diffusion separable convolution layer 202; and along the forward propagation direction, the 5*5 diffusion separable convolution layer 201 and the 7*7 dilated diffusion separable convolution layer 202 are arranged in sequence; wherein the dilation rate of the 7*7 dilated diffusion separable convolution layer 202 is configured as 3. The combination of a 5*5 diffusion separable convolution layer 201 and a 7*7 dilated diffusion separable convolution layer 202 with a dilation rate configured as 3 achieves the effect of 21*21 convolution. Specifically, it can be understood as: decomposing the large kernel 21*21 convolution, the decomposition principle is: decomposing the N*N large kernel volume into a (2d-1)*(2d-1) diffusion separable convolution and a (N / d)*(N / d) dilated diffusion separable convolution, d is the dilation rate; based on this decomposition principle, the 21*21 volume is decomposed into a 5*5 diffusion separable convolution and a 7*7 dilated diffusion separable convolution with a dilation rate of 3; where N is 21 and d is 3.

[0123] For a feature map of size H*W, the number of parameters and floating-point operations for a conventional convolution is as follows:

[0124] P=K*K*C i *C o ;

[0125] F=K*K*C i *C o *H*W;

[0126] P is the number of parameters; F is the number of floating-point operations; K is the size of the convolution kernel; C i The number of input channels for the feature map; C o is the number of output channels of the feature map; H is the height of the feature map; W is the width of the feature map.

[0127] The number of parameters and floating-point operations after the large core convolution is decomposed are as follows:

[0128] ;

[0129] ;

[0130] Among them, P is the parameter quantity; F is the floating point operation quantity; K is the size of the convolution kernel; d is the expansion rate; C i The number of input channels for the feature map; C o is the number of output channels of the feature map; H is the height of the feature map; W is the width of the feature map.

[0131] It can be seen that when the expansion rate d increases, the increase in the denominator will lead to a corresponding decrease in the amount of floating-point operations and the number of parameters. Therefore, the use of decomposed large-kernel convolution can not only increase the receptive field of the Local Representation part, but also reduce resource usage, making up for the increase in parameter calculation caused by the introduction of the coordinate attention mechanism.

[0132] Furthermore, a multi-head self-attention mechanism is embedded in the global feature extraction block, and the global feature extraction block also has a point convolution block, which is used to simulate the interactive operation between multiple heads in the multi-head self-attention mechanism. By using the point convolution block to simulate the interactive operation between multiple heads in the multi-head self-attention mechanism, the number of parameters of the model is further reduced.

[0133] The process of performing global feature extraction based on the multi-head self-attention mechanism includes:

[0134] The input feature map is dimensionally reduced based on a diffusion separable convolution operation; the feature map after dimension reduction is mapped to a query Q, a key K, and a value V respectively through a linear transformation; an attention score is obtained based on the query Q and the key K; the attention score is normalized by applying a Softmax function to obtain an attention weight; the attention weight is multiplied by the value V to obtain a weighted value; the weighted value is normalized, and global feature extraction is performed according to the result of the normalization; wherein the self-attention of the multi-head self-attention mechanism is calculated according to the following formula:

[0135] ;

[0136] Where Q represents the query of the multi-head self-attention mechanism; K represents the key of the multi-head self-attention mechanism; V represents the value of the multi-head self-attention mechanism; d k represents the dimension of K; K T Represents the transpose of K; Conv() represents the diffusion separable convolution operation; Softmax() represents the Softmax function; IN() represents the implementation normalization.

[0137] Based on the above specific embodiments, the embodiment of the present invention reduces the computational complexity and the number of parameters by optimizing the standard multi-head self-attention mechanism while maintaining or improving the performance of the model.

[0138] Furthermore, the training process of the image classification model includes obtaining an image sample set; constructing an image classification loss function based on the output of the last feature extraction module; iteratively training the coordinate attention processing module and multiple feature extraction modules using image samples in the image sample set until the value of the image classification loss function is minimized, thereby obtaining a trained image classification model; wherein the image classification loss function is:

[0139] ;

[0140] i represents the sample number; n represents the total number of samples; y i Represents the true value probability of the i-th sample; represents the probability that the i-th sample is evaluated as a positive sample; ln represents the natural logarithm function.

[0141] In order to verify the effectiveness of the image classification model provided by the present application, the embodiment of the present application uses the coordinate attention processing module and feature extraction module in the model to replace the preprocessing module and feature extraction module in various deep learning networks, and conducts experiments on several visual benchmark data sets. The experiment is described in detail below:

[0142] The dataset used for training and testing is the HAM dataset, which has 10,015 images and a resolution of 600×450. It contains dermoscopic images of seven different categories of pixels, namely: melanocytic nevi (NV), basal cell carcinoma (BCC), melanoma (MEL), benign keratosis (BKL), dermatofibroma (DF), vascular skin lesion (VASC) and actinic keratosis (AKIEC). Except for basal cell carcinoma (BCC) and melanoma (MEL), which are malignant tumors, the rest of the skin-related lesions are benign. The diagnosis of all melanomas is confirmed by histopathological evaluation of biopsy, and the diagnosis of melanocytic nevi is confirmed by histopathological examination (24%), expert consensus (54%) or other diagnostic methods.

[0143] The experiment and the hardware and software are shown in Table 1. During training, the data set is divided into training set, validation set and test set in a ratio of 8:1.5:0.5, and there is no intersection between the training set and the validation set. The experiment sets the hyperparameters to train the network, the batch size is 128, the model loss function is set to the cross entropy loss function, the learning rate is 0.0002, the optimizer is AdamW, and the training round (Epoch) is 150.

[0144] Table 1

[0145] hardware software CPU: 8× Xeon E5-2686 v4 Matplotlib 3.3.4 Memory: 30 GB Visual Studio Code 1.78.0 GPU: NVIDIA RTX A4000 Pytorch 1.12.1 Video memory: 16 GB CUDA 11.3 Operating system: Ubuntu 18.04 Python 3.9

[0146] A comprehensive evaluation is conducted from four dimensions: precision, recall, specificity and F1 value. The specific evaluation results are shown in Table 2:

[0147] Table 2

[0148] Clinical diagnosis Accuracy Recall Specificity F1 AKIEC 98.35 97.35 99.73 97.84 BCC 96.00 98.20 99.31 97.08 BKL 93.98 88.37 99.06 91.08 DF 99.09 99.90 99.85 99.49 MEL 93.18 90.02 98.89 91.57 NV 88.85 95.12 97.98 91.88 VASC 99.90 100.00 99.98 99.95 Among them, the units of precision, recall, specificity and F1 are all %

[0149] The image classification model proposed in the scheme of the present invention achieved an overall accuracy of 95.32, an average precision of 95.35%, an average recall of 95.35%, an average specificity of 99.22% and an average F1 value of 95.35% for the HAM dataset classification task. As shown in Table 2, the VASC class has the best classification effect, surpassing other classes in all indicators. Figure 7 The test set accuracy graph of the image classification model is shown in Figure 2. Figure 8 The figure shows the test set loss graph of the image classification model, with 150 training rounds. It can be seen that as the number of training rounds increases, the accuracy rate steadily increases and eventually stabilizes, reaching a maximum of 95.54%, and the loss value gradually decreases and stabilizes around 0.13. Experiments have shown that the model of the implementation scheme of the present invention can effectively perform classification prediction on the HAM10000 data set.

[0150] The confusion matrix of the image classification model of the present invention for each skin condition is as follows: Fig. 9 As shown, according to Fig. 9 It is clear that the classification effect of VASC is the best among all categories. Except for the slight confusion with NV, there are almost no other errors, and the recall rate reaches 100%. The classification effect of DF is also good, with an accuracy of 99.09%. The recognition effect of VASC and DF is the best, with F1 values ​​of 99.95% and 99.49%. These two clinical diagnoses have obvious characteristics under dermoscopy. DF is a skin fibroma, a benign tumor with obvious traces of fibrosis, which is relatively special among other categories. VASC is a vascular skin disease, and the blood vessels are obvious in the dermoscopic image. It is also special compared with other types of lesions and easy to identify. From the confusion matrix, it can be seen that the degree of confusion between NV and MEL is high. NV is a benign nevus cell, and MEL is a malignant cell derived from the deterioration of dysplastic nevus cells. Therefore, there is a smooth transition process between NV and MEL, which may be homologous development, which makes it challenging to identify. The implementation scheme of the present invention makes it possible to increase various types of clinical manifestations in various evaluation indicators, and the specificity reaches more than 95%, which proves that the method implemented by the present invention effectively eliminates confounding targets and is more robust.

[0151] Ablation experiment:

[0152] In order to explore the coordinate attention mechanism, the multi-head self-attention mechanism and the decomposed large kernel convolution respectively improve the network effect, an ablation experiment is set up. The impact of different methods on the model is shown in Table 3 below:

[0153] After adding the CA attention mechanism to the image classification model, the number of parameters increased by about 0.01mb, and the floating-point calculation amount increased by 0.8%, but the model accuracy increased by 2.39%; after adding the decomposed large-core convolution to the model, the number of parameters decreased by 0.1mb, the floating-point calculation amount decreased by about 8.24%, and the performance was slightly improved. Later, the efficient multi-head attention mechanism was added to make the model parameter amount reach 0.82mb, the floating-point calculation law decreased by about 15.71% compared with the original model, and the accuracy increased by about 1.48% compared with the original model. The use of decomposed large-core convolution and efficient multi-head attention mechanism better balances the model reasoning speed brought by CA coordinate attention. Therefore, the implementation scheme of the present invention combines the decomposed large-core convolution, efficient multi-head attention mechanism and CA coordinate attention mechanism to improve the model, the network precision is improved by 3.51% compared with the original network, the network recall rate is improved by 3.37% compared with the original network, the final accuracy is improved by 3.42%, the computational complexity is reduced by about 14.89%, and the number of parameters is reduced by about 8.79%. The method implemented by the present invention has a good classification effect on the HAM10000 data set.

[0154] Table 3

[0155] method Final accuracy / % Accuracy / % Recall rate / % Parameter amount / MB <![CDATA[Floating-point operation amount / 10 6 > MobileViT 92.12 92.11 92.20 0.91 546.13 MobileViT+CA 94.51 94.50 94.54 0.92 550.60 MobileViT+EMSA 93.45 93.66 93.46 0.91 505.31 MobileViT+LKA 92.35 92.37 92.35 0.81 501.09 MobileViT+EMSA+LKA 93.58 93.58 93.61 0.82 460.28 Solution of the present invention 95.54 95.62 95.57 0.83 464.76

[0156] Embodiment 2

[0157] In the second aspect, in order to solve the above-mentioned technical problems, the present invention provides a dermoscopic image classification system based on a neural network, which is used to execute the above-mentioned dermoscopic image classification method based on a neural network, and can effectively realize the image classification of dermoscopic images. The technical effects that can be achieved are as described in the above-mentioned embodiments and will not be repeated here.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A dermatoscopic image classification method based on a neural network, characterized in that: include, Acquire a pre-trained image classification model; wherein the image classification model includes a coordinate attention processing module and a plurality of feature extraction modules sequentially connected along the forward propagation direction, each of the feature extraction modules includes a multi-head self-attention feature extraction submodule; each of the multi-head self-attention feature extraction submodule includes a local feature extraction block, a global feature extraction block and a feature fusion module; the local feature extraction block and the global feature extraction block both include a depth convolution unit and a point-by-point convolution unit; Inputting the dermoscopic image to be classified into the coordinate attention processing module to obtain spatial enhancement features; Inputting the spatial enhancement feature into the depth convolution unit of the local feature extraction block of the first feature extraction module to extract the local features of a single channel, and obtaining the first local features of each channel; Inputting the first local features of each channel into the point-by-point convolution unit of the local feature extraction block of the first feature extraction module to perform linear combination between channels to obtain a first local feature map; Inputting the first local feature map into the deep convolution unit of the global feature extraction block of the first feature extraction module to perform global feature extraction of a single channel to obtain a first global feature of each channel; Inputting the first global features of each channel into the point-by-point convolution unit of the global feature extraction block of the first feature extraction module to perform linear combination between channels to obtain a first global feature map; Inputting the spatial enhancement feature, the first local feature map and the first global feature map into the feature fusion module for feature fusion to obtain a fused feature map; The fused feature map is used as the input of the next feature extraction module; and the dermoscopic image is classified according to the fused feature map output by the last feature extraction module.

2. The neural network-based dermoscopic image classification method according to claim 1, characterized in that: The dermoscopic image to be classified is input into the coordinate attention processing module to obtain spatial enhancement features, including: Performing preliminary processing on the dermoscopic image to obtain preliminary features; Performing an average pooling operation in the horizontal direction on the preliminary features to extract X-direction features; Performing an average pooling operation in the vertical direction on the preliminary features to extract features in the Y direction; Performing stitching on the X-direction feature and the Y-direction feature to obtain a stitching feature; Segmenting the splicing feature to obtain two parts of segmentation features; Performing convolution operations on the two parts of the segmentation features respectively to obtain X-direction weight information and Y-direction weight information; wherein the weight information represents the importance of the corresponding position feature; Multiplying the X-direction weight information by the preliminary feature to obtain an X-direction weighted feature; multiplying the Y-direction weight information by the preliminary feature to obtain a Y-direction weighted feature; taking the sum of the X-direction weighted feature and the Y-direction weighted feature as a weighted feature; A residual block is introduced, and the spatial enhancement feature is obtained according to the weighted feature and the preliminary feature; wherein the spatial enhancement feature is obtained by calculating in the following manner: ; Among them, F is the spatial enhancement feature; F1 is the weighted feature; F2 is the preliminary feature; is the weight parameter.

3. The neural network-based dermoscopic image classification method according to claim 2, characterized in that: The weight parameter Acquired through learning, which includes, Initialize the weight parameters ; The weight parameter is defined as a vector of the same size as the preliminary feature, each element corresponding to a feature channel; Add a network model to the residual block, and train the network model by adjusting hyperparameters to obtain the weight parameters; Wherein, training the network model includes: Define the hyperparameter space and set candidate values ​​for each hyperparameter that needs to be optimized; Creating a model function, the model function is used to receive a set of the hyperparameters and return a model performance indicator; Train and validate all hyperparameter combinations, and record the performance indicators under each set of hyperparameters; Select the best set of hyperparameters based on the performance on the validation set; The weight parameters are obtained according to the optimal set of hyperparameters.

4. The neural network-based dermoscopic image classification method according to claim 2, characterized in that: The dermoscopic image is preliminarily processed to obtain preliminary features, including: Performing a channel dimension-upgrading operation on the dermoscopic image, and then normalizing and activating the channel dimension-upgrading operation to obtain a channel dimension-upgrading feature; Performing a diffusion separable convolution operation on the channel dimension-raising feature to obtain a convolution feature; Normalizing and activating the convolutional features to obtain the preliminary features; After obtaining the spatial enhancement feature, the method further includes performing a channel dimensionality reduction operation on the spatial enhancement feature, and inputting the result of the channel dimensionality reduction operation into the feature extraction module.

5. The neural network-based dermoscopic image classification method according to claim 1, characterized in that: The depth convolution unit of the local feature extraction block includes a 5*5 diffusion separable convolution layer and a 7*7 dilated diffusion separable convolution layer; and along the forward propagation direction, the 5*5 diffusion separable convolution layer and the 7*7 dilated diffusion separable convolution layer are arranged in sequence; wherein the dilation rate of the 7*7 dilated diffusion separable convolution layer is configured to be 3.

6. The neural network-based dermoscopic image classification method according to claim 1, characterized in that: A multi-head self-attention mechanism is embedded in the global feature extraction block, and the global feature extraction block also has a point convolution block, which is used to simulate the interactive operation between multiple heads in the multi-head self-attention mechanism.

7. The neural network-based dermoscopic image classification method according to claim 6, characterized in that: The process of performing global feature extraction based on the multi-head self-attention mechanism includes: Reduce the dimension of the input feature map based on the diffusion separable convolution operation; Map the dimensionally reduced feature map to query Q, key K, and value V respectively through linear transformation; Get an attention score based on the query Q and key K; Apply the Softmax function to normalize the attention score and obtain the attention weight; Multiply the attention weight by the value V to obtain a weighted value; Normalizing the weighted values, and performing global feature extraction according to the normalized results; The self-attention of the multi-head self-attention mechanism is calculated according to the following formula: ； Where Q represents the query of the multi-head self-attention mechanism; K represents the key of the multi-head self-attention mechanism; V represents the value of the multi-head self-attention mechanism; d k represents the dimension of K; K T Represents the transpose of K; Conv() represents the diffusion separable convolution operation; Softmax() represents the Softmax function; IN() represents the implementation normalization.

8. The neural network-based dermoscopic image classification method according to claim 1, characterized in that: The training process of the image classification model includes: Get a sample set of images; Constructing an image classification loss function based on the output of the last feature extraction module; Iteratively training the coordinate attention processing module and the plurality of feature extraction modules using the image samples in the image sample set until the value of the image classification loss function is minimized, thereby obtaining a trained image classification model; Among them, the image classification loss function is: ； i represents the sample number; n represents the total number of samples; y i Represents the true value probability of the i-th sample; represents the probability that the i-th sample is evaluated as a positive sample; ln represents the natural logarithm function.

9. The neural network-based dermoscopic image classification method according to claim 1, characterized in that: The image classification model also includes a coordinate attention downsampling module; wherein the output of the coordinate attention processing module is connected to the coordinate attention downsampling module; the output of the feature extraction module is connected to the coordinate attention downsampling module; the coordinate attention downsampling module is used to perform downsampling operations on the connected input features.

10. A dermatoscopic image classification system based on a neural network, characterized in that: Used to perform the neural network-based dermoscopic image classification method as described in any one of claims 1-9.

Images