A deep interpretable skin disease recognition method based on small sample learning

Abstract

The present invention discloses a deep interpretable skin disease recognition method based on small sample learning, which belongs to the field of image processing. The same or different skin disease images are respectively input into a double-branch small sample contrast learning network, the lesion area of ​​the skin disease image is segmented to generate a lesion segmentation mask, the shape information is extracted, and the minimum square frame containing the lesion area is located, and the lesion image is obtained by cropping and scaling; the depth and shallow features of the corresponding branches of the two branch lesion images are extracted, and the contrast loss of the double-branch depth and shallow features is calculated to optimize the network parameters, and multi-scale texture information is extracted at the same time; metadata containing shape information and texture information is multi-modally fused with the depth and shallow features of the image for skin disease recognition, and the morphology, texture and other features of the lesion area are gradually extracted and fused through the combination of a multi-stage deep learning network and medical diagnosis logic, so as to improve the classification accuracy of skin disease recognition and the interpretability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing in computer vision, and in particular to a deep interpretable skin disease recognition method based on small sample learning. Background Art

[0002] In recent years, the incidence of skin diseases has increased year by year and has become an important issue affecting human health. Early detection and treatment of skin diseases can significantly improve patient prognosis and greatly reduce mortality. For example, the survival rate of patients with the most dangerous melanoma can be increased from 25% to 99%. Due to the limitations of clinicians' experience level, pressure or other factors, misdiagnosis may occur. Therefore, the use of computer vision technology based on deep learning to assist in skin disease identification has broad application prospects and social value.

[0003] Although existing deep learning methods have achieved some success in skin disease recognition, they still face some significant challenges. First, existing deep learning-based skin disease recognition methods generally rely on large-scale annotated datasets. Due to the difficulty of collecting skin disease datasets, limited sample numbers, and uneven distribution of species, traditional deep learning methods perform poorly under small sample conditions, which seriously affects the accuracy and robustness of the model. In addition, existing deep learning methods often lack sufficient interpretability, resulting in the diagnosis process being a "black box" for doctors, lacking transparency, and failing to provide clear pathological evidence, thus affecting the trust of clinical decision-making. At the same time, the lesion areas of skin diseases vary in morphology and are often accompanied by background interference. Existing methods often lack the ability to accurately segment the lesion area and comprehensively extract multi-level features such as morphology and texture, resulting in the diagnosis effect being difficult to meet actual needs. Finally, the diagnosis of skin diseases usually relies not only on image information, but also requires a comprehensive judgment based on the patient's metadata. However, many current deep learning models only analyze image features and fail to effectively integrate multimodal data for comprehensive diagnosis, which restricts the applicability of the model in actual clinical environments. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a deep interpretable skin disease recognition method based on small sample learning in response to the shortcomings of the background technology.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] A deep interpretable skin disease recognition method based on small sample learning includes the following steps:

[0007] S1. Randomly extract two original skin disease images of the same or different categories from the skin disease image dataset and input them into the dual-branch small sample contrastive learning network DB-FSCLNet with the same structure respectively;

[0008] S2, using the IRA-UNet segmentation module to segment the lesion area of ​​the original skin disease image and generate lesion segmentation masks of two corresponding branches;

[0009] S3, inputting the lesion segmentation mask into the morphological positioning module, extracting shape information from the lesion segmentation mask using a geometric morphological feature extraction method, determining the minimum square frame of the lesion area, cropping and scaling it, and obtaining the shape information of the corresponding branch and the lesion image;

[0010] S4. In the MC-GLCM texture extraction module, the texture information of the two branch lesion images at different scales is calculated using the multi-scale gray-level co-occurrence matrix, and the feature variance weighted strategy is used for weighted fusion to obtain the texture information;

[0011] S5, using the CGRNet feature extraction module to extract the shallow features and deep features of the lesion image obtained in S3, and fusing the shallow features and deep features through multiple convolutional gated recurrent units to obtain the shallow and deep features of the two branches respectively;

[0012] S6. Using the contrastive learning module, the contrast loss between the deep and shallow features of the two branches is calculated to optimize the CGRNet network parameters and enhance the feature representation ability of small sample data;

[0013] S7. Use the CMFF fusion module to perform multimodal fusion of shape information, texture information and depth features to classify and identify skin diseases;

[0014] S8. Output interpretable results, including skin disease categories, shape information, texture information, and stage visualization maps.

[0015] As a further preferred embodiment of the deep interpretable skin disease recognition method based on small sample learning of the present invention, in step S1, the dual-branch small sample comparative learning network is combined with the actual diagnosis logic of professional doctors, including the following stages:

[0016] The focus and magnification stage simulates the doctor's focus on the lesion image and the lesion area during the diagnosis process. The lesion area of ​​the original skin disease image is segmented through the IRA-UNet segmentation module to generate a lesion segmentation mask. The shape information of the lesion segmentation mask is extracted using the morphological localization network, and the minimum square frame including the lesion area is cropped and scaled.

[0017] The feature extraction and comparison stage simulates the doctor's observation of the lesion area and the comparison of different lesions during the diagnosis process. The MC-GLCM texture extraction network is used to extract multi-scale texture information, and the depth and shallow features of the lesion image are obtained through the CGRNet feature extraction module. The contrast loss between the depth and shallow features of the two branches is calculated through the contrast learning module to optimize the CGRNet network parameters.

[0018] The fusion recognition stage simulates the doctor's diagnosis process by integrating image information and clinical information to make diagnostic decisions. The CMFF fusion module is used to perform multimodal fusion of shape information, texture information metadata information and image information of depth and shallowness features for classification decisions and output of interpretable results.

[0019] As a further preferred embodiment of a deep interpretable skin disease recognition method based on small sample learning of the present invention, in step S2, the IRA-UNet segmentation module segments the lesion area of ​​the original skin disease image to obtain lesion segmentation masks of two branches, including the following steps:

[0020] S2.1, the encoder part uses the inverted residual structure to perform dimension increase, feature extraction, and dimension reduction operations on the feature map of each layer input in sequence. Dimension increase improves the feature representation capability, dimension reduction reduces the computational overhead, and extracts rich semantic information in the high-dimensional space. The calculation process is expressed as:

[0021] in, X , Y Represent the input feature map and output feature map respectively, Represents a 1×1 standard convolution, which is used to increase the spatial dimension; Indicates that the convolution kernel is 3×3 channel-by-channel convolution, which is used for feature extraction; Indicates that the convolution kernel is 1×1 point-by-point convolution, which is used to reduce the spatial dimension;

[0022] S2.2, the decoder part uses deconvolution operation for upsampling, and learns the weight parameters of the convolution kernel through back propagation. The deconvolution operation accurately restores the original spatial structure in the layer-by-layer reconstruction process and improves the boundary details of the lesion area;

[0023] S2.3, add channel space attention block CSA to the jump connection between the encoder of each layer and the decoder of the corresponding layer to transfer features, calculate feature weights from the channel dimension and spatial dimension respectively, perform weighted fusion processing on the input feature map, focus on key areas and improve segmentation effect. CSA contains channel attention block and spatial attention block. The channel attention block uses spatial average pooling and spatial maximum pooling to integrate the semantic information of each channel. The channel attention weight map is obtained through the fully connected layer and then combined with the input feature map. F Perform pixel-level multiplication to obtain the features enhanced by the channel attention block The spatial attention block uses channel average pooling and channel maximum pooling to model the spatial relationship, strengthens the element connection in the feature through the convolution layer, obtains the spatial attention weight map, and then combines it with the input feature F Perform pixel-level multiplication to obtain the features enhanced by the spatial attention block , in order to summarize the feature information of the channel attention block and the spatial attention block, and and F Perform weighted fusion to obtain , the calculation formula is as follows: , , ; where ⊙ represents element-by-element multiplication, W 1 and b 1 is the weight and bias of the fully connected layer, σ is the sigmoid activation function, [;] represents the concatenation operation, represents spatial average pooling, Represents the maximum spatial pooling, Conv 7×7 represents a convolution operation with a convolution kernel of 7×7. represents channel average pooling, represents channel maximum pooling, represents the weighted fusion feature, α and β are two learnable weighting factors with an initial value of 0, which are used to adjust the weights of channel attention features and spatial attention features.

[0024] As a further preferred embodiment of the method for deeply interpretable skin disease recognition based on small sample learning of the present invention, in step S3, the shape information is extracted from the lesion segmentation mask using the morphological localization module, and the minimum square bounding box of the lesion area is determined for cropping and scaling, including the following steps:

[0025] S3.1, calculating the lesion pixel area, pixel perimeter, and lesion ratio through the lesion segmentation mask, wherein the lesion segmentation mask M is a binary image, i and j Represents a pixel, M ( i , j ) = 1 indicates a diseased pixel, M ( i , j ) = 0 indicates background pixels, S ={( i , j )| M ( i , j )=1} represents the pixel set of the lesion area, represents the pixel area of ​​the lesion area, represents the pixel perimeter of the lesion area, where Indicates that the boundary pixels of the lesion area are obtained by edge detection algorithm. The lesion ratio is shown by calculating the pixel area of ​​the lesion areaA Total pixel area of ​​the image A total The ratio of

[0026] S3.2, calculate the multi-angle symmetry of the lesion area, and mirror-flip the lesion area according to the given flip axes at 0°, 45°, 90° and 135°. The areas of the lesion area after flipping are expressed as A 0° , A 45° , A 90° and A 135° , the multi-angle symmetry calculation formula is as follows: ;

[0027] S3.3, the regularity of the edge of the lesion area is evaluated using a comprehensive regularity evaluation index that combines roundness, edge smoothness, and edge complexity. The calculation formula is as follows: ;in, , and is the learnable weight coefficient, is the circularity, which is used to measure the similarity between the lesion area and the circle. The closer the circularity is to 1, the closer the lesion area is to a circle. is the edge complexity, which is used to measure the curvature change of the edge of the lesion area. is the curvature of each point on the edge, is the mean of the edge curvatures, O is the number of edge points;

[0028] S3.4, calculate the minimum bounding rectangle of the lesion area, and the pixel coordinates corresponding to the upper, lower, left and right boundaries of the lesion area are , , , , then the side length of the bounding box can be expressed as ,by and Construct the minimum square bounding box coordinates for the lesion center as , crop it, remove irrelevant background and scale the cropped lesion area to the standard size of 224×224 to ensure the consistency of subsequent input.

[0029] As a further preferred embodiment of a deep interpretable skin disease recognition method based on small sample learning of the present invention, in step S4, multi-scale texture information of the lesion area is extracted by the MC-GLCM texture extraction module, and weighted by the feature variance weighted fusion strategy to obtain texture information, including the following steps:

[0030] S4.1, quantize the pixel values ​​of the lesion area and discretize the original image values ​​into G grayscale levels, generate a grayscale image I , its image value i ∈{0,1,2,…, G −1}, the gray value of each pixel is determined by the pixel intensity of the original image, and the size is D s × D s The lesion area is sampled by a sliding window of s ∈{1,2,…, S} represents different scales, and the multi-scale gray-level co-occurrence matrix is ​​calculated. The calculation formula is as follows: in, Indicated in scale W s × W s and angle θ Lower gray value i and j The co-occurrence frequency, relative displacement ( dx ( θ ), dy ( θ )) means at a given angle θ The offset between adjacent pixels ，I s Indicated in D s × D s The image area at scale, Indicated in the image I Satisfied and The number of pixel pairs for the condition;

[0031] S4.2, through the multi-scale gray level co-occurrence matrix, the texture information at different scales is calculated respectively. F s , multi-scale texture features are obtained through feature variance weighted fusion strategy F Texture , the calculation formula is as follows: in, F s Indicates s Texture feature vector at each scale, Var( F s ) represents the characteristic variance, is the weight coefficient assigned to each scale, given by sThe ratio of the feature variance of texture features at a scale to the total feature variance of all scales reflects the variation range or significance of the texture information at that scale. The larger the variance, the more significant the texture information at that scale and the higher the weight.

[0032] As a further preferred embodiment of a deep interpretable skin disease recognition method based on small sample learning of the present invention, in step S5, the CGRNet feature extraction module realizes the fusion of shallow features and deep features, and obtains the deep and shallow features of the two branches respectively, including the following steps:

[0033] S5.1, CGRNet uses CNN architecture to extract shallow and deep features of lesion images, and uses multiple convolutional gated recurrent units to fuse features stage by stage, where convolution operations are used to replace the linear structure of the gated unit. The current stage features X i And the output features of the previous stage H t-1 Keep the feature dimensions consistent in the convolutional space and choose the gate r t , Fusion Door z t and the generated status , the calculation formula is as follows: in, t Represents the first t Stage, Conv t ( t =1,2,3…, T ) represents the convolution operation of different channels to ensure the dimensional consistency of the input features. σ represents the Sigmoid activation function, ∗ represents the pixel-level product;

[0034] S5.2, convolutional gated recurrent unit by selecting gate r t Dynamically regulate the characteristics of the previous stage H t-1 The contribution ratio of shallow information can still play a role in deep feature fusion; through the fusion gate z t Selectively fuse the current input X t And the output features of the previous stage H i-1 , realize dynamic updating and filtering of information, discard unimportant historical information and retain key features, selectively filter and dynamically adjust features at different stages, and output state features H t The calculation formula is as follows: There is no previous output feature at the initial stage of the model Ht-1 , then by introducing the zero-state initialization strategy, the hidden state is generated H 0=0.

[0035] As a further preferred embodiment of a deep interpretable skin disease recognition method based on small sample learning of the present invention, in step S6, the contrast learning module optimizes the CGRNet network parameters under the condition of small sample by designing a contrast loss function between the double-branch deep and shallow features, including the following steps:

[0036] S6.1, obtain the depth features of the two branches from the CGRNet feature extraction module f a and f b , the Euclidean distance between the two is , calculate the dynamic weighting factor based on the characteristic distance of the sample w , the calculation formula is as follows: in, d max is a normalization factor, which indicates the maximum value of the feature distance and ensures that the weight is in the range of [0, 1]. γ is the adjustment factor to control the difference in weights of the contribution of difficult-to-classify samples and easy-to-classify samples to the loss. ϵ is a smoothing factor to avoid numerical instability during normalization. u represents the sample category, u =1 is the same sample, u =0 indicates heterogeneous samples;

[0037] S6.2, using adaptive threshold m Dynamically distinguish between similar and heterogeneous samples, m It is obtained by calculating the average feature distance of samples from different batches. The calculation formula is as follows: in, Indicates the average value of the characteristic distance of the current batch of samples, reflecting the dynamic changes of sample distribution; and is a hyperparameter used to adjust the flexibility and fixed offset of the threshold;

[0038] S6.3, Calculate the contrast loss function by combining dynamic weighting factor and adaptive threshold L contrastive , the calculation formula is as follows: .

[0039] As a further preferred embodiment of the method for deeply interpretable skin disease recognition based on small sample learning of the present invention, in step S7, the CMFF fusion module performs multimodal fusion of shape information, texture information and depth features for skin disease recognition, including the following steps:

[0040] S7.1, uses multi-head attention to transform image features of shape information, texture information metadata features and depth features, through N A linear transformation matrix transforms the metadata features into query vectors q n , using image features as key vectors k n Sum value vector v n ,capturing the association between image features and metadata features from multiple perspectives, q n , k n and v n The specific expressions are as follows: in, F meta Represents metadata characteristics, F image Represents image features, with sizes of 1×1× g ,

[0041] is the linear transformation matrix, is the linear transformation projection n Size;

[0042] S7.2, using the improved cross-attention mechanism to calculate q n and k n The similarity of is obtained to obtain the multimodal weight map. The tanh function is used as the gating function. The tanh function maps the attention weight to the range of [−1,1]. When the gating value is close to 1, it indicates that the feature has a higher weight in the fusion process; when the gating value is close to -1, the influence of the feature is suppressed, and the key information of different modalities is fused. The multimodal weight map is similar to v n The attention vector is obtained by multiplication, and then it is concatenated into 1×1×( N × g ) size features, and then obtain multimodal fusion features through linear changes , the specific calculation is as follows: , ;in, head n Indicates n attention vector, is the linear transformation matrix.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The present invention adopts a dual-branch network to calculate the feature distance loss of the two branches through a contrast learning module to optimize the network parameters, which can learn more effective and robust feature representations on small sample data, reduce the dependence on large-scale labeled data, and improve recognition accuracy and model generalization ability;

[0045] 2. By designing an interpretable network architecture, the present invention can clearly demonstrate the features and judgment process that the model relies on when making diagnostic decisions, providing important support for subsequent model optimization and in-depth analysis of the diagnostic process, and enhancing the credibility and acceptance in clinical applications;

[0046] 3. The present invention combines multi-dimensional information such as skin images, shape learning and texture information to generate multimodal fusion features. Through the complementarity of different modal information, the model can more comprehensively characterize the characteristics of skin diseases and improve the recognition accuracy of lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of the deep interpretable skin disease recognition method based on small sample learning of the present invention;

[0048] Figure 2 This is a diagram of the overall model structure of the dual-branch small sample contrast learning network of the present invention;

[0049] Figure 3 This is the IRA-Unet segmentation network structure diagram of the present invention;

[0050] Figure 4 This is the CGRNet feature extraction network structure diagram of the present invention;

[0051] Figure 5 This is the CMFF feature fusion network structure diagram of the present invention;

[0052] Figure 6 The confusion matrix tested by the HAM10000 data set in the present invention; wherein, (a) is the confusion matrix tested by the HAM10000 data set in the present invention; (b) is the confusion matrix tested by the Derm7pt data set in the present invention;

[0053] Figure 7 This is a line chart comparing the accuracy of the DB-FSCLNet model of the present invention and other models;

[0054] Figure 8 This is a comparison diagram of the segmentation effect of the IRA-UNet segmentation module of the present invention;

[0055] Fig. 9 This is a visualization effect diagram of the SC-GLCM texture feature extraction module of the present invention;

[0056] Fig.10 This is a diagram showing the interpretability output results of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the application is further elaborated in detail below in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments involved in the present invention. All non-innovative embodiments of other researchers in the field on this embodiment belong to the protection scope of the present invention. At the same time, for the step numbering in the embodiment of the present invention, it is only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In one embodiment of the present invention, a flowchart of a deep interpretable skin disease recognition method based on small sample learning is shown as follows: Figure 1 As shown, the following steps are included:

[0059] S1, randomly extract two original skin disease images of the same or different categories from the skin disease image dataset, and input them into the dual-branch small sample contrastive learning network DB-FSCLNet with the same structure. The model structure is as follows Figure 2 As shown in the figure, the model combines the actual diagnostic logic of professional doctors and includes the following stages:

[0060] The focus and magnification stage simulates the doctor's focus on the lesion image and the lesion area during the diagnosis process. The lesion area of ​​the original skin disease image is segmented through the IRA-UNet segmentation module to generate a lesion segmentation mask. The shape information of the lesion segmentation mask is extracted using the morphological localization network, and the minimum square frame including the lesion area is cropped and scaled.

[0061] The feature extraction and comparison stage simulates the doctor's observation of the lesion area and the comparison of different lesions during the diagnosis process. The MC-GLCM texture extraction network is used to extract multi-scale texture information, and the depth and shallow features of the lesion image are obtained through the CGRNet feature extraction module. The contrast loss between the depth and shallow features of the two branches is calculated through the contrast learning module to optimize the CGRNet network parameters.

[0062] The fusion recognition stage simulates the doctor's diagnosis process of integrating image information and clinical information to make diagnostic decisions. The CMFF fusion module is used to perform multimodal fusion of metadata information such as shape information and texture information with image information of depth and shallowness features for classification decisions and output of explainable results.

[0063] S2, the original skin disease image is input into the IRA-UNet segmentation module to segment its lesion area and generate two corresponding branches of lesion segmentation masks. Figure 3 As shown, the specific process includes the following:

[0064] S2.1, the encoder part uses an inverted residual structure to perform dimension increase, feature extraction, and dimension reduction operations on the feature map of each layer input in turn. The dimension increase operation increases the spatial dimension of the feature map through 1×1 standard convolution, which increases the model's ability to express complex patterns and semantic information in high-dimensional space; then, 3×3 channel-by-channel convolution is used for feature extraction to extract richer local features in the spatial dimension; finally, 1×1 point-by-point convolution reduces the spatial dimension to reduce computational overhead and avoid memory and computational bottlenecks caused by excessively high dimensions. Through this process of dimension increase and dimension reduction, the model can extract more detailed and rich lesion information in high-dimensional space while maintaining a low computational cost. The computational process is expressed as:

[0065] in, X , Y Represent the input feature map and output feature map respectively, Represents a 1×1 standard convolution, which is used to increase the spatial dimension; Indicates that the convolution kernel is 3×3 channel-by-channel convolution, which is used for feature extraction; Indicates that the convolution kernel is 1×1 point-by-point convolution, which is used to reduce the spatial dimension;

[0066] S2.2, the decoder part uses deconvolution operation for upsampling, and learns the weight parameters of the convolution kernel through back propagation. The deconvolution operation accurately restores the original spatial structure in the layer-by-layer reconstruction process and improves the boundary details of the lesion area;

[0067] S2.3, add a channel and spatial attention block (CSA) to the jump connection between the encoder of each layer and the decoder of the corresponding layer to transfer features, calculate feature weights from the channel dimension and spatial dimension respectively, perform weighted fusion processing on the input feature map, focus on key areas and improve segmentation effect. CSA contains channel attention blocks and spatial attention blocks. The channel attention block uses spatial average pooling and spatial maximum pooling to integrate the semantic information of each channel. The channel attention weight map is obtained through the fully connected layer and then combined with the input feature map. F Perform pixel-level multiplication to obtain the features enhanced by the channel attention block The spatial attention block uses channel average pooling and channel maximum pooling to model the spatial relationship, strengthens the element connection in the feature through the convolution layer, obtains the spatial attention weight map, and then combines it with the input feature FPerform pixel-level multiplication to obtain the features enhanced by the spatial attention block , in order to summarize the feature information of the channel attention block and the spatial attention block, and and F Perform weighted fusion to obtain , the calculation formula is as follows: , , ; where ⊙ represents element-by-element multiplication, W 1 and b 1 is the weight and bias of the fully connected layer, σ is the sigmoid activation function, [;] represents the concatenation operation, represents spatial average pooling, Represents the maximum spatial pooling, Conv 7×7 represents a convolution operation with a convolution kernel of 7×7. represents channel average pooling, represents channel maximum pooling, represents the weighted fusion feature, α and β are two learnable weighting factors with an initial value of 0, which are used to adjust the weights of channel attention features and spatial attention features.

[0068] S3, input the lesion segmentation mask into the morphological positioning module, extract shape information from the lesion segmentation mask using the geometric morphological feature extraction method, determine the minimum square border of the lesion area and crop and scale it, obtain the morphological information and lesion image of the pixel perimeter, pixel area, symmetry, and edge regularity of the lesion of the corresponding branch, and perform quantitative analysis on the geometric features of the lesion, which specifically includes the following steps:

[0069] S3.1, calculating the lesion pixel area, pixel perimeter, and lesion ratio through the lesion segmentation mask, wherein the lesion segmentation mask M is a binary image, i and j Represents a pixel, M ( i , j ) = 1 indicates a diseased pixel, M ( i , j ) = 0 indicates background pixels, S ={( i , j )| M ( i , j )=1} represents the pixel set of the lesion area, represents the pixel area of ​​the lesion area, represents the pixel perimeter of the lesion area, where Indicates that the boundary pixels of the lesion area are obtained by edge detection algorithm. The lesion ratio is shown by calculating the pixel area of ​​the lesion area A Total pixel area of ​​the image A total The ratio of

[0070] S3.2, calculate the multi-angle symmetry of the lesion area, and mirror-flip the lesion area according to the given flip axes at 0°, 45°, 90° and 135°. The areas of the lesion area after flipping are expressed as A 0° , A 45° , A 90° and A 135° , the multi-angle symmetry calculation formula is as follows: ;

[0071] S3.3, the regularity of the edge of the lesion area is evaluated using a comprehensive regularity evaluation index that combines roundness, edge smoothness, and edge complexity. The calculation formula is as follows: ;in, , and is the learnable weight coefficient, is the circularity, which is used to measure the similarity between the lesion area and the circle. The closer the circularity is to 1, the closer the lesion area is to a circle. is the edge complexity, which is used to measure the curvature change of the edge of the lesion area. is the curvature of each point on the edge, is the mean of the edge curvatures, O is the number of edge points;

[0072] S3.4, calculate the minimum bounding rectangle of the lesion area, and the pixel coordinates corresponding to the upper, lower, left and right boundaries of the lesion area are , , , , then the side length of the bounding box can be expressed as ,by and Construct the minimum square bounding box coordinates for the lesion center as , crop it, remove irrelevant background and scale the cropped lesion area to the standard size of 224×224 to ensure the consistency of subsequent input.

[0073] S4, in the MC-GLCM texture extraction module, the multi-scale gray-level co-occurrence matrix is ​​used to calculate the texture information of the two branch lesion images at different scales, including contrast, energy, uniformity, and entropy, and the feature variance weighted strategy is used for weighted fusion to obtain the texture information, which specifically includes the following process:

[0074] S4.1, quantize the pixel values ​​of the lesion area and discretize the original image values ​​into G grayscale levels, generate a grayscale image I , its image value i ∈{0,1,2,…, G −1}, the gray value of each pixel is determined by the pixel intensity of the original image, and the size is D s × D s The lesion area is sampled by a sliding window of s ∈{1,2,…, S} represents different scales, and the multi-scale gray-level co-occurrence matrix is ​​calculated. The calculation formula is as follows: in, Indicated in scale W s × W s and angle θ Lower gray value i and j The co-occurrence frequency, relative displacement ( dx ( θ ), dy ( θ )) means at a given angle θ The offset between adjacent pixels ，I s Indicated in D s × D s The image area at scale, Indicated in the image I Satisfied and The number of pixel pairs for the condition;

[0075] S4.2, calculate the contrast at different scales through the multi-scale gray level co-occurrence matrix C ontrast ,energy E nergy , Uniformity H omogeneity and entropy H entro , the texture information is represented as F s ={C ontrast , E nergy , H omogeneity , H entrop}, contrast measures the degree of change in the grayscale in the image. Generally, the higher the contrast, the more obvious the texture difference in the image; energy measures the energy density of the image texture and is usually used to indicate the degree of detail of the image; uniformity indicates the uniformity of the grayscale in the image, and the larger the value, the smoother the texture; entropy measures the complexity of the image, and the higher the entropy value, the more complex the image texture. The specific calculation formula is as follows:

[0076] , , , Multi-scale texture features are obtained through feature variance weighted fusion strategy F Texture , the calculation formula is as follows: in, F s Indicates s Texture feature vector at each scale, Var( F s ) represents the characteristic variance, is the weight coefficient assigned to each scale, given by s The ratio of the feature variance of texture features at a scale to the total feature variance of all scales reflects the variation range or significance of the texture information at that scale. The larger the variance, the more significant the texture information at that scale and the higher the weight.

[0077] S5, the CGRNet feature extraction module is used to extract the shallow features and deep features of the lesion image obtained in S3, and the shallow features and deep features are fused through multiple convolutional gated recurrent units to obtain the shallow and deep features of the two branches respectively. The structure diagram of CGRNet is shown in Figure 4 As shown in the figure, this example CNN architecture uses ResNet50 as the backbone network. Other CNN models can be applied. Specifically, the following steps are included:

[0078] S5.1, CGRNet uses CNN architecture to extract shallow and deep features of lesion images, and uses multiple convolutional gated recurrent units to fuse features stage by stage, where convolution operations are used to replace the linear structure of the gated unit. The current stage features X i And the output features of the previous stage H t-1 Keep the feature dimensions consistent in the convolutional space and choose the gate r t , Fusion Door zt and the generated status , the calculation formula is as follows: in, t Represents the first t Stage, Conv t ( t =1,2,3…, T ) represents the convolution operation of different channels to ensure the dimensional consistency of the input features. σ represents the Sigmoid activation function, ∗ represents the pixel-level product;

[0079] S5.2, convolutional gated recurrent unit by selecting gate r t Dynamically regulate the characteristics of the previous stage H t-1 The contribution ratio of shallow information can still play a role in deep feature fusion; through the fusion gate z t Selectively fuse the current input X t And the output features of the previous stage H i-1 , realize dynamic updating and filtering of information, discard unimportant historical information and retain key features, selectively filter and dynamically adjust features at different stages, and output state features H t The calculation formula is as follows: There is no previous output feature at the initial stage of the model H t-1 , then by introducing the zero-state initialization strategy, the hidden state is generated H 0=0.

[0080] S6, using the contrastive learning module, combined with the dynamic weighting factor and the adaptive threshold to design the contrast loss function between the deep and shallow features of the two branches, calculate the contrast loss between the deep and shallow features of the two branches, optimize the CGRNet network parameters under the condition of small samples, and enhance the feature representation ability of small sample data, specifically including the following steps:

[0081] S6.1, obtain the depth features of the two branches from the CGRNet feature extraction module f a and f b , the Euclidean distance between the two is , calculate the dynamic weighting factor based on the characteristic distance of the sample w , which weights the loss according to the feature distance of the samples, so that the contribution of samples that are difficult to distinguish to the loss increases, while the contribution of samples that are easy to distinguish to the loss decreases. The calculation formula is as follows: in, d max is a normalization factor, which indicates the maximum value of the feature distance and ensures that the weight is in the range of [0, 1]. γ is the adjustment factor to control the difference in weights of the contribution of difficult-to-classify samples and easy-to-classify samples to the loss. ϵ is a smoothing factor to avoid numerical instability during normalization. u represents the sample category, u =1 is the same sample, u =0 indicates heterogeneous samples;

[0082] S6.2, using adaptive threshold m Dynamically distinguish between similar and heterogeneous samples, m It is obtained by calculating the average feature distance of samples from different batches. The calculation formula is as follows: in, Indicates the average value of the characteristic distance of the current batch of samples, reflecting the dynamic changes of sample distribution; and is a hyperparameter used to adjust the flexibility and fixed offset of the threshold; the dynamic threshold m reflects the distribution of the current batch of samples and is adaptively adjusted according to the average value of the feature distance. When the distribution of samples changes, the threshold m will also be updated, helping the model to better handle difficult and easy samples in different batches.

[0083] S6.3, Calculate the contrast loss function by combining dynamic weighting factor and adaptive threshold L contrastive , the calculation formula is as follows: .

[0084] S7, using the CMFF fusion module to perform multimodal fusion of the metadata features of shape information and texture information with the image features of depth features to classify and identify skin diseases. The CMFF fusion module is as follows Figure 5 As shown, the specific steps include:

[0085] S7.1, uses multi-head attention to transform image features of shape information, texture information metadata features and depth features, through N A linear transformation matrix transforms the metadata features into query vectors q n , using image features as key vectors k n Sum value vector v n ,capturing the association between image features and metadata features from multiple perspectives, q n , k n and vn The specific expressions are as follows: in, F meta Represents metadata characteristics, F image Represents image features, with sizes of 1×1× g ,

[0086] is the linear transformation matrix, is the linear transformation projection n Size;

[0087] S7.2, using the improved cross-attention mechanism to calculate q n and k n The similarity of is obtained to obtain the multimodal weight map. The tanh function is used as the gating function. The tanh function maps the attention weight to the range of [−1,1]. When the gating value is close to 1, it indicates that the feature has a higher weight in the fusion process; when the gating value is close to -1, the influence of the feature is suppressed, and the key information of different modalities is fused. The multimodal weight map is similar to v n The attention vector is obtained by multiplication, and then it is concatenated into 1×1×( N × g ) size features, and then obtain multimodal fusion features through linear changes , the specific calculation is as follows: , ;in, head n Indicates n attention vector, is the linear transformation matrix.

[0088] S8, outputs interpretable results, including skin disease categories, shape information, texture information, and stage visualization graphs.

[0089] In addition, the HAM10000 dataset, Derm7pt dataset and ISIC2019 dataset are used to test the examples of the present invention. The confusion matrices of the test results of the HAM10000 dataset and Derm7pt dataset are as follows: Figure 6 As shown, the HAM10000 dataset is used to compare with other CI-Net, Eff2Net, ResNet50 and other models. The category accuracy is as follows Figure 7 The overall performance is shown in Table 1, where the best value is indicated in bold:

[0090] Table 1

[0091]

[0092] In summary, it can be seen that the accuracy of DB-FSCLNet can be stabilized at around 95%, which is higher than that of other models, and its performance on small sample categories BCC, AKIEC, VASC, and DF is better than that of other models.

[0093] At the same time, ablation experiments were conducted on each stage module of DB-FSCLNet, and skin disease recognition experiments were conducted based on ResNet50, IRA-UNet+ResNet50, IRA-UNet+CGRNet and the complete DB-FSCLNet model, and the accuracy of each category was recorded. The detailed results are shown in Table 2, where bold indicates the optimal value on the HAM10000 dataset, and bold plus underline indicates the optimal value on the ISIC2019 dataset:

[0094] Table 2

[0095]

[0096] It can be seen that the recognition accuracy of the complete DB-FSCLNet for small samples is better than the combination of any modules.

[0097] Finally, the segmentation effect comparison of the IRA-UNet segmentation module is visualized. Figure 8 The texture information of the SC-GLCM texture feature extraction module is shown in Fig. 9 As shown, the information display of the interpretable result output is shown as Fig.10 As shown in these experimental results, it can be found that a deep interpretable skin disease recognition method based on small sample learning in an embodiment of the present invention can well realize the type recognition and interpretable output of small sample skin data.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A deep interpretable skin disease recognition method based on small sample learning, characterized in that: The following steps are involved: S1. Randomly extract two original skin disease images of the same or different categories from the skin disease image dataset and input them into the dual-branch small sample contrastive learning network DB-FSCLNet with the same structure respectively; S2, using the IRA-UNet segmentation module to segment the lesion area of ​​the original skin disease image and generate lesion segmentation masks of two corresponding branches; S3, inputting the lesion segmentation mask into the morphological positioning module, extracting shape information from the lesion segmentation mask using a geometric morphological feature extraction method, determining the minimum square frame of the lesion area, cropping and scaling it, and obtaining the shape information of the corresponding branch and the lesion image; S4. In the MC-GLCM texture extraction module, the texture information of the two branch lesion images at different scales is calculated using the multi-scale gray-level co-occurrence matrix, and the feature variance weighted strategy is used for weighted fusion to obtain the texture information; S5, using the CGRNet feature extraction module to extract the shallow features and deep features of the lesion image obtained in S3, and fusing the shallow features and deep features through multiple convolutional gated recurrent units to obtain the shallow and deep features of the two branches respectively; S6. Using the contrastive learning module, the contrast loss between the deep and shallow features of the two branches is calculated to optimize the CGRNet network parameters and enhance the feature representation ability of small sample data; S7. Use the CMFF fusion module to perform multimodal fusion of shape information, texture information and depth features to classify and identify skin diseases; S8. Output interpretable results, including skin disease categories, shape information, texture information, and stage visualization maps.

2. A deep interpretable skin disease recognition method based on small sample learning according to claim 1, characterized in that: In step S1, the dual-branch small sample contrast learning network is combined with the actual diagnosis logic of professional doctors, including the following stages: The focus and magnification stage simulates the doctor's focus on the lesion image and the lesion area during the diagnosis process. The lesion area of ​​the original skin disease image is segmented through the IRA-UNet segmentation module to generate a lesion segmentation mask. The shape information of the lesion segmentation mask is extracted using the morphological localization network, and the minimum square frame including the lesion area is cropped and scaled. The feature extraction and comparison stage simulates the doctor's observation of the lesion area and the comparison of different lesions during the diagnosis process. The MC-GLCM texture extraction network is used to extract multi-scale texture information, and the depth and shallow features of the lesion image are obtained through the CGRNet feature extraction module. The contrast loss between the depth and shallow features of the two branches is calculated through the contrast learning module to optimize the CGRNet network parameters. The fusion recognition stage simulates the doctor's diagnosis process by integrating image information and clinical information to make diagnostic decisions. The CMFF fusion module is used to perform multimodal fusion of shape information, texture information metadata information, and image information of depth and shallowness features for classification decisions and output of interpretable results.

3. The method for deeply interpretable skin disease recognition based on small sample learning according to claim 1, characterized in that: In step S2, the IRA-UNet segmentation module segments the lesion area of ​​the original skin disease image to obtain lesion segmentation masks of two branches, including the following steps: S2.1, the encoder part uses the inverted residual structure to perform dimension increase, feature extraction, and dimension reduction operations on the feature map of each layer input in sequence. Dimension increase improves the feature representation capability, dimension reduction reduces the computational overhead, and extracts rich semantic information in the high-dimensional space. The calculation process is expressed as: in, X , Y Represent the input feature map and output feature map respectively, Represents a 1×1 standard convolution, which is used to increase the spatial dimension; Indicates that the convolution kernel is 3×3 channel-by-channel convolution, which is used for feature extraction; Indicates that the convolution kernel is 1×1 point-by-point convolution, which is used to reduce the spatial dimension; S2.2, the decoder part uses deconvolution operation for upsampling, and learns the weight parameters of the convolution kernel through back propagation. The deconvolution operation accurately restores the original spatial structure in the layer-by-layer reconstruction process and improves the boundary details of the lesion area; S2.3, add channel space attention block CSA to the jump connection between the encoder of each layer and the decoder of the corresponding layer to transfer features, calculate feature weights from the channel dimension and spatial dimension respectively, perform weighted fusion processing on the input feature map, focus on key areas and improve segmentation effect. CSA contains channel attention block and spatial attention block. The channel attention block uses spatial average pooling and spatial maximum pooling to integrate the semantic information of each channel. The channel attention weight map is obtained through the fully connected layer and then combined with the input feature map. F Perform pixel-level multiplication to obtain the features enhanced by the channel attention block The spatial attention block uses channel average pooling and channel maximum pooling to model the spatial relationship, strengthens the element connection in the feature through the convolution layer, obtains the spatial attention weight map, and then combines it with the input feature F Perform pixel-level multiplication to obtain the features enhanced by the spatial attention block , in order to summarize the feature information of the channel attention block and the spatial attention block, and and F Perform weighted fusion to obtain , the calculation formula is as follows: , , ; where ⊙ represents element-by-element multiplication, W 1 and b 1 is the weight and bias of the fully connected layer, σ is the sigmoid activation function, [;] represents the concatenation operation, represents spatial average pooling, Represents the maximum spatial pooling, Conv 7×7 represents a convolution operation with a convolution kernel of 7×7. represents channel average pooling, represents channel maximum pooling, represents the weighted fusion feature, α and β are two learnable weighting factors with an initial value of 0, which are used to adjust the weights of channel attention features and spatial attention features.

4. The method for deeply interpretable skin disease recognition based on small sample learning according to claim 1, characterized in that: In step S3, the shape information is extracted from the lesion segmentation mask using the morphological localization module, and the minimum square bounding box of the lesion area is determined for cropping and scaling, including the following steps: S3.1, calculating the lesion pixel area, pixel perimeter, and lesion ratio through the lesion segmentation mask, wherein the lesion segmentation mask M is a binary image, i and j Represents a pixel, M ( i , j ) = 1 indicates a diseased pixel, M ( i , j ) = 0 indicates background pixels, S ={( i , j )| M ( i , j )=1} represents the pixel set of the lesion area, represents the pixel area of ​​the lesion area, represents the pixel perimeter of the lesion area, where Indicates that the boundary pixels of the lesion area are obtained by edge detection algorithm. The lesion ratio is shown by calculating the pixel area of ​​the lesion area A Total pixel area of ​​the image A total The ratio of S3.2, calculate the multi-angle symmetry of the lesion area, and mirror-flip the lesion area according to the given flip axes at 0°, 45°, 90° and 135°. The areas of the lesion area after flipping are expressed as A 0° , A 45° , A 90° and A 135° , the multi-angle symmetry calculation formula is as follows: ; S3.3, the regularity of the edge of the lesion area is evaluated using a comprehensive regularity evaluation index that combines roundness, edge smoothness, and edge complexity. The calculation formula is as follows: ;in, , and is the learnable weight coefficient, is the circularity, which is used to measure the similarity between the lesion area and the circle. The closer the circularity is to 1, the closer the lesion area is to a circle. is the edge complexity, which is used to measure the curvature change of the edge of the lesion area. is the curvature of each point on the edge, is the mean of the edge curvatures, O is the number of edge points; S3.4, calculate the minimum bounding rectangle of the lesion area, and the pixel coordinates corresponding to the upper, lower, left and right boundaries of the lesion area are , , , , then the side length of the bounding box can be expressed as ,by and Construct the minimum square bounding box coordinates for the lesion center as , crop it, remove irrelevant background and scale the cropped lesion area to the standard size of 224×224 to ensure the consistency of subsequent input.

5. The method for deeply interpretable skin disease recognition based on small sample learning according to claim 1, characterized in that: In step S4, the multi-scale texture information of the lesion area is extracted by the MC-GLCM texture extraction module, and weighted by the feature variance weighted fusion strategy to obtain the texture information, including the following steps: S4.1, quantize the pixel values ​​of the lesion area and discretize the original image values ​​into G grayscale levels, generate a grayscale image I , its image value i ∈{0,1,2,…, G −1}, the gray value of each pixel is determined by the pixel intensity of the original image, and the size is D s × D s The lesion area is sampled by a sliding window of s ∈{1,2,…, S } represents different scales, and the multi-scale gray-level co-occurrence matrix is ​​calculated. The calculation formula is as follows: in, Indicated in scale W s × W s and angle θ Lower gray value i and j The co-occurrence frequency, relative displacement ( dx ( θ ), dy ( θ )) means at a given angle θ The offset between adjacent pixels ，I s Indicated in D s × D s The image area at scale, Indicated in the image I Satisfied and The number of pixel pairs for the condition; S4.2, through the multi-scale gray level co-occurrence matrix, the texture information at different scales is calculated respectively. F s , multi-scale texture features are obtained through feature variance weighted fusion strategy F Texture , the calculation formula is as follows: in, F s Indicates s Texture feature vector at each scale, Var( F s ) represents the characteristic variance, is the weight coefficient assigned to each scale, given by s The ratio of the feature variance of texture features at a scale to the total feature variance of all scales reflects the variation range or significance of the texture information at that scale. The larger the variance, the more significant the texture information at that scale and the higher the weight.

6. The method for deeply interpretable skin disease recognition based on small sample learning according to claim 1, characterized in that: In step S5, the CGRNet feature extraction module realizes the fusion of shallow features and deep features to obtain the shallow and deep features of the two branches respectively, including the following steps: S5.1, CGRNet uses CNN architecture to extract shallow and deep features of lesion images, and uses multiple convolutional gated recurrent units to fuse features stage by stage, where convolution operations are used to replace the linear structure of the gated unit. The current stage features X i And the output features of the previous stage H t-1 Keep the feature dimensions consistent in the convolutional space and choose the gate r t , Fusion Door z t and the generated status , the calculation formula is as follows: in, t Represents the first t Stage, Conv t ( t =1,2,3…, T ) represents the convolution operation of different channels to ensure the dimensional consistency of the input features. σ represents the Sigmoid activation function, ∗ represents the pixel-level product; S5.2, convolutional gated recurrent unit by selecting gate r t Dynamically regulate the characteristics of the previous stage H t-1 The contribution ratio of shallow information can still play a role in deep feature fusion; through the fusion gate z t Selectively fuse the current input X t And the output features of the previous stage H i-1 , realize dynamic updating and filtering of information, discard unimportant historical information and retain key features, selectively filter and dynamically adjust features at different stages, and output state features H t The calculation formula is as follows: There is no previous output feature at the initial stage of the model H t-1 , then by introducing the zero-state initialization strategy, the hidden state is generated H 0=0.

7. The method for deeply interpretable skin disease recognition based on small sample learning according to claim 1, characterized in that: In step S6, the contrastive learning module optimizes the CGRNet network parameters under the condition of small samples by designing a contrast loss function between the deep and shallow features of the dual branches, including the following steps: S6.1, obtain the depth features of the two branches from the CGRNet feature extraction module f a and f b , the Euclidean distance between the two is , calculate the dynamic weighting factor based on the characteristic distance of the sample w , the calculation formula is as follows: in, d max is a normalization factor, which indicates the maximum value of the feature distance and ensures that the weight is in the range of [0, 1]. γ is the adjustment factor to control the difference in weights of the contribution of difficult-to-classify samples and easy-to-classify samples to the loss. ϵ is a smoothing factor to avoid numerical instability during normalization. u represents the sample category, u =1 means the same sample, u =0 indicates heterogeneous samples; S6.2, using adaptive threshold m Dynamically distinguish between similar and heterogeneous samples, m It is obtained by calculating the average feature distance of samples from different batches. The calculation formula is as follows: in, Indicates the average value of the characteristic distance of the current batch of samples, reflecting the dynamic changes of sample distribution; and is a hyperparameter used to adjust the flexibility and fixed offset of the threshold; S6.3, Calculate the contrast loss function by combining dynamic weighting factor and adaptive threshold L contrastive , the calculation formula is as follows: .

8. The method for deeply interpretable skin disease recognition based on small sample learning according to claim 1, characterized in that: In step S7, the CMFF fusion module performs multimodal fusion of shape information, texture information and depth features for skin disease recognition, including the following steps: S7.1, uses multi-head attention to transform image features of shape information, texture information metadata features and depth features, through N A linear transformation matrix transforms the metadata features into query vectors q n , using image features as key vectors k n Sum value vector v n ,capturing the association between image features and metadata features from multiple perspectives, q n , k n and v n The specific expressions are as follows: in, F meta Represents metadata characteristics, F image Represents image features, with sizes of 1×1× g , is the linear transformation matrix, is the linear transformation projection n Size; S7.2, using the improved cross-attention mechanism to calculate q n and k n The similarity of is obtained to obtain the multimodal weight map. The tanh function is used as the gating function. The tanh function maps the attention weight to the range of [−1,1]. When the gating value is close to 1, it indicates that the feature has a higher weight in the fusion process; when the gating value is close to -1, the influence of the feature is suppressed, and the key information of different modalities is fused. The multimodal weight map is similar to v n The attention vector is obtained by multiplication, and then it is concatenated into 1×1×( N × g ) size features, and then obtain multimodal fusion features through linear changes , the specific calculation is as follows: , ;in, head n Indicates n attention vector, is the linear transformation matrix.

Images