A skin disease image classification method based on three-branch feature fusion network

Abstract

The present invention discloses a dermoscopic image classification method based on a three-branch feature fusion network, the method comprising: obtaining dermoscopic image data to be classified; inputting the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage; fusing the global and local features at each stage through an adaptive hierarchical feature fusion module to obtain adaptive fusion feature information; classifying the fusion information at the last stage to obtain and output the classification result. The present invention generates augmented image data through a diffusion model guided by a classifier gradient to train the three-branch hierarchical multi-scale feature fusion network, and the adaptive hierarchical feature fusion module of the three-branch feature fusion network is used to fully fuse and mine the local and global features in the dermoscopic image to improve the accuracy of dermoscopic image classification.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a skin disease image classification method based on a three-branch feature fusion network. Background Art

[0002] Skin diseases are common and frequently occurring diseases, and their incidence is increasing due to factors such as environmental pollution, changes in lifestyle, and an aging population. With the rapid development of computer vision and artificial intelligence technologies, the study of computer-based automated classification of skin disease images has become increasingly important and challenging. Traditional skin disease image classification relies on experience and expertise, and has problems of subjectivity and uncertainty. Automated skin disease image classification methods can effectively solve these problems, and have the advantages of being fast, accurate, and repeatable. Therefore, the study of automated classification methods for skin disease images based on computer vision and artificial intelligence technologies has become a hot topic in recent years.

[0003] Although the application of deep learning in image classification, target detection and other fields has provided new ideas and methods for the automatic classification of skin disease images. However, for the classification of skin disease images, the performance of deep learning models still has a lot of room for improvement. First of all, the amount of data is insufficient. Deep learning algorithms require a large amount of training data to train models, but in some fields, there are relatively few public data sets available for training, and there are certain differences in the types, quantity and annotation quality of cases in each data set. Secondly, data annotation is difficult, and professional doctors are required to annotate images. However, due to the experience and subjective factors of the annotators, there are certain differences between different annotation results. Thirdly, the model lacks interpretability. Deep learning models are usually regarded as black box models, and their internal structure and decision-making process are difficult to explain, which makes it difficult for doctors to understand the judgment results of the model, thereby limiting its promotion and application.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a skin disease image classification method based on a three-branch feature fusion network is provided, aiming to solve the problems of insufficient data volume, difficulty in data labeling and lack of interpretability of the model in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a skin disease image classification method based on a three-branch feature fusion network, wherein the method comprises:

[0008] Acquiring dermoscopic image data to be classified;

[0009] Inputting the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, wherein the three-branch hierarchical multi-scale feature fusion network is trained by generating augmented image data through a diffusion model guided by a classifier gradient;

[0010] The global features and multi-scale local features of each stage are fused through an adaptive hierarchical feature fusion module to obtain adaptive fused feature information;

[0011] The adaptive fusion feature information is classified to obtain a classification result and output it.

[0012] In one implementation, the dermoscopic image data to be classified is input into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, including:

[0013] Constructing a three-branch hierarchical multi-scale feature fusion network, wherein the three-branch hierarchical multi-scale feature fusion network comprises a global feature extraction module, a local feature extraction module and an adaptive hierarchical feature fusion module;

[0014] The global feature extraction module obtains subtle features in the image and global features at each stage through bottleneck layers and dense connections;

[0015] The local feature extraction module obtains subtle features and local features in the image through multi-scale perception.

[0016] In one implementation, the global and multi-scale local features of each stage are fused by an adaptive hierarchical feature fusion module to obtain adaptive fused feature information, including:

[0017] The global features and multi-scale local features of each stage are respectively input into the adaptive hierarchical feature fusion module, and in the adaptive hierarchical feature fusion module, the global features and local features of each stage are fused by matrix addition to obtain adaptive fused feature information of each stage.

[0018] In one implementation, the global features and local features of each stage are fused by matrix addition to obtain adaptive fused feature information of each stage, including:

[0019] Through the adaptive hierarchical feature fusion module, local features, global features from each stage and adaptive fused feature information output from the previous level are adaptively fused according to the input features to obtain adaptive fused feature information of each stage, wherein the input features include the subtle features, global features of each stage and local features.

[0020] In one implementation, the three-branch hierarchical multi-scale feature fusion network generates augmented image data for training through a diffusion model guided by a classifier gradient, specifically including:

[0021] Construct a denoising diffusion probability model;

[0022] Using a classifier to guide the denoised diffusion probability model to obtain a diffusion model guided by a classifier gradient;

[0023] Inputting the dermoscopic image training data into the classifier-based gradient-guided diffusion model for diffusion enhancement to obtain enhanced data;

[0024] The three-branch hierarchical multi-scale feature fusion network to be trained is trained by the enhanced data to obtain the three-branch hierarchical multi-scale feature fusion network.

[0025] In one implementation, the classifying according to the adaptive fusion information, obtaining and outputting the classification result, includes:

[0026] The adaptive fusion feature information is identified and classified through the three-branch hierarchical multi-scale feature fusion network to obtain and output the classification result.

[0027] In a second aspect, an embodiment of the present invention further provides a skin disease image classification device based on a three-branch feature fusion network, wherein the device comprises:

[0028] An image acquisition module, used for acquiring dermatoscopic image data to be classified and detected;

[0029] A feature extraction module, used for inputting the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, wherein the three-branch hierarchical multi-scale feature fusion network is trained by generating augmented image data through a diffusion model guided by a classifier gradient;

[0030] A fusion module is used to fuse the global and multi-scale local features of each stage through an adaptive hierarchical feature fusion module to obtain adaptive fusion feature information;

[0031] The classification module is used to perform classification according to the fusion information, obtain classification results and output them.

[0032] In a third aspect, an embodiment of the present invention further provides an intelligent terminal, wherein the intelligent terminal includes a memory, a processor, and a skin disease image classification program based on a three-branch feature fusion network stored in the memory and executable on the processor, and when the processor executes the skin disease image classification program based on the three-branch feature fusion network, the steps of the skin disease image classification method based on the three-branch feature fusion network as described in any one of the above items are implemented.

[0033] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a skin disease image classification program based on a three-branch feature fusion network is stored on the computer-readable storage medium, and when the skin disease image classification program based on the three-branch feature fusion network is executed by a processor, the steps of the skin disease image classification method based on the three-branch feature fusion network as described in any one of the above items are implemented.

[0034] Beneficial effects: Compared with the prior art, the present invention provides a skin disease image classification method based on a three-branch feature fusion network. First, the original medical image data is obtained, the diffusion model is used to enhance the original medical image data, and then the sample quality is further improved through classifier gradient guidance. Secondly, a three-branch feature fusion network is used to capture global and local features, and a new multi-scale module is designed to obtain multi-dimensional image information, so that good results can be obtained without building a very deep network, and the problems of gradient disappearance and feature information loss are effectively avoided. Thirdly, we designed an adaptive hierarchical feature fusion module to fuse the global and local features of each stage to obtain hierarchical fusion features. The adaptive hierarchical feature fusion module contains spatial attention and channel attention, which can adaptively fuse the shortcuts of semantic information between various scale features of each stage. Finally, classification is performed according to the hierarchical fusion features to obtain classification results, realizing intelligent classification and improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flowchart of a skin disease image classification method based on a three-branch feature fusion network provided by an embodiment of the present invention.

[0037] Figure 2 It is a schematic diagram of a three-branch feature fusion network model provided by an embodiment of the present invention.

[0038] Figure 3 It is a schematic diagram of a gradient-guided diffusion model of a classifier provided in an embodiment of the present invention.

[0039] Figure 4 It is a schematic diagram of a multi-scale feature extraction module provided by an embodiment of the present invention.

[0040] Figure 5 It is a schematic diagram of an adaptive hierarchical feature fusion module provided in an embodiment of the present invention.

[0041] Figure 6 It is a principle block diagram of a skin disease image classification device based on a three-branch feature fusion network provided by an embodiment of the present invention.

[0042] Figure 7 It is a block diagram of the internal structure principle of the intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0046] Skin diseases affect 1.9 billion people worldwide. It is necessary to design a method for automatic diagnosis of skin diseases by automatically identifying skin disease images. Although convolutional neural networks (CNNs) can identify hidden features between different skin diseases, the impact of morphological lesion size variation and confusing visual imaging results on network accuracy has not been clearly addressed. At the same time, due to data imbalance, the large difference in the number of samples of different types of skin disease data leads to insufficient model generalization ability. To address this problem, we designed a three-branch attention fusion network based on diffusion enhancement to classify different skin diseases. The method consists of two components: 1) a diffusion generation model guided by the classifier gradient to enhance the original data to synthesize missing image information and generate multimodal diversity features and 2) a three-branch hierarchical multi-scale feature fusion network to multi-classify skin disease images. First, we use dense connections to extract global information in the global feature extraction block, and then the multi-scale feature fusion (MFF) module fuses local information at different scales. Finally, we designed an adaptive hierarchical feature fusion module (A-HFF). The adaptive hierarchical feature fusion (A-HFF) module contains spatial attention (SA), channel attention (CA), and adaptively fuses the global and local information of each stage and the multi-level semantic information of each stage.

[0047] Exemplary Methods

[0048] This embodiment provides a skin disease image classification method based on a three-branch feature fusion network. Figure 1 As shown, the method comprises the following steps:

[0049] Step S100, obtaining dermoscopic image data;

[0050] Specifically, skin diseases are common and frequently occurring diseases, and their incidence is increasing with the influence of factors such as environmental pollution, changes in lifestyle, and an aging population. The dermoscopic image data acquired in this embodiment may include various skin diseases such as telangiectasia, melanoma, vitiligo, disseminated xanthomas, and rosacea.

[0051] Step S200: input the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, wherein the three-branch hierarchical multi-scale feature fusion network generates expanded image data for training through a diffusion model guided by a classifier gradient.

[0052] In one implementation, step S200 in this embodiment includes the following steps:

[0053] Step S201, constructing a denoising diffusion probability model;

[0054] Step S202: using a classifier to guide the denoising diffusion probability model to obtain a classifier gradient guided diffusion model;

[0055] Specifically, the original medical imaging data may have insufficient data volume and difficulty in data labeling, which is the problem of poor interpretability of the model. Deep learning models are usually regarded as black box models, and their internal structure and decision-making process are difficult to explain, which makes it difficult for doctors to understand the judgment results of the model, thereby limiting its promotion and application in clinical applications. Finally, for the adaptability to new diseases, the deep learning model can only learn the lesion characteristics existing in the training set during training. For new and unseen diseases, the model may not be able to make accurate judgments and requires retraining or transfer learning. Therefore, in this embodiment, the dermoscopic image data will be diffusely enhanced to obtain enhanced data, thereby expanding the dermoscopic image data.

[0056] Specifically, Figure 3 As shown in the figure, the Diffusion Denoising Probabilistic Model (DDPM) learns an implicit probability distribution so that it can generate samples similar to the training data from random noise. The DDPM can be used for tasks such as image generation, denoising, and super-resolution. The DDPM includes two processes, the forward process and the reverse process. The forward process is used to add noise to the image. It starts from a simple noise distribution and goes through a series of diffusion processes to finally generate a complex image. In the forward process, the noise is gradually accumulated, and the details and structure of the image are gradually revealed. The reverse process is used for image denoising. It starts from a noisy image and goes through a series of denoising processes to finally obtain a more diverse image. In the reverse process, the noise is gradually eliminated, and the details and structure of the image are gradually restored.

[0057] Step S203: input the dermoscopic image training data into the classifier-based gradient-guided diffusion model for diffusion enhancement to obtain enhanced data.

[0058] Specifically, in addition to adopting a well-designed architecture, the generative adversarial network (GAN) for conditional image synthesis in this embodiment also makes extensive use of class labels. In order to better control the properties and features of the generated samples, thereby improving the quality and diversity of the generated samples, this embodiment designs a diffusion model guided by the classifier gradient. We have incorporated the class information into the adaptive group normalization layer and used the gradient of the classifier to guide the diffusion generator. Specifically, we trained a classifier For the noise image at time step t For classification, the classifier P can be any basic neural network. Here we choose the residual neural network (ResNet). Then use the gradient The sample generation process of the guided diffusion model tends to a given type label y. The original medical image data is input into the classifier gradient guided diffusion model for diffusion enhancement to obtain enhanced data, thereby expanding the original data set. This embodiment uses a generation model based on diffusion enhancement. Compared with the traditional generative adversarial network (GAN), the samples generated by the generation model based on diffusion enhancement have higher authenticity and diversity, and the interventionability of training is also improved.

[0059] Step S204: training the three-branch hierarchical multi-scale feature fusion network to be trained by using the enhanced data to obtain the three-branch hierarchical multi-scale feature fusion network.

[0060] In one implementation, the step S200 includes:

[0061] Step M200, constructing a three-branch hierarchical multi-scale feature fusion network, wherein the three-branch hierarchical multi-scale feature fusion network includes a global feature extraction module, a local feature extraction module and an adaptive hierarchical feature fusion module; the global feature extraction module obtains subtle features in the image and global features in each stage through a bottleneck layer and dense connections; the local feature extraction module obtains subtle features and local features in the image through multi-scale perception.

[0062] Specifically, in this embodiment, the skin disease image classification model based on the three-branch feature fusion network is used as a new dermoscopic medical image classification method. It uses a parallel structure to extract the global and local information of the dermoscopic image from the global and local feature blocks, and effectively obtains the local spatial and global feature information of images of different scales.

[0063] Specifically, the overall structure of the three-branch hierarchical multi-scale feature fusion network in this embodiment is as follows: Figure 2As shown in Table 1. The local feature extraction module, i.e., the local branch, is used to extract local features of the image, and the global feature extraction module, i.e., the global branch, is used to extract the global semantic representation of the image. The adaptive hierarchical feature fusion module, i.e., the fusion branch, is used for feature fusion. These three branches are composed of 4 stages and are used for feature extraction at different scales. The specific parameters are shown in Table 1.

[0064] Table 1. Specific parameters of the three-branch hierarchical multi-scale feature fusion network

[0065]

[0066] Among them, the multi-scale feature extraction module composed of the global feature extraction module and the local feature extraction module in this embodiment is as follows Figure 4 As shown in the figure. In order to improve the local information of skin images, perceiving information at multiple scales is necessary for accurate skin lesion detection. Since the use of group convolution instead of standard convolution blocks can add multi-scale information to the network architecture, this embodiment uses a hierarchy of convolution kernels on the feature volume slices of the middle layer to capture multi-scale information through the combination of different receptive fields. Specifically, this example segmented feature map volume (H, W, D represent height, bandwidth and number of channels respectively) is 4 slices , , , , each A corresponding segmentation output will be generated The final output Y is obtained by connecting the split multi-scale feature maps and adding them pixel-wise with the residual connection of the feature map X. Assume represents a convolution operation with a convolution kernel of 3, ⊕ represents a pixel-level addition operation, and the expression of the multi-scale feature map Y is as follows:

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] In one implementation, the global feature extraction module described in this embodiment performs downsampling in the average pooling layer.

[0073] Specifically, in order to promote the transmission of global information so that the network can better utilize the features extracted by the previous layer, this embodiment designs a global feature extraction module to achieve dense connections between different layers. Specifically, each convolutional layer in the global feature extraction block receives the inputs of all previous convolutional layers and connects them to the output of the current layer. In addition, the global feature extraction module we designed can also effectively alleviate the gradient vanishing problem and improve the training efficiency of the model.

[0074] The global feature extraction module of each stage is mainly composed of N bottleneck layers connected in series, and finally down-sampled after a convolution operation and an average pooling layer. The process formula is as follows:

[0075]

[0076]

[0077]

[0078] in, represents the output of the global feature extraction module in the previous stage, represents the output of the global feature extraction module at this stage, is the output of the i-th bottleneck layer, The Concat function represents the concatenation operation of feature channels. represents a convolution operation with a convolution kernel of 1, R is the RULE activation function, and LN is the normalization operation. Represents an average pooling operation.

[0079] Step S300, fusing the global features and multi-scale local features of each stage through an adaptive hierarchical feature fusion module to obtain adaptive fused feature information;

[0080] Specifically, since the self-attention in the global feature block can capture spatial and temporal global information, the hierarchical feature fusion (HFF) module feeds the incoming global features into the spatial attention (SA) mechanism to enhance local details and suppress irrelevant areas. This block contains spatial attention, channel attention, and shortcut connections to adaptively fuse the semantic information between features of different scales in each branch. Finally, the global features and local features are fused on the fusion path. To a certain extent, the problems of gradient vanishing, explosion, and network degradation are prevented, thereby effectively capturing the global and local feature information at each level.

[0081] Specifically, the adaptive hierarchical feature fusion module can adaptively fuse the local features from each stage, the global representation, and the semantic information after fusion of the previous level according to the input features to obtain hierarchical fusion features, such as Figure 5As shown. Among them, represents the feature matrix output by the global feature block, Represents the feature matrix output by the local feature block, represents the feature matrix output by the hierarchical feature fusion (HFF) module in the previous stage, Represents the feature matrix generated by the hierarchical feature fusion (HFF) module at this stage.

[0082] In one implementation, step S300 in this embodiment includes the following steps:

[0083] Step S301: inputting the global features and multi-scale local features of each stage into the adaptive hierarchical feature fusion module respectively, and fusing the global features and local features of each stage by matrix addition in the adaptive hierarchical feature fusion module to obtain adaptive fused feature information of each stage;

[0084] The adaptive hierarchical feature fusion module can adaptively fuse local features from different levels, global representations, and the adaptive fusion information output from the previous level according to the input features, such as Figure 5 As shown. Among them, represents the feature matrix output by the global feature block, Represents the feature matrix output by the multi-scale feature fusion module, represents the feature matrix output by AFF in the previous stage, Represents the feature matrix generated by AFF fusion at this stage.

[0085] Since the self-attention in the feature block can capture spatial and temporal information, the HFF block feeds the incoming global features into the spatial attention (SA) mechanism to enhance local details and suppress irrelevant areas. We use a multi-scale feature fusion module in the lower branch network structure to obtain multi-scale feature channels of image data, and we use channel attention (CA) to learn the importance weights of different channels. Finally, the global features and local features are fused on the fusion path. To a certain extent, the problems of gradient vanishing, explosion and network degradation are prevented, thereby effectively capturing the global and local feature information at each level. We let Expressed as The feature map output by spatial attention, Expressed as The feature map output by spatial attention is an adaptive feature map that integrates global information and local multi-scale features. The expression is as follows:

[0086]

[0087]

[0088]

[0089] in is the activation function. The reason why we don’t use the RELU activation function here is It has stronger smoothness, which can provide better continuity and gradient in the task of fusing spatial information and channel information, thus helping to better model complex data distribution. It is the feature matrix output by the adaptive fusion feature information of the previous stage after planar pooling. It is the intermediate result representation of feature fusion.

[0090] Step S400: classify the adaptive fusion feature information, obtain the classification result and output it.

[0091] In one implementation, step S400 in this embodiment includes the following steps:

[0092] Step S401: identify and classify the adaptive fusion feature information through the three-branch hierarchical multi-scale feature fusion network, obtain the classification result and output it.

[0093] Specifically, this embodiment uses four public HAM10000 datasets, ISIC (International Skin Imaging Collaboration) dataset, PH2 dataset and DermNet dataset to verify the network model.

[0094] HAM10000 is a dataset of 10,000 training images for the detection of pigmented skin lesions. The authors collected dermoscopic images from a diverse population, acquired and stored in different ways. The dataset was acquired over 20 years at the Medical University of Vienna, Austria, and in the private practice of Dr. Cliff Rosendahl, Australia. The dataset includes seven diagnostic categories: melanoma (11.1%), nevus (66.9%), basal cell carcinoma (5.1%), actinic keratosis or Bowen's disease (3.3%), benign keratosis (sun spots, seborrheic keratosis, or lichen planus keratosis, 11%), dermatofibroma (1.1%), and vascular lesions (1.4%). More than 50% of the lesions were confirmed by histopathology, and the remaining cases were confirmed by follow-up examination, expert consensus, or in vivo confocal microscopy.

[0095] The ISIC archive contains the world's largest collection of dermoscopic images. ISIC is an international academic-industry partnership initiated by the International Society for Digital Skin Imaging to "promote the use of digital skin imaging to help reduce melanoma mortality". This dataset is widely used for the development and evaluation of machine learning and deep learning algorithms, especially in skin lesion classification and diagnosis. The images in the ISIC dataset are provided by multiple institutions and individuals, including medical research institutions, hospitals, clinics, and individuals. These images have been annotated and classified by professional medical experts, covering benign tumors, malignant tumors, and other types of skin lesions.

[0096] The ISIC dataset provides multiple tasks and dataset versions, including:

[0097] ISIC 2016: This version of the dataset contains 900 high-quality images of skin lesions, including both benign and malignant skin lesions.

[0098] ISIC 2017: This version of the dataset contains 2,000 high-quality images of skin lesions, covering 7 common skin lesion types.

[0099] ISIC 2018: This version of the dataset contains more than 10,000 images of skin lesions, including benign tumors, malignant tumors, and other types of skin lesions.

[0100] The PH2 dataset is a publicly available melanoma image dataset that contains 200 high-quality skin images for melanoma classification and diagnosis research. Melanoma is a common skin cancer, but it is very challenging to diagnose in its early stages.

[0101] The images in the PH2 dataset were provided by medical experts and researchers, and they have been professionally annotated and classified into benign lesions and malignant melanomas. This dataset provides high-resolution skin images and related diagnostic results, which can be used to develop and evaluate machine learning and deep learning algorithms to help doctors diagnose melanoma more accurately.

[0102] The DermNet dataset is a public dataset containing more than 23,000 skin lesion images. It is a dataset created by New Zealand skin scientists and clinicians and contains many different types of skin lesion images.

[0103] The images in the DermNet dataset are provided by clinicians and skin scientists, and are professionally annotated and classified into categories including benign and malignant skin lesions, infections, and other types of skin lesions.

[0104] Through the verification of the data set, a trained three-branch feature fusion network is obtained. The medical image data to be classified is input into the trained three-branch feature fusion network for identification to obtain the classification result, which can indicate what skin disease the medical image data represents. Thus, intelligent classification is realized and the classification accuracy is improved.

[0105] Exemplary Devices

[0106] like Figure 6 As shown in , this embodiment also provides a skin disease image classification device based on a three-branch feature fusion network, the device comprising:

[0107] An image acquisition module 10 is used to acquire dermatoscopic image data to be classified and detected;

[0108] A feature extraction module 20 is used to input the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, wherein the three-branch hierarchical multi-scale feature fusion network generates augmented image data for training through a diffusion model guided by a classifier gradient;

[0109] A fusion module 30 is used to fuse the global and multi-scale local features of each stage through an adaptive hierarchical feature fusion module to obtain adaptive fusion feature information;

[0110] The classification module 40 is used to perform classification according to the fusion information, obtain classification results and output them.

[0111] In one implementation, the feature extraction module 20 in this embodiment includes:

[0112] The network construction unit is used to construct a three-branch hierarchical multi-scale feature fusion network, wherein the three-branch hierarchical multi-scale feature fusion network includes a global feature extraction module, a local feature extraction module and an adaptive hierarchical feature fusion module; the global feature extraction module obtains subtle features in the image and global features in each stage through a bottleneck layer and dense connections; the local feature extraction module obtains subtle features and local features in the image through multi-scale perception.

[0113] In one implementation, the fusion module 30 in this embodiment includes:

[0114] The fusion unit is used to input the global features and multi-scale local features of each stage into the adaptive hierarchical feature fusion module respectively, and in the adaptive hierarchical feature fusion module, the global features and local features of each stage are fused by matrix addition to obtain the adaptive fusion feature information of each stage.

[0115] In one implementation, the fusion unit described in this embodiment includes:

[0116] The adaptive fusion feature information acquisition subunit is used to input the global features and multi-scale local features of each stage into the adaptive hierarchical feature fusion module respectively, and in the adaptive hierarchical feature fusion module, the global features and local features of each stage are fused by matrix addition to obtain the adaptive fusion feature information of each stage.

[0117] In one implementation, the device described in this embodiment further includes:

[0118] A first feature extraction unit is used to input the dermoscopic image data to be classified and detected into the global feature extraction module, and obtain subtle features and four scale global features in the image through a bottleneck layer and dense connections;

[0119] The second feature extraction unit is used to input the dermoscopic image data to be classified and detected into the local feature extraction module, and obtain subtle features in the image and global features of four scales through multi-scale perception.

[0120] A weight calculation unit is used to calculate the importance weight of each channel in the three-branch feature fusion network through a channel attention module, and train the enhanced data according to the importance weight to obtain the local feature extraction branch.

[0121] A downsampling unit is used to downsample the global feature extraction branch in the average pooling layer.

[0122] A denoising diffusion probability model building unit, used for building a denoising diffusion probability model;

[0123] A diffusion model acquisition unit, used to use a classifier to guide the denoised diffusion probability model to obtain a diffusion model guided by a classifier gradient;

[0124] An enhanced data acquisition unit, used for inputting the dermoscopic image training data into the classifier-based gradient-guided diffusion model to perform diffusion enhancement to obtain enhanced data;

[0125] A training unit is used to train the three-branch hierarchical multi-scale feature fusion network to be trained through the enhanced data to obtain the three-branch hierarchical multi-scale feature fusion network.

[0126] In one implementation, the classification module 50 in this embodiment includes:

[0127] The classification unit is used to identify and classify the adaptive fusion feature information through the three-branch hierarchical multi-scale feature fusion network, obtain the classification result and output it.

[0128] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a skin disease image classification method based on a three-branch feature fusion network is implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the intelligent terminal is pre-set inside the intelligent terminal to detect the operating temperature of the internal device.

[0129] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In one embodiment, a smart terminal is provided, the smart terminal comprising a memory, a processor, and a skin disease image classification program based on a three-branch feature fusion network stored in the memory and executable on the processor. When the processor executes the skin disease image classification program based on the three-branch feature fusion network, the following operation instructions are implemented:

[0131] An image acquisition module, which acquires dermoscopic image data to be classified and detected;

[0132] A feature extraction module, which inputs the dermoscopic image data to be classified and detected into a three-branch feature fusion network to obtain global features and local features at multiple scales;

[0133] A fusion module, inputting the multi-scale global features and local features into an adaptive hierarchical feature fusion module to obtain adaptive fusion feature information;

[0134] The classification module obtains a classification result according to the adaptive fusion feature information.

[0135] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0136] In summary, the present invention discloses a skin disease image classification method based on a three-branch feature fusion network, the method comprising: acquiring original medical image data; performing diffusion enhancement on the original medical image data to obtain enhanced data; inputting the enhanced data into a three-branch feature fusion network for training to obtain global and local features at multiple scales; fusing the global features and local features of each scale through an adaptive hierarchical feature fusion module to obtain hierarchical fusion features; performing classification according to the hierarchical fusion features to obtain classification results. The present invention designs new local and global feature blocks, and fuses local and global representations of various semantic scales through an adaptive hierarchical feature fusion module. The fusion process of the present invention can fully exploit the depth and global local features of the lesion area in the medical image classification task, thereby improving the accuracy of classification.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A skin disease image classification method based on a three-branch feature fusion network, characterized in that: The method comprises: Acquiring dermoscopic image data to be classified; Inputting the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, wherein the three-branch hierarchical multi-scale feature fusion network is trained by generating augmented image data through a diffusion model guided by a classifier gradient; The global features and multi-scale local features of each stage are fused through an adaptive hierarchical feature fusion module to obtain adaptive fused feature information; Classifying the adaptive fusion feature information, obtaining a classification result and outputting it; The adaptive hierarchical feature fusion module is used to fuse the global and multi-scale local features of each stage to obtain adaptive fused feature information, including: The global features and multi-scale local features of each stage are respectively input into the adaptive hierarchical feature fusion module, and in the adaptive hierarchical feature fusion module, the global features and local features of each stage are fused by matrix addition to obtain adaptive fused feature information of each stage; The global features and local features of each stage are fused by matrix addition to obtain adaptive fusion feature information of each stage, including: Adaptively fusing local features, global features and adaptive fused feature information output from the previous layer from each stage according to input features through an adaptive hierarchical feature fusion module, to obtain adaptive fused feature information of each stage, wherein the input features include subtle features, global features and local features of each stage; The expression of the adaptive fusion feature information of each stage is: ， in, It is the feature matrix output by the adaptive fusion feature information of the previous stage after planar pooling, expressed as , is the intermediate result of feature fusion, expressed as , i is the stage number, is the feature matrix output by the global feature block, for The feature map output by spatial attention, is the feature matrix output by the local feature block, for The feature map output by spatial attention, is the activation function, is a convolution operation with a convolution kernel of 1, It is a convolution operation with a convolution kernel of 3; The dermoscopic image data to be classified is input into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, including: Constructing a three-branch hierarchical multi-scale feature fusion network, wherein the three-branch hierarchical multi-scale feature fusion network comprises a global feature extraction module, a local feature extraction module and an adaptive hierarchical feature fusion module; The global feature extraction module obtains subtle features in the image and global features at each stage through bottleneck layers and dense connections; The local feature extraction module obtains subtle features and local features in the image through multi-scale perception; The local feature extraction module obtains subtle features and local features in the image through multi-scale perception, specifically including: Use a hierarchy of convolutional kernels on the feature volume slices of the intermediate layers to segment the feature map volume For 4 slices , , , , each A corresponding segmentation output will be generated , where H, W, and D represent height, bandwidth, and number of channels, respectively; The multi-scale feature map Y is obtained by connecting the split multi-scale feature maps and adding them to the residual connection of the feature map X at the pixel level. The expression is as follows: ， ， ， ， ， in represents a convolution operation with a convolution kernel of 3, and ⊕ represents a pixel-level addition operation; The global feature extraction module performs downsampling in the average pooling layer, and the process is as follows: ， ， ， in, represents the output of the global feature extraction module in the previous stage, represents the output of the global feature extraction module at this stage, is the output of the i-th bottleneck layer, , the Concat function represents the concatenation operation of feature channels, represents a convolution operation with a convolution kernel of 1, R is the RULE activation function, and LN is the normalization operation. represents the average pooling operation; The three-branch hierarchical multi-scale feature fusion network generates augmented image data for training through a diffusion model guided by a classifier gradient, specifically including: Construct a denoising diffusion probability model; Using a classifier to guide the denoised diffusion probability model to obtain a diffusion model guided by a classifier gradient; Inputting the dermoscopic image training data into the classifier-based gradient-guided diffusion model for diffusion enhancement to obtain enhanced data; The three-branch hierarchical multi-scale feature fusion network to be trained is trained by the enhanced data to obtain the three-branch hierarchical multi-scale feature fusion network.

2. The skin disease image classification method based on a three-branch feature fusion network according to claim 1 is characterized in that: The step of classifying the adaptive fusion feature information to obtain and output a classification result includes: The adaptive fusion feature information is identified and classified through the three-branch hierarchical multi-scale feature fusion network to obtain and output the classification result.

3. A skin disease image classification device based on a three-branch feature fusion network, characterized in that: The device comprises: An image acquisition module, used for acquiring dermatoscopic image data to be classified and detected; A feature extraction module, used for inputting the dermoscopic image data to be classified into a three-branch hierarchical multi-scale feature fusion network to obtain global features and multi-scale local features at each stage, wherein the three-branch hierarchical multi-scale feature fusion network is trained by generating augmented image data through a diffusion model guided by a classifier gradient; A fusion module is used to fuse the global and multi-scale local features of each stage through an adaptive hierarchical feature fusion module to obtain adaptive fusion feature information; The classification module is used to classify according to the fusion information, obtain the classification results and output them; The fusion module includes: A fusion unit, used to input the global features and multi-scale local features of each stage into the adaptive hierarchical feature fusion module, and fuse the global features and local features of each stage by matrix addition in the adaptive hierarchical feature fusion module to obtain adaptive fused feature information of each stage; The global features and local features of each stage are fused by matrix addition to obtain adaptive fused feature information of each stage, including adaptively fusing local features, global features and adaptive fused feature information outputted from a previous layer from each stage according to input features through an adaptive hierarchical feature fusion module to obtain adaptive fused feature information of each stage, wherein the input features include subtle features, global features of each stage and local features; The expression of the adaptive fusion feature information of each stage is: ， in, It is the feature matrix output by the adaptive fusion feature information of the previous stage after planar pooling, expressed as , is the intermediate result of feature fusion, expressed as , i is the stage number, is the feature matrix output by the global feature block, for The feature map output by spatial attention, is the feature matrix output by the local feature block, for The feature map output by spatial attention, is the activation function, is a convolution operation with a convolution kernel of 1, It is a convolution operation with a convolution kernel of 3; The feature extraction module comprises: A network construction unit is used to construct a three-branch hierarchical multi-scale feature fusion network, wherein the three-branch hierarchical multi-scale feature fusion network includes a global feature extraction module, a local feature extraction module and an adaptive hierarchical feature fusion module; the global feature extraction module obtains subtle features in the image and global features at each stage through a bottleneck layer and dense connections; the local feature extraction module obtains subtle features and local features in the image through multi-scale perception; The local feature extraction module obtains subtle features and local features in the image through multi-scale perception, including: Use a hierarchy of convolutional kernels on the feature volume slices of the intermediate layers to segment the feature map volume For 4 slices , , , , each A corresponding segmentation output will be generated , where H, W, and D represent height, bandwidth, and number of channels, respectively; The multi-scale feature map Y is obtained by connecting the split multi-scale feature maps and adding them to the residual connection of the feature map X at the pixel level. The expression is as follows: ， ， ， ， ， in represents a convolution operation with a convolution kernel of 3, and ⊕ represents a pixel-level addition operation; The global feature extraction module performs downsampling in the average pooling layer, and the process is as follows: ， ， ， in, represents the output of the global feature extraction module in the previous stage, represents the output of the global feature extraction module at this stage, is the output of the i-th bottleneck layer, , the Concat function represents the concatenation operation of feature channels, represents a convolution operation with a convolution kernel of 1, R is the RULE activation function, and LN is the normalization operation. represents the average pooling operation; The device also includes: A denoising diffusion probability model building unit, used for building a denoising diffusion probability model; A diffusion model acquisition unit, used to use a classifier to guide the denoised diffusion probability model to obtain a diffusion model guided by a classifier gradient; An enhanced data acquisition unit, used for inputting the dermoscopic image training data into the classifier-based gradient-guided diffusion model to perform diffusion enhancement to obtain enhanced data; A training unit is used to train the three-branch hierarchical multi-scale feature fusion network to be trained through the enhanced data to obtain the three-branch hierarchical multi-scale feature fusion network.

4. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a skin disease image classification program based on a three-branch feature fusion network stored in the memory and run on the processor. When the processor executes the skin disease image classification program based on the three-branch feature fusion network, the steps of the skin disease image classification method based on the three-branch feature fusion network as described in any one of claims 1-2 are implemented.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a skin disease image classification program based on a three-branch feature fusion network. When the skin disease image classification program based on the three-branch feature fusion network is executed by a processor, the steps of the skin disease image classification method based on a three-branch feature fusion network as described in any one of claims 1-2 are implemented.

Images