Melanoma classification method and system based on cyclegan and bafcnn

Abstract

The application discloses a melanoma classification method and system based on CycleGAN and BAFCNN, and belongs to the technical field of artificial intelligence. The technical problem to be solved by the application is how to reasonably extract the lesion part of an original skin image and the normal skin tissue around the lesion, and improve the accuracy of melanoma classification. The technical scheme adopted is as follows: the method is used for balancing a data set by synthesizing representative insufficient samples in the data set through a conditional image, and data features are extracted by using a basic auxiliary fusion neural network, and the extracted data features are mapped to an output space for melanoma processing and classification; specifically, the conditional image is synthesized by using a cycle-consistent CycleGAN network; patch images are extracted by using a pixel block difference patch strategy; data is preprocessed; features are extracted by using a basic auxiliary fusion neural network to extract features of patch images and original skin images, and information of a skin lesion image is acquired; features are fused; and indexes are evaluated.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a melanoma classification method and system based on CycleGAN and BAFCNN. Background Technology

[0002] Skin cancer is a common cancer worldwide. Skin cancer includes squamous cell carcinoma, basal cell carcinoma, and melanoma. Melanoma accounts for a smaller proportion of skin cancers than other types, but it is more likely to invade nearby tissues and spread throughout the body. Statistics show that if melanoma is diagnosed early, the 5-year survival rate exceeds 95%, while the 5-year survival rate for advanced melanoma is less than 15%. Doctors typically use dermoscopy to diagnose melanoma. This non-invasive imaging technique provides magnified images of specific areas of the skin at a relatively high resolution, offering clear information about the lesion and helping doctors identify it. Although dermoscopy has proven successful in detecting skin lesions, it only achieves average sensitivity in diagnosing melanoma. Therefore, applying other classification methods is essential to help dermatologists more accurately and efficiently differentiate melanoma.

[0003] In recent years, deep learning-based classification methods have dominated the field of melanoma diagnosis, especially solutions based on deep convolutional neural networks, which have achieved significant performance improvements in melanoma diagnosis. However, improving the accuracy of melanoma diagnosis remains a significant challenge due to the following five factors:

[0004] (i) The number of images in this field is relatively small: typically, skin lesion datasets contain only a few thousand or even a few hundred images of skin lesions, which presents challenges in acquiring image data and ensuring accurate annotation. The lack of training samples limits the success of neural network methods in this field.

[0005] (ii) The ratio of the number of target categories to the number of non-target categories in the same dataset is vastly different: As shown in Table 1, most of the images in the dataset are non-melanoma images, which will easily lead to overfitting during the training process.

[0006] Table 1: Number of each lesion type in the ISIC2016 dataset. Row A represents the statistics of the original data in the ISIC2016 dataset. Row B represents the result of conditional image synthesis; Row C represents the number of each category input to the training set during training, which includes fused conditional synthesized images and original training images with the data volume copied to 5 times the original data volume.

[0007]

[0008] (iii) There is a large intra-class difference and a small inter-class difference between melanoma and non-melanoma. As shown in (e, f, g, h) of FIG. 8, skin lesion images belonging to the same class have significant differences in color, shape, and other features, and skin lesion images belonging to different classes have high similarity. It is still a challenge for even a doctor with professional knowledge to correctly classify under the premise of such differences. Figure 1 (iv) Previous methods of using machine learning to classify skin lesion images generally lack effective semantic focusing ability on the lesion area. The lesion area in the skin lesion image only accounts for a small part of it, and most of it is normal skin tissue. As shown in FIG. 9, these tissues are not only irrelevant to melanoma classification, but also can interfere with lesion classification. Although deep learning methods are widely used in skin lesion analysis, few studies have explained which part of the image the model focuses on meaningful semantic information.

[0009] Figure 1 (v) Single deep convolutional neural network has weak generalization ability. Ensemble Learning (EL) or increasing external data can be used to solve this difficulty. Using external data is the simplest way to solve this problem, and even many DCNN (deep convolutional network) methods use external data to improve classification performance, but in most medical fields, a large number of labeled images are not available, and using different external data as training data will affect the fairness of comparison between different methods. Therefore, using EL method becomes the key to solve this problem. EL method is to combine different output results with multiple rules to obtain better generalization performance than a single classifier. The researchers believe that different DCNNs can extract different semantic information from skin lesion images, and EL-based methods usually combine different powerful DCNN structures.

[0010] Therefore, how to reasonably extract the lesion part of the original skin image and the normal skin tissue around the lesion to improve the accuracy of melanoma classification is a technical problem to be solved. SUMMARY

[0011] The technical task of the present application is to provide a melanoma classification method and system based on CycleGAN and BAFCNN, to solve the problem of how to reasonably extract the lesion part of the original skin image and the normal skin tissue around the lesion to improve the accuracy of melanoma classification.

[0012] The technical task of the present application is to provide a melanoma classification method and system based on CycleGAN and BAFCNN, to solve the problem of how to reasonably extract the lesion part of the original skin image and the normal skin tissue around the lesion to improve the accuracy of melanoma classification.

[0013] ​The technical objective of this invention is achieved as follows: a melanoma classification method based on CycleGAN and BAFCNN. This method balances the dataset by using underrepresented samples from a conditional image synthesis dataset, and extracts data features using a basic auxiliary fusion neural network (BAFCNN network). The extracted data features are then mapped onto the output space for melanoma processing and classification. Specifically:

[0014] Conditional image synthesis: Conditional images are synthesized using a Cyclic Consistent CycleGAN (Adversarial Network);

[0015] Patch image extraction: The patch image is extracted using the Pixel Block Difference Patch Strategy (PBD) slicing algorithm.

[0016] Data preprocessing: The size of the original skin image is adjusted to 224x224, the size of the patch image is adjusted to 128x128, and the patch images are upsampled by 8x and 4x respectively, and then the alumniation operation is performed to compensate for the problem of insufficient data volume;

[0017] Feature extraction: Features of the patch image and the original skin image are extracted using a Basic Auxiliary Fusion Neural Network (BAFCNN network) to obtain information about the skin lesion image;

[0018] Feature fusion: A weight fusion strategy is used to fuse and classify the features output by the basic auxiliary fusion neural network (BAFCNN network);

[0019] Indicator Evaluation: Different evaluation indicators were used to evaluate the melanoma binary classification prediction results.

[0020] As a preferred embodiment, conditional image synthesis is performed as follows:

[0021] Non-melanoma images and melanoma images in the dataset are defined as the X domain and Y domain of the CycleGAN network, respectively.

[0022] Non-melanoma images in the X domain and melanoma images in the Y domain are fed into a pair of game-theoretic networks for training; the CycleGAN network includes a generator G. XY Generator G YX Discriminator D XY and Discriminator D YX ;

[0023] Generator G XY Generator G is used to synthesize a sample Y' from an X-domain image that is as similar as possible to a sample from the Y-domain image; YXX' domain feature map used to conditionally synthesize X domain samples using Y domain images as much as possible;

[0024] Discriminator D XY Used to score the features of samples in the Y domain and sample Y'; discriminator D YX Used to score X-domain samples and X'-domain feature maps;

[0025] The final loss is obtained by summing the feature scores of the Y-domain samples and sample Y', and the feature map scores of the X-domain samples and X'.

[0026] The generator G is updated appropriately based on the evaluation results, and finally an equilibrium state is reached through game theory.

[0027] As a preferred method, the patch image is extracted as follows:

[0028] Divide the original skin image into T pixel blocks on both central axes;

[0029] Calculate the sum of pixels and values ​​in each pixel block;

[0030] Obtain the difference between two adjacent pixel blocks and determine whether the difference between the adjacent pixel blocks is greater than a threshold.

[0031] If so, it means that a significant color change has occurred in two adjacent pixel blocks, that is, the lesion boundary of the skin lesion image occurs between these two adjacent pixel blocks;

[0032] Based on the lesion boundary of the skin lesion image, the position far away from the lesion tissue in the pixel block with a difference greater than the threshold is selected as the cutting line of the original skin image;

[0033] The cutting lines of the original skin image are obtained as follows:

[0034] The original skin image is divided into T blocks along its two central axes, denoted as t1, t2, ..., t3. n ;

[0035] Calculate the difference between the sum of pixels in every two adjacent blocks;

[0036] Starting from both ends of the central axis, find pixels with a difference greater than a threshold as the boundary lines of the sliced ​​image, using the following formula:

[0037]

[0038]

[0039]

[0040]

[0041] Wherein, L, R, U, D represent the distance of the target patch image from the left edge, the right edge, the upper edge and the lower edge of the original skin image respectively; W, H represent the width and height of the original skin image respectively; T represents the number of grids, the larger the T is, the more refined the patch image obtained is; T i represents the i-th pixel block; S(T i ) represents the sum value of the i-th pixel block; B represents the i value of the first or last pixel block whose difference value between two pixel blocks is greater than the threshold value;

[0042] When the number of pixel features of the slice image is less than one fourth of the original skin image, a geometric slice patch strategy (GS strategy) is used to make up for the defects of the sample.

[0043] As a preferred, the basic auxiliary fusion neural network (BAFCNN network) comprises a basic neural network (BCNN), an auxiliary neural network (ACNN) and a fusion neural network (FCNN);

[0044] The basic neural network (BCNN) is used to extract the features of the patch image;

[0045] The auxiliary neural network (ACNN) comprises an ACNN_WI network and an ACNN_O network; the ACNN_WI network is used to input the corresponding global image; the ACNN_O network is used to input the patch image, and the intermediate layer feature matrix of the basic neural network (BCNN) and the intermediate layer feature matrix of the ACNN_WI network are fused through a weighted integration strategy;

[0046] The fusion neural network (FCNN) is used to obtain the local fine-grained information and the global information of the lesion image and output the classification result;

[0047] The feature extraction is specifically as follows:

[0048] The patch image is 4 times up-sampled through the ACNN_O network; specifically, the ACNN_O network comprises seven convolution network layers, BatchNorm2d layers and an activation function, the patch image is up-sampled and then sent into a network comprising seven convolution layers to obtain an output vector;

[0049] The output vector is sent into the ACNN_WI network; wherein, the ACNN_WI network comprises eight convolution network layers, BatchNorm2d layers and an activation function;

[0050] The output of the intermediate layer of the basic neural network (BCNN) is input into the ACNN_WI network as a kind of global guidance, and the final output result is obtained in combination with a weighted integration strategy, and the formula of the weighted integration strategy is as follows:

[0051]

[0052] wherein F = (f1, f2,... fn) represents the intermediate feature of the base neural network (BCNN); G = (g1, g2,... gn) represents the local feature of the auxiliary neural network (ACNN); f i represents the feature vector of the i-th spatial position; g i represents the feature vector of the i-th spatial position; represents the concatenation operation of two feature vectors; Θ represents a complex calculation method, F and G are multiplied pixel by pixel with G and F respectively, and then the result is square rooted.

[0053] As a preferred, the fusion strategy is to change the extracted feature information, and then perform pixel-level fusion operation on the three two-dimensional feature vectors, and then send them into the network block;

[0054] The fusion strategy adopts weight feature fusion (WFS), which recombines the output vectors of the three feature extraction according to three weight values, and sends the result into a neural network composed of two network blocks; wherein each network block is composed of a fully connected layer, BatchNorm1d and an activation layer, and the formula is as follows:

[0055]

[0056] wherein RB = (rb1, rb2,... rb n ) ;

[0057] RAWI = (rawi1, rawi2,... rawi n ) ;

[0058] RAO = (rao1, rao2,... rao n ) ;

[0059] wherein RB, RAWI and RAO represent the feature values of the output results of the base neural network (BCNN), ACNN_WI network and ACNN_O network after dimensionality reduction operation respectively; W1, W2 and W3 all represent weight values; represents the vector concatenation operation; represents the pixel-level vector value addition operation; Π xi represents i fully connected layers.

[0060] More preferably, the index evaluation is as follows:

[0061] Different evaluation indicators are used to evaluate the melanoma binary classification prediction results, and the accuracy, precision, recall, F1_score and AUC indicators are used to avoid the unfairness of evaluation by only one indicator;

[0062] Among them, the evaluation index accuracy is used for judgment, and the formula is as follows:

[0063]

[0064] The purpose of evaluating the accuracy is to detect the ratio of the number of correctly classified positive examples to the total number of samples; in order to improve the credibility, other evaluation standards are also proposed; Accuracy represents the accuracy of the index; TP represents the number of people with normal labels and predictions; FN represents the number of people with normal labels and predictions; FP represents the number of people with normal labels and predictions; TN represents the number of people with normal labels and predictions;

[0065] Precision reflects the proportion of the number of predicted positive samples to the number of positive examples, and the purpose is to measure the "precision", and the formula is as follows:

[0066]

[0067] Recall reflects the proportion of the number of predicted positive samples to the number of all positive samples, and the purpose is to measure the "recall", and the formula is as follows:

[0068]

[0069] In order to evaluate the advantages and disadvantages of different algorithms, the concept of F1_score value is proposed based on precision and recall, which is used to evaluate the overall precision and recall, and the formula is as follows:

[0070]

[0071] AUC is used to summarize the information contained in the ROC curve, and the ROC curve draws the TPR and FPR at various thresholds, that is, the false positive rate; the larger the AUC value, the better the performance in distinguishing melanoma and non-melanoma images.

[0072] A melanoma classification system based on CycleGAN and BAFCNN, the system comprises,

[0073] The conditional image synthesis module is used to synthesize conditional images using the cycle-consistent CycleGAN network (adversarial network).

[0074] The patch image extraction module is used to extract patch images using the pixel block difference patch strategy (PBD strategy slicing algorithm).

[0075] The preprocessing module is used to adjust the size of the original skin image to 224x224, adjust the size of the patch image to 128x128, and upsample the patch image by 8x and 4x respectively, and then perform alumnimentation operations to compensate for the problem of insufficient data.

[0076] The feature extraction module is used to extract features from patch images and original samples using a basic auxiliary fusion neural network (BAFCNN network) and to obtain information about the skin lesion images;

[0077] The feature fusion module is used to fuse and classify the features output by the basic auxiliary fusion neural network using a weight fusion strategy.

[0078] The indicator evaluation module is used to evaluate the melanoma binary classification prediction results using different evaluation indicators.

[0079] Preferably, the conditional image synthesis module includes,

[0080] The image definition submodule is used to define non-melanoma images and melanoma images in the dataset as the X and Y domains of the CycleGAN network, respectively.

[0081] The image training submodule is used to feed non-melanoma images in the X domain and melanoma images in the Y domain into a pair of game-theoretic networks for training; wherein, the CycleGAN network includes a generator G. XY Generator G YX Discriminator DXY and Discriminator D YX ;

[0082] Generator G XY , used to synthesize a sample Y' from an X-domain image that is as similar as possible to a Y-domain sample;

[0083] Generator G YX , used to conditionally synthesize the X' domain feature map of X domain samples using the Y domain image as much as possible;

[0084] Discriminator D XY , used to score the features of samples in the Y domain and sample Y';

[0085] Discriminator D YX , used to score X domain samples and X' domain feature maps;

[0086] An adding submodule is configured to add the feature scores of the Y-domain samples and the Y' domain samples and the feature maps of the X-domain samples and the X' domain to obtain a final loss;

[0087] An updating submodule is configured to update the generator G according to the evaluation result, and finally reach a balance state through a game;

[0088] The patch image extraction module comprises,

[0089] A pixel block acquisition submodule is configured to divide two median axes of the original skin image into T pixel blocks evenly;

[0090] A pixel block and value acquisition submodule is configured to calculate the pixel and value in each pixel block;

[0091] A pixel block difference value acquisition and judgment submodule is configured to acquire the difference value of two adjacent pixel blocks and judge whether the difference value of the adjacent pixel blocks is greater than a threshold value

[0092] If yes, it indicates that a significant color change occurs in the two adjacent pixel blocks, i.e., the lesion boundary of the lesion image occurs between the two adjacent pixel blocks;

[0093] A selection submodule is configured to select, according to the lesion boundary of the lesion image, a position far away from the lesion tissue in the pixel block with the difference value greater than the threshold value as a cutting line of the original skin image;

[0094] The basic auxiliary fusion neural network (BAFCNN network) comprises a basic neural network (BCNN), an auxiliary neural network (ACNN) and a fusion neural network (FCNN);

[0095] The basic neural network (BCNN) is configured to extract features of the patch image;

[0096] The auxiliary neural network (ACNN) comprises an ACNN_WI network and an ACNN_O network; the ACNN_WI network is configured to input a corresponding global image; and the ACNN_O network is configured to input the patch image and fuse the intermediate layer feature matrix of the basic neural network (BCNN) and the intermediate layer feature matrix of the ACNN_WI network through a weighted integration strategy;

[0097] The fusion neural network (FCNN) is configured to obtain local fine-grained information and global information of the lesion image and output a classification result;

[0098] The feature extraction module comprises,

[0099] The upsampling submodule is used to upsample the patch image by a factor of 4 through the ACNN_O network. Specifically, the ACNN_O network consists of seven convolutional network layers, a BatchNorm2d layer, and an activation function. It upsamples the patch image and then feeds it into the network containing seven convolutional layers to obtain the output vector.

[0100] The input submodule is used to feed the output vector into the ACNN_WI network; the ACNN_WI network consists of eight convolutional network layers, a BatchNorm2d layer, and an activation function.

[0101] The output submodule uses the output of the intermediate layers of the base neural network (BCNN) as a global guide input to the ACNN_WI network, and combines it with a weighted ensemble strategy to obtain the final output result. The formula for the weighted ensemble strategy is as follows:

[0102]

[0103] Where F = (f1, f2, ..., fn) represents the intermediate features of the basic neural network (BCNN); G = (g1, g2, ..., gn) represents the local features of the auxiliary neural network (ACNN); f i The eigenvector representing the i-th spatial location; g i The feature vector representing the i-th spatial location; This indicates that two feature vectors are concatenated; Θ represents a complex calculation method in which F and G are multiplied pixel by pixel with G and F respectively, and then the square root of the result is taken.

[0104] The feature fusion module employs Weighted Feature Fusion (WFS), which reassembles the output vectors obtained from the three feature extractions according to three weight values, and feeds the result into a neural network composed of two network blocks for classification. Each network block consists of a fully connected layer, a BatchNorm1d layer, and an activation layer, as shown in the following formula:

[0105]

[0106] Where RB = (rb1, rb2, ..., rb n );

[0107] RAWI = (rawi1,rawi2,```,rawi n );

[0108] RAO = (rao1,rao2,```,rao n );

[0109] Where RB, RAWI, and RAO represent the feature values ​​of the output results of the basic neural network (BCNN), ACNN_WI network, and ACNN_O network after dimensionality reduction, respectively; W1, W2, and W3 all represent weight values. This represents a vector concatenation operation; Represents a pixel-level vector value addition operation; Π xi This represents i fully connected layers.

[0110] An electronic device includes: a memory and at least one processor;

[0111] The memory contains computer programs;

[0112] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the melanoma classification method based on CycleGAN and BAFCNN as described above.

[0113] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the melanoma classification method based on CycleGAN and BAFCNN as described above.

[0114] The melanoma classification method and system based on CycleGAN and BAFCNN of the present invention have the following advantages:

[0115] (I) This invention uses a conditional image synthesis method to synthesize samples with insufficient representativeness in the dataset to balance the dataset, then uses a basic auxiliary fusion neural network to extract data features, and finally maps the extracted features to the output space for melanoma processing and classification.

[0116] (II) This invention uses a patching algorithm to extract patch images from the original image. The extracted patch images and the original training images are then input into the BAFCNN network in pairs to further improve the ability to diagnose melanoma. The patch images are designed to include all lesion sites in the skin lesion image as well as some normal skin tissue around the lesion. This purposeful manual operation enables the neural network to correctly and quickly focus on meaningful information in the skin lesion image, thereby improving classification performance.

[0117] (iii) This invention can extract complementary discriminative information from dermoscopy images by adapting different weight values ​​in FCNN, thereby further improving the classification accuracy of melanoma;

[0118] (iv) The present invention can reasonably extract the lesion site and the normal skin tissue around the lesion from the original training image, so that the network model can more accurately focus on the meaningful site. Attached Figure Description

[0119] The application will be further described below with reference to the drawings.

[0120] Figure 1 is a schematic diagram of the original skin image, patch image, and conditionally synthesized image. Figure 1 Figure 1 is a schematic diagram of the original skin image, patch image, and conditionally synthesized image; wherein the a column, e column, f column, g column, and h column are original training images; the b column is a patch image of the a column; the c column is a conditionally synthesized image of the a column; the d column is a patch image of the c column image; the e column and f column are melanoma samples; the g column and h column are non-melanoma samples.

[0121] Figure 2 is a schematic diagram of the GS strategy. Figure 2 Figure 2 is a schematic diagram of the GS strategy; wherein the numbers represent the original image slice indexes, 1-4 represent four equal divisions of the original image, and 5-8 represent eight equal divisions of the original image, and the final five groups are combined to form patch images.

[0122] Figure 3 is a schematic diagram of the PBD strategy. Figure 3 Figure 3 is a schematic diagram of the PBD strategy; wherein the numbers represent the original image slice indexes, 1-4 represent four equal divisions of the original image, and 5-8 represent eight equal divisions of the original image, and the final five groups are combined to form patch images.

[0123] Figure 4 is a schematic diagram of the distribution of the pixel sizes of the original image and the slice image. Figure 4 Figure 4 is a schematic diagram of the distribution of the pixel sizes of the original image and the slice image; wherein the colors represent the number of images with corresponding width and height, and the reference scale is given on the right side.

[0124] Figure 5 is a schematic diagram of the patch images generated by applying the PBD strategy and the GS strategy, respectively. Figure 5 Figure 5 is a schematic diagram of the patch images generated by applying the PBD strategy and the GS strategy, respectively.

[0125] Figure 6 is a schematic diagram of the GS strategy. Figure 6 Figure 6 is a schematic diagram of the GS strategy; wherein the numbers represent the original image slice indexes, 1-4 represent four equal divisions of the original image, and 5-8 represent eight equal divisions of the original image, and the final five groups are combined to form patch images.

[0126] Figure 7 is a structural diagram of the BCNN and ACNN (ACNN_WI, ACNN_O). Figure 7 Figure 7 is a structural diagram of the BCNN and ACNN (ACNN_WI, ACNN_O); wherein the BCNN inputs the training image and performs pixel-level operation on the intermediate layer features and the intermediate layer features of the ACNN_WI network; it is worth noting that the up-sampling operation is performed before the training image is input into the ACNN network, highlighting the features of the patch image, and the BCNN network and the ACNN network output the same dimension result vector.

[0127] attached Figure 8 is a structural schematic diagram of the fusion strategy. DETAILED DESCRIPTION

[0128] The CycleGAN and BAFCNN-based melanoma classification method and system of the present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0129] Embodiment 1:

[0130] As shown in the attached Figure 2 , the present embodiment provides a CycleGAN and BAFCNN-based melanoma classification method, which is used to balance the data set by synthesizing representative insufficient samples in the data set through conditional image synthesis, and extract data features using a basic auxiliary fusion neural network (BAFCNN network) to map the extracted data features to the output space for melanoma processing and classification; the specific steps are as follows:

[0131] S1, conditional image synthesis: using a cycle-consistent CycleGAN network (adversarial network) to synthesize conditional images;

[0132] S2, extract patch images: using a pixel block difference patch strategy (Pixel block difference patch strategy, PBD strategy slicing algorithm) to extract patch images;

[0133] S3, data preprocessing, specifically as follows:

[0134] Experiments were conducted on the ISIC 2016 data set. The ISIC 2016 data set contains two categories of images, non-melanoma images and melanoma images, and the number of each category is shown in Table I (A). The conditional image synthesis method is used to generate melanoma images from non-melanoma images in the data set, as shown in columns c and d in the attached Figure 1 , and the generated images are fused into the original data set. The number of images of each category used finally is shown in Table I (C).

[0135] The size of the original skin image is adjusted to 224x224, the size of the patch image is adjusted to 128x128, and the patch image is 8 times and 4 times up-sampled, respectively, and then the albumentations operation is performed to make up for the problem of insufficient data quantity.

[0136] S4, feature extraction: using a basic auxiliary fusion neural network (BAFCNN network) to extract features of patch images and original samples, and obtaining information of the lesion image;

[0137] S5, feature fusion: using a weight fusion strategy to fuse and classify the features output by the auxiliary fusion neural network (BAFCNN network);

[0138] S6, index evaluation: using different evaluation indexes to evaluate the melanoma binary classification prediction results.

[0139] In the field of lesion classification, the number of images in the data set is relatively small, and the proportion of melanoma images is lower. A new method is adopted to solve this problem, that is, to use the inter-class difference of data distribution to complete the conditional image synthesis to solve the imbalance problem, and to learn the inter-class mapping and use the unpaired image to image conversion to synthesize representative insufficient class samples. In recent years, various image synthesis methods based on adversarial network have been used to solve the image synthesis problem, but most of these methods use a pair of mutually paired image samples for conditional image synthesis. Since this method is not suitable for melanoma image data sets, we use a cycle-consistent adversarial network (CycleGAN) to automatically train an image-to-image conversion model without paired examples. Since CycleGAN is only a framework, it contains a discriminator and a generator, and the generator uses a U-Net architecture, and the discriminator uses a PatchGAN architecture. The conditional image synthesis in step S1 of the embodiment is as follows:

[0140] S101, the non-melanoma images and melanoma images in the data set are defined as the X domain and Y domain of the CycleGAN network respectively;

[0141] S102, the non-melanoma images in the X domain and the melanoma images in the Y domain are respectively sent into a pair of game networks for training; wherein the CycleGAN network includes a generator G XY , a generator G YX , a discriminator D XY and a discriminator D YX ;

[0142] S103, the generator G XY is used to try to synthesize Y' samples similar to the Y domain samples from the X domain images; the generator G YX is used to conditionally synthesize X' domain feature maps of X domain samples using Y domain images;

[0143] S104, the discriminator D XY is used to score the Y domain samples and sample Y' features; the discriminator D YX is used to score the X domain samples and X' domain feature maps;

[0144] S105, the scores of the Y domain samples and sample Y' features and the scores of the X domain samples and X' domain feature maps are added to obtain the final loss.

[0145] S106, update the generator G appropriately according to the evaluation result, and finally reach a balanced state through the game, as shown in the accompanying Figure 3 .

[0146] Wherein, according to the characteristics of the lesion image data, the loss ratio is adjusted during loss calculation, so that the discriminator can better guide the generator to generate target condition synthesis image. Through experiments, the final adjustment of the two ratios to 8:2 will produce the best results.

[0147] Because the effective semantic information of the lesion image is gathered in the center position of the image, and only occupies a small part in the image, how to make the model effectively focus on the meaningful area is the research focus in this field. There is a new patch-based attention architecture in the prior art, which provides global context information between small, high-resolution patches. The results show that the proposed attention mechanism method is superior to the previous method, and the average sensitivity is improved by 7%. However, this method has many disadvantages. First, the segmentation network model is large and occupies resources, which is slow in training and is not conducive to the application of melanoma classification. Second, this method only retains the melanoma part after segmenting the image, without normal skin tissue information. It is believed that such lesion images ignore the transition information between normal skin and target classification area, which is not conducive to melanoma classification. In addition, there is a method for extracting effective semantic information from dermoscopy images. This method extracts images with effective semantic information from the original dermoscopy images as patch images. The method of extracting patch images cannot guarantee to contain all the lesion sites, and may contain too much normal skin tissue, with very uncertain factors.

[0148] Therefore, in order to effectively remove the interference factors in the lesion image, the patch image extraction in step S2 of the embodiment is as follows:

[0149] S201, divide the two center axes of the original skin image into T pixel blocks on average;

[0150] S202, calculate the sum of pixels in each pixel block;

[0151] S203, obtain the difference value of two adjacent pixel blocks, and judge whether the difference value of the adjacent pixel blocks is greater than the threshold

[0152] If yes, it means that there is a significant change in color between the two adjacent pixel blocks, that is, the lesion boundary of the lesion image occurs between the two adjacent pixel blocks;

[0153] S204, according to the lesion boundary of the lesion image, select the position away from the lesion tissue in the pixel block with a difference value greater than the threshold as the cutting line of the original skin image;

[0154] The cutting line acquisition of the original skin image in step S204 of the embodiment is specifically as follows:

[0155] S20401, divide the two center axes of the original skin image into T blocks averagely, respectively t1, t2, ··· t n ;

[0156] S20402, calculate the difference value of the pixel sum of each adjacent two blocks;

[0157] S20403, find the pixel point with a difference value greater than a threshold value as the boundary line of the slice image from the two ends of the center axis, and the formula is as follows:

[0158]

[0159]

[0160]

[0161]

[0162] Wherein, L, R, U, D represent the distance of the target patch image from the left edge, the right edge, the upper edge and the lower edge of the original skin image respectively; W, H represent the width and height of the original skin image respectively; T represents the number of grids, the larger the T, the more refined the patch image obtained; T i represents the ith pixel block; S(T i ) represents the sum value of the ith pixel block; B represents the i value of the first or last pixel block with a difference value greater than a threshold value between two pixel blocks;

[0163] When the number of pixel features of the slice image is less than one fourth of the original skin image, the geometric slice patch strategy (GS strategy) is used to make up for the defects of the sample.

[0164] In order to prove the effectiveness of the proposed PBD strategy in the experiment, the PBD strategy is compared with the geometric slice patch strategy (Geometric slice patch strategy, GS strategy). The extraction algorithm is to divide the two center axes of the original image into four equal parts, connect the four equal parts, and take the middle part of the image as the patch image, as shown in the accompanying Figure 4 .

[0165] Before the experiment, the patch images generated by the two patch strategies are statistically analyzed. The mask image of the lesion in the ISIC2016 dataset is used as the benchmark, and the operation process is all pixel-level operation.

[0166] First, observe the slice results, as shown in the attached image. Figure 5 As shown in the attached image. By comparing the original image with example images of patched images generated by PBD and GS strategies respectively, it was found that... Figure 5 In columns a, b, e, and f, the patch images generated by the PBD strategy not only completely extract the lesion area but also some surrounding normal skin tissue. However, the patch images generated by the GS strategy in columns a, e, and f can only extract a portion of the lesion image, resulting in the loss of a significant amount of meaningful semantic information. Comparing columns c and d, the patch images generated by the PBD strategy contain most of the lesion location but also a large amount of information about non-lesion areas, while the patch images generated by the GS strategy contain less information about non-lesion areas. We believe that both types of patch images can be successfully focused by the network. Furthermore, when the training images are affected by artifacts, as shown in column g, the PBD strategy ensures that the entire lesion area is included, while the GS strategy deletes some lesion areas. Obviously, in this case, the PBD strategy is more suitable for patch image extraction.

[0167] In addition to comparing the sliced ​​images, statistical analysis was also performed on the two types of images.

[0168] First, the lesion sites in the PBD patch images comprise 95.41% of the original image's lesion sites, and occupy 55.15% of the patch image area, while the lesion area in the original training image is only 27.0%. In the GS patch images, the lesion portion accounts for 27.87% of the original image's lesion portion and 27.83% of the patch image's lesion portion. These data comparisons demonstrate that the patch images extracted by the PBD strategy can contain the vast majority of the lesion locations in the original image and can remove some of the normal skin tissue from the original image. Based on the PBD operating mechanism, we consider the skin tissue contained in the slice images to be the tissue surrounding the lesion. Such images not only enable BAFCNN to find meaningful semantic information in the image faster and more accurately, but also largely remove the interference of normal skin tissue on BAFCNN.

[0169] Second, the PBD patch image accounts for 47.24% of the total original image. When the proportion of the slice image in the original image is less than 45%, the proportion of the lesion part in the slice image accounts for 94.72% of the lesion part in the original image. When the proportion is between 45% and 80%, the proportion accounts for 94.24%. When the proportion is greater than 80%, the proportion accounts for 99.32%. The patch image generated by the GS strategy accounts for 25% of the original image. The above data shows that the PBD strategy can accurately locate the lesion position even if it extracts a small part of the original image. The reason for the increase in the slice image is that the lesion area in the original image is large, not the algorithm itself. The patch we propose can intelligently extract the patch image according to the size of the lesion, accurately locate the lesion part of the original dermatoscope image, effectively reduce the size of the training image, and avoid missing the effective semantic information. As shown in the accompanying Figure 6

[0170] Third, the proportion of the lesion part in the slice image is also counted. It is found that when the proportion of the slice image in the original image is less than 45%, the lesion area accounts for about 51% of the slice image. When the proportion is between 45% and 80%, the lesion area accounts for about 63.88% of the slice image. When the proportion is greater than 80%, the lesion area in the slice image accounts for about 55.23% of the slice image. Thus, it is confirmed that the PBD strategy can keep the lesion and non-lesion area ratio in the slice image stable, avoid the non-lesion area in the dermatoscope image being too large, and make the training network pay too much attention to irrelevant positions, thereby obtaining better network performance. In summary, the PBD strategy is considered to have better performance.

[0171] It is assumed that the improvement of melanoma classification performance is not only due to the simultaneous extraction of global and local features in the BAFCNN network, but also due to the PBD strategy we proposed. In addition to the statistical analysis of the PBD strategy and the GS strategy in the previous text, the following comparative experiments are conducted to analyze the effect of the PBD strategy on the experimental results.

[0172] ​Firstly, only the BCNN patch image is trained, and the results are shown in Table 2 (BAFCNN). By comparing BAFCNN with Efficientnet-B6, it is found that PBCNN obtains better performance, can obtain higher AUC value and identify more non-melanoma. However, it will cause the prediction accuracy of the test set to decrease, and the reason for the decrease is that the patch image misses the global information of the original training image and part of the semantic information of the lesion site. Therefore, the ACNN and FCNN are added to the network model, that is, the BAFCNN network model, so as to effectively extract the global image and the effective semantic information of the patch image. Since the BAFCNN network has three inputs, five experiments are designed to verify the influence of different input methods on the results, and the BAFCNN experiment shows the best performance on the test set. Therefore, it can be proved that the BAFCNN network can effectively extract the effective semantic information of the patch image, and can also apply the weighted integration strategy to fuse the features of the patch image and the global image, so that the final classification result is greatly improved.

[0173] Table 2: Comparison of classification performance between different patch strategies and between different data volumes. The best result and the second best result are represented by “underline” and “black bold”, respectively. “*” indicates that ImageNet weight file is used for transfer learning.

[0174]

[0175] As shown in the accompanying Figure 7 , the basic auxiliary fusion neural network (BAFCNN network) in the embodiment includes a basic neural network (BCNN), an auxiliary neural network (ACNN) and a fusion neural network (FCNN);

[0176] The basic neural network (BCNN) is used to extract the features of the patch image;

[0177] The auxiliary neural network (ACNN) includes an ACNN_WI network and an ACNN_O network; the ACNN_WI network is used to input the corresponding global image; the ACNN_O network is used to input the patch image, and the weighted integration strategy is used to fuse the intermediate layer feature matrix of the basic neural network (BCNN) and the intermediate layer feature matrix of the ACNN_WI network;

[0178] The fusion neural network (FCNN) is used to obtain the local fine-grained information and global information of the lesion image and output the classification result;

[0179] The feature extraction in step S4 of the embodiment is specifically as follows:

[0180] S401, 4 times up-sampling the patch image through the ACNN_O network; specifically, the ACNN_O network includes seven convolutional network layers, BatchNorm2d layers and an activation function, the patch image is up-sampled and then sent into a network including seven convolutional layers to obtain an output vector;

[0181] S402, sending the output vector into the ACNN_WI network; wherein the ACNN_WI network includes eight convolutional network layers, BatchNorm2d layers and an activation function;

[0182] S403, the output of the intermediate layer of the base neural network (BCNN) is input into the ACNN_WI network as a kind of global guidance, and a final output result is obtained by combining a weighted integration strategy, and the formula of the weighted integration strategy is as follows:

[0183]

[0184] Wherein, F=(f1, f2,..., fn) represents the intermediate feature of the base neural network (BCNN); G=(g1, g2,..., gn) represents the local feature of the auxiliary neural network (ACNN); f i represents the feature vector of the i-th spatial position; g i represents the feature vector of the i-th spatial position; represents the splicing operation of the two feature vectors; Θ represents a complex calculation method, F and G are multiplied pixel by pixel with G and F respectively, and then the result is square rooted.

[0185] The fusion strategy in step S5 of the embodiment is to change the extracted feature information, and to perform pixel-level fusion operation on the three two-dimensional feature vectors, and then to send them into a network block;

[0186] As shown in the accompanying Figure 8 , the fusion strategy in step S5 of the embodiment adopts weight feature fusion (WFS), which recombines the output vectors obtained by extracting three features according to three weight values, and sends the result into a neural network composed of two network blocks for classification; wherein each network block is composed of a fully connected layer, BatchNorm1d and an activation layer, and the formula is as follows:

[0187]

[0188] Wherein, RB=(rb1, rb2,..., rb n );

[0189] RAWI=(rawi1, rawi2,..., rawi n );

[0190] RAO = (rao1, rao2,..., rao n );

[0191] Wherein, RB, RAWI and RAO represent the feature values of the output results of the base neural network (BCNN), ACNN WI network and ACNN O network after dimensionality reduction operation respectively; W1, W2, W3 all represent weight values; represents a vector splicing operation; represents a pixel-level vector value addition operation; Π xi represents i fully connected layers.

[0192] The index evaluation in step S6 of the embodiment is specifically as follows:

[0193] Different evaluation indexes are used to evaluate the melanoma binary classification prediction results, and the accuracy, precision, recall, F1_score and AUC indexes are used, so as to avoid the unfairness of only using one index for evaluation;

[0194] Wherein, the evaluation index accuracy (Accuracy) is used for evaluation, and the formula is as follows:

[0195]

[0196] The purpose of evaluating the accuracy is to detect the ratio of the number of correctly classified positive examples to the total number of samples; in order to improve the credibility, other evaluation standards are also proposed; Accuracy represents the accuracy of the index; TP represents the number of people whose label and prediction are both normal; FN represents the number of people whose label is normal and prediction is melanoma; FP represents the number of people whose label is melanoma and prediction is normal; TN represents the number of people whose label and prediction are both melanoma;

[0197] The precision (Precision) reflects the proportion of the number of predicted positive samples to the number of positive instances, and the purpose is to measure the "precision", and the formula is as follows:

[0198]

[0199] The recall (Recall) reflects the proportion of the number of predicted positive samples to the number of all positive samples, and the purpose is to measure the "recall", and the formula is as follows:

[0200]

[0201] In order to evaluate the advantages and disadvantages of different algorithms, the concept of F1_score value is proposed on the basis of precision and recall, and the overall evaluation of precision and recall is carried out, and the formula is as follows:

[0202]

[0203] AUC is used to summarize the information contained in the ROC curve, and the ROC curve plots the TPR and FPR at various thresholds, i.e. false positive rate; the larger the AUC value, the better the performance in distinguishing melanoma and non-melanoma images.

[0204] In this embodiment, the conditional image synthesis and patch picture extraction are offline operations, and then the picture is sent to the training network. The CycleGAN network training period is 500, the Adam optimizer with a batch size of 1 is used, the learning rate is 0.0002, and the training is 30 times. BAFCNN adopts ten-fold cross-validation, Adam optimizer with a learning rate of 0.0005, and BCEWithLogitsLoss loss function, for ACNN picture, the input size is 224x224x3, the input image size of BCNN is 128x128x3, and the picture is subjected to albumentations before being sent to BAFCNN. During the network training process, the weight file with high AUC value of the verification set is saved. The patch strategy in this embodiment uses a threshold of 30, Pytorch is used to build our network architecture, and a measurement function of evaluation index is provided, and the experiment is carried out on a platform configured with Intel(R) Xeon(R) Gold5118 CPU @ 2.30GHz and two RTX 2080Ti GPUs.

[0205] Embodiment 2:

[0206] The embodiment provides a melanoma classification system based on CycleGAN and BAFCNN, the system comprises,

[0207] The conditional image synthesis module is configured to synthesize a conditional image using a cycle-consistent CycleGAN network (adversarial network);

[0208] The patch image extraction module is configured to extract a patch image using a pixel block difference patch strategy (PBD strategy slicing algorithm);

[0209] The preprocessing module is configured to resize the original skin image to 224x224, resize the patch image to 128x128, and perform 8 times and 4 times up-sampling on the patch image respectively, and then perform an albumentations operation to make up for the problem of insufficient data amount.

[0210] The feature extraction module is configured to extract features of the patch image and the original sample by using the basic auxiliary fusion neural network, and obtain information of the lesion image.

[0211] The feature fusion module is configured to fuse and classify the features output by the basic auxiliary fusion neural network by using a weight fusion strategy.

[0212] The index evaluation module is configured to evaluate the melanoma binary classification prediction result by using different evaluation indexes.

[0213] The conditional image synthesis module in the embodiment comprises,

[0214] The image definition submodule is configured to define the non-melanoma images and the melanoma images in the data set as the X domain and the Y domain of the CycleGAN network respectively.

[0215] The image training submodule is configured to send the non-melanoma images in the X domain and the melanoma images in the Y domain into a pair of game networks for training respectively; wherein the CycleGAN network comprises a generator G XY , a generator G YX , a discriminator D XY and a discriminator D YX .

[0216] The generator G XY is configured to try to synthesize a sample Y' similar to the Y domain sample from the X domain image.

[0217] The generator G YX is configured to try to conditionally synthesize an X' domain feature map of the X domain sample from the Y domain image.

[0218] The discriminator D XY is configured to score the Y domain sample and the sample Y' feature.

[0219] The discriminator D YX is configured to score the X domain sample and the X' domain feature map.

[0220] The summation submodule is configured to perform a summation operation on the scores of the Y domain sample and the sample Y' feature and the scores of the X domain sample and the X' domain feature map to obtain a final loss.

[0221] The update submodule is configured to appropriately update the generator G according to the evaluation result, and finally reach a balanced state through the game.

[0222] The patch image extraction module in the embodiment comprises,

[0223] The pixel block obtaining submodule is configured to divide the two median axes of the original skin image into T pixel blocks evenly;

[0224] The pixel block and value obtaining submodule is configured to calculate the sum and value of pixels in each pixel block;

[0225] The pixel block difference obtaining and judging submodule is configured to obtain the difference between two adjacent pixel blocks and judge whether the difference between the two adjacent pixel blocks is greater than a threshold value

[0226] If yes, it indicates that there is a significant change in color between the two adjacent pixel blocks, that is, the lesion boundary of the lesion image occurs between the two adjacent pixel blocks;

[0227] The selecting submodule is configured to select, according to the lesion boundary of the lesion image, a position away from the lesion tissue in the pixel block with the difference greater than the threshold value as the cutting line of the original skin image.

[0228] The basic auxiliary fusion convolutional neural network (BAFCNN network) in the embodiment comprises a basic convolutional neural network (BCNN), an auxiliary convolutional neural network (ACNN) and a fusion convolutional neural network (FCNN);

[0229] The basic convolutional neural network (BCNN) is configured to extract features of the patch image;

[0230] The auxiliary convolutional neural network (ACNN) comprises an ACNN_WI network and an ACNN_O network; the ACNN_WI network is configured to input a corresponding global image; the ACNN_O network is configured to input the patch image and fuse the intermediate layer feature matrix of the basic convolutional neural network (BCNN) and the intermediate layer feature matrix of the ACNN_WI network through a weighted integration strategy;

[0231] The fusion convolutional neural network (FCNN) is configured to obtain local fine-grained information and global information of the lesion image and output a classification result.

[0232] The feature extraction module in the embodiment comprises,

[0233] The up-sampling submodule is configured to up-sample the patch image by 4 times through the ACNN_O network; specifically, the ACNN_O network comprises seven convolutional network layers, a BatchNorm2d layer and an activation function, the patch image is up-sampled and then input into a network comprising seven convolutional layers to obtain an output vector;

[0234] The input submodule is configured to input the output vector into the ACNN_WI network; wherein, the ACNN_WI network comprises eight convolutional network layers, a BatchNorm2d layer and an activation function;

[0235] The output submodule uses the output of the intermediate layers of the base neural network (BCNN) as a global guide input to the ACNN_WI network, and combines it with a weighted ensemble strategy to obtain the final output result. The formula for the weighted ensemble strategy is as follows:

[0236]

[0237] Where F = (f1, f2, ..., fn) represents the intermediate features of the basic neural network (BCNN); G = (g1, g2, ..., gn) represents the local features of the auxiliary neural network (ACNN); f i The eigenvector representing the i-th spatial location; g i The feature vector representing the i-th spatial location; This indicates that two feature vectors are concatenated; Θ represents a complex calculation method in which F and G are multiplied pixel by pixel with G and F respectively, and then the square root of the result is taken.

[0238] In this embodiment, the feature fusion module employs Weighted Feature Fusion (WFS). Weighted Feature Fusion reassembles the output vectors obtained from the three feature extractions according to three weight values, and then feeds the result into a neural network composed of two network blocks for classification. Each network block consists of a fully connected layer, a BatchNorm1d layer, and an activation layer, as shown in the following formula:

[0239]

[0240] Where RB = (rb1, rb2, ..., rb n );

[0241] RAWI = (rawi1,rawi2,```,rawi n );

[0242] RAO = (rao1,rao2,```,rao n );

[0243] Where RB, RAWI, and RAO represent the feature values ​​of the output results of the basic neural network (BCNN), ACNN_WI network, and ACNN_O network after dimensionality reduction, respectively; W1, W2, and W3 all represent weight values. This represents a vector concatenation operation; Represents a pixel-level vector value addition operation; Π xi This represents i fully connected layers.

[0244] Example 3:

[0245] This embodiment also provides an electronic device, including: a memory and a processor;

[0246] The memory stores computer-executable instructions.

[0247] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the CycleGAN and BAFCNN-based melanoma classification method in any of the embodiments of the present application.

[0248] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), programmable logic devices (PLD) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0249] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, etc.; and the data storage area can store data created during use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0250] Embodiment 4:

[0251] The embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are loaded by a processor to make the processor execute the CycleGAN and BAFCNN-based melanoma classification method in any of the embodiments of the present application. Specifically, a system or device provided with a storage medium can be provided, and the storage medium stores software program codes for realizing the functions of any of the above embodiments, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0252] In this case, the program codes read from the storage medium can realize the functions of any of the above embodiments, and therefore the program codes and the storage medium storing the program codes constitute a part of the present application.

[0253] The storage medium for providing the program code includes floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, nonvolatile memory cards and ROM. Alternatively, the program code can be downloaded from a server computer through a communication network.

[0254] In addition, it should be clear that not only the program code read by the computer can be executed, but also the operating system or the like operating on the computer can be caused to perform part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0255] In addition, it can be understood that the program code read by the storage medium is written into the memory provided in the expansion board inserted into the computer or the memory provided in the expansion unit connected to the computer, and then the CPU or the like installed on the expansion board or the expansion unit is caused to perform part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can still be modified, or part or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A melanoma classification method based on CycleGAN and BAFCNN, characterized in that, This method balances the dataset by using underrepresented samples from conditional image synthesis datasets, and extracts data features using a basic-assisted fusion neural network. These extracted features are then mapped onto the output space for melanoma processing and classification. Specifically: Conditional image synthesis: Conditional images are synthesized using a Cyclic Consistent CycleGAN network; Patch image extraction: The patch image is extracted using a pixel block difference patching strategy; Data preprocessing: The size of the original skin image is adjusted to 224x224, the size of the patch image is adjusted to 128x128, and the patch images are upsampled by 8x and 4x respectively, and then the albmentations operation is performed. Feature extraction: A basic auxiliary fusion neural network is used to extract features from the patch image and the original skin image, and to obtain information about the skin lesion image; Feature fusion: A weighted fusion strategy is used to fuse and classify the features output by the basic auxiliary fusion neural network; Evaluation metrics: Different evaluation metrics were used to assess the results of the binary classification prediction of melanoma. Among them, the basic-assisted fusion neural network includes the basic neural network, the auxiliary neural network, and the fusion neural network; A basic neural network is used to extract features from the patch image; The auxiliary neural network includes the ACNN_WI network and the ACNN_O network; the ACNN_WI network is used to input the corresponding global image; the ACNN_O network is used to input the patch image and fuses the intermediate layer feature matrix of the basic neural network with the intermediate layer feature matrix of the ACNN_WI network through a weighted ensemble strategy. A fusion neural network is used to obtain local fine-grained information and global information of skin lesion images and output classification results; The feature extraction process is as follows: The patch image is upsampled by a factor of 4 using the ACNN_O network. Specifically, the ACNN_O network consists of seven convolutional network layers, a BatchNorm2d layer, and an activation function. The patch image is upsampled and then fed into the network containing seven convolutional layers to obtain the output vector. The output vector is fed into the ACNN_WI network, which consists of eight convolutional network layers, a BatchNorm2d layer, and an activation function. The output of the intermediate layers of the basic neural network is used as a global guiding input to the ACNN_WI network, and combined with a weighted ensemble strategy to obtain the final output result. The formula for the weighted ensemble strategy is as follows: Where F = (f1, f2, ..., fn) represents the intermediate features of the basic neural network; G = (g1, g2, ..., gn) represents the local features of the auxiliary neural network; f i The eigenvector representing the i-th spatial location; g i The feature vector representing the i-th spatial location; This indicates that two feature vectors are concatenated; Θ indicates that F and G are multiplied pixel by pixel with G and F respectively, and then the result is squared.

2. The melanoma classification method based on CycleGAN and BAFCNN according to claim 1, characterized in that, The specific details of conditional image synthesis are as follows: Non-melanoma images and melanoma images in the dataset are defined as the X domain and Y domain of the CycleGAN network, respectively. Non-melanoma images in the X domain and melanoma images in the Y domain are fed into a pair of game-theoretic networks for training; the CycleGAN network includes a generator G. XY Generator G YX Discriminator D XY and Discriminator D YX ; Generator G XY Generator G is used to synthesize a sample Y' similar to a sample in the Y domain from an image in the X domain; YX X' domain feature map used to conditionally synthesize X domain samples using Y domain images; Discriminator D XY Used to score the features of samples in the Y domain and sample Y'; discriminator D YX Used to score X-domain samples and X'-domain feature maps; The final loss is obtained by summing the feature scores of the Y-domain samples and sample Y', and the feature map scores of the X-domain samples and X'. The generator G is updated based on the evaluation results, and finally an equilibrium state is reached through game theory.

3. The melanoma classification method based on CycleGAN and BAFCNN according to claim 1, characterized in that, The specific steps for extracting the patch image are as follows: Divide the original skin image into T pixel blocks on both central axes; Calculate the sum of pixels and values ​​in each pixel block; Obtain the difference between two adjacent pixel blocks and determine whether the difference between the adjacent pixel blocks is greater than a threshold. If so, it means that a significant color change has occurred in two adjacent pixel blocks, that is, the lesion boundary of the skin lesion image occurs between these two adjacent pixel blocks; Based on the lesion boundary of the skin lesion image, the position far away from the lesion tissue in the pixel block with a difference greater than the threshold is selected as the cutting line of the original skin image; The cutting lines of the original skin image are obtained as follows: The original skin image is divided into T blocks along its two central axes, denoted as t1, t2, ..., t3. n ; Calculate the difference between the sum of pixels in every two adjacent blocks; Starting from both ends of the central axis, find pixels with a difference greater than a threshold as the boundary lines of the sliced ​​image, using the following formula: Where L, R, U, and D represent the distances from the target patch image to the left, right, top, and bottom edges of the original skin image, respectively; W and H represent the width and height of the original skin image, respectively; T represents the number of grid lines, with a larger T resulting in a more refined patch image; T i S(T) represents the i-th pixel block; i ) indicates calculating the sum of the i-th pixel block; B indicates returning the i-th value of the first or last pixel block where the difference between two pixel blocks is greater than the threshold; When the number of pixel features in a sliced ​​image is less than one-quarter of that in the original skin image, a geometric slice patching strategy is used to compensate for the defects in the sample.

4. The melanoma classification method based on CycleGAN and BAFCNN according to claim 1, characterized in that, The fusion strategy involves modifying the extracted feature information, fusing the three two-dimensional feature vectors at the pixel level, and then feeding them into the network block. The fusion strategy employs weighted feature fusion, which involves recombining the output vectors obtained from the three feature extractions according to three weight values, and then feeding the result into a neural network consisting of two network blocks for classification. Each network block consists of a fully connected layer, a BatchNorm1d layer, and an activation layer, as shown in the following formula: Where RB = (rb 1, rb 2,```, rb n ); RAWI=(rawi 1, rawi2,```,rawi n ); RAO = (rao 1, Advertisement 2, ```,rao n ); Where RB, RAWI, and RAO represent the feature values ​​of the output results of the basic neural network, ACNN_WI network, and ACNN_O network after dimensionality reduction, respectively; W1, W2, and W3 all represent weight values. This represents a vector concatenation operation; Represents a pixel-level vector value addition operation; Π xi This represents i fully connected layers.

5. The melanoma classification method based on CycleGAN and BAFCNN according to any one of claims 1 to 4, characterized in that, The specific indicators for evaluation are as follows: Different evaluation metrics were used to evaluate the melanoma binary classification prediction results, including accuracy, precision, recall, F1 score, and AUC. The evaluation is based on the accuracy of the evaluation index, as shown in the following formula: The purpose of the evaluation metric accuracy is to detect the ratio of correctly classified positive cases to the total number of samples; Accuracy represents the metric accuracy; TP represents the number of people who are both labeled and predicted to be normal; FN represents the number of people who are labeled as normal but predicted to have melanoma; FP represents the number of people who are labeled to have melanoma but predicted to be normal; TN represents the number of people who are both labeled and predicted to have melanoma. Precision is the proportion of samples that predict positive examples out of the total number of positive instances. Its purpose is to measure "precision," and the formula is as follows: Recall reflects the proportion of samples predicted as positive out of all positive samples; its purpose is to measure "complete detection." The formula is as follows: To evaluate the performance of different algorithms, the concept of F1 score is proposed based on precision and recall, providing an overall evaluation of precision and recall. The formula is as follows: AUC is used to summarize the information contained in the ROC curve. The ROC curve plots the TPR and FPR at various thresholds, i.e., the false alarm rate. The larger the AUC value, the better the performance in distinguishing between melanoma and non-melanoma images.

6. A melanoma classification system based on CycleGAN and BAFCNN, characterized in that, The system includes, A conditional image synthesis module is used to synthesize conditional images using a Cyclic Consistent CycleGAN network. The patch image extraction module is used to extract patch images using a pixel block difference patching strategy. The preprocessing module is used to adjust the size of the original skin image to 224x224, adjust the size of the patch image to 128x128, and upsample the patch image by 8x and 4x respectively, and then perform albummentations. The feature extraction module is used to extract features from patch images and original samples using a basic auxiliary fusion neural network, and to obtain information about the skin lesion images; The feature fusion module is used to fuse and classify the features output by the basic auxiliary fusion neural network using a weight fusion strategy. The indicator evaluation module is used to evaluate the melanoma binary classification prediction results using different evaluation indicators; Among them, the basic-assisted fusion neural network includes the basic neural network, the auxiliary neural network, and the fusion neural network; A basic neural network is used to extract features from the patch image; The auxiliary neural network includes the ACNN_WI network and the ACNN_O network; the ACNN_WI network is used to input the corresponding global image; the ACNN_O network is used to input the patch image and fuses the intermediate layer feature matrix of the basic neural network with the intermediate layer feature matrix of the ACNN_WI network through a weighted ensemble strategy. A fusion neural network is used to obtain local fine-grained information and global information of skin lesion images and output classification results; The feature extraction module includes, The upsampling submodule is used to upsample the patch image by a factor of 4 through the ACNN_O network. Specifically, the ACNN_O network consists of seven convolutional network layers, a BatchNorm2d layer, and an activation function. It upsamples the patch image and then feeds it into the network containing seven convolutional layers to obtain the output vector. The input submodule is used to feed the output vector into the ACNN_WI network; the ACNN_WI network consists of eight convolutional network layers, a BatchNorm2d layer, and an activation function. The output submodule uses the output of the intermediate layers of the basic neural network as a global guide input to the ACNN_WI network, and combines it with a weighted ensemble strategy to obtain the final output result. The formula for the weighted ensemble strategy is as follows: Where F = (f1, f2, ..., fn) represents the intermediate features of the basic neural network; G = (g1, g2, ..., gn) represents the local features of the auxiliary neural network; f i The eigenvector representing the i-th spatial location; g i The feature vector representing the i-th spatial location; This indicates that two feature vectors are concatenated; Θ indicates that F and G are multiplied pixel by pixel with G and F respectively, and then the result is squared.

7. The melanoma classification system based on CycleGAN and BAFCNN according to claim 6, characterized in that, The conditional image synthesis module includes, The image definition submodule is used to define non-melanoma images and melanoma images in the dataset as the X and Y domains of the CycleGAN network, respectively. The image training submodule is used to feed non-melanoma images in the X domain and melanoma images in the Y domain into a pair of game-theoretic networks for training; wherein, the CycleGAN network includes a generator G. XY Generator G YX Discriminator D XY and Discriminator D YX ; Generator G XY , used to synthesize a sample Y' from an X-domain image that is similar to a Y-domain sample; Generator G YX , used to synthesize the X' domain feature map of X domain samples using Y domain image conditions; Discriminator D XY , used to score the features of samples in the Y domain and sample Y'; Discriminator D YX , used to score X domain samples and X' domain feature maps; The summation submodule is used to sum the feature scores of samples in the Y domain and sample Y' and the feature map scores of samples in the X domain and X' to obtain the final loss. The update submodule is used to update the generator G based on the evaluation results, and finally reach an equilibrium state through game theory. The patch image extraction module includes, The pixel block acquisition submodule is used to divide the two central axes of the original skin image into T pixel blocks on an equal basis; The pixel block and value acquisition submodule is used to calculate the pixel sum and value in each pixel block; The pixel block difference acquisition and judgment submodule is used to acquire the difference between two adjacent pixel blocks and determine whether the difference between adjacent pixel blocks is greater than a threshold. If so, it means that a significant color change has occurred in two adjacent pixel blocks, that is, the lesion boundary of the skin lesion image occurs between these two adjacent pixel blocks; The selection submodule is used to select the location far from the lesion tissue in the pixel block with a difference greater than a threshold as the cutting line of the original skin image based on the lesion boundary of the skin lesion image; The feature fusion module employs weighted feature fusion, which reassembles the output vectors obtained from the three feature extractions according to three weight values, and feeds the result into a neural network composed of two network blocks for classification. Each network block consists of a fully connected layer, a BatchNorm1d layer, and an activation layer, as shown in the following formula: Where RB = (rb 1, rb 2,```, rb n ); RAWI=(rawi 1, rawi2,```,rawi n ); RAO = (rao 1, Advertisement 2, ```,rao n ); Where RB, RAWI, and RAO represent the feature values ​​of the output results of the basic neural network, ACNN_WI network, and ACNN_O network after dimensionality reduction, respectively; W1, W2, and W3 all represent weight values. This represents a vector concatenation operation; Represents a pixel-level vector value addition operation; Π xi This represents i fully connected layers.

8. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the melanoma classification method based on CycleGAN and BAFCNN as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the melanoma classification method based on CycleGAN and BAFCNN as described in any one of claims 1 to 5.

Images