Monkey pox image classification method, device and system and storage medium
By constructing a Siamese neural network and a similarity prediction method, the problem of the inability to provide the probability of class labels for monkeypox image samples in existing technologies is solved, and accurate classification of monkeypox images and improvement of dataset balance are achieved.
Patent Information
- Application Number
- CN202410958302.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing Siamese neural networks can only determine whether paired input samples belong to the same category, but cannot provide the category label of the sample and the probability of each label, thus failing to effectively solve the classification problem of monkeypox images.
By combining the concepts of Siamese neural networks and similarity prediction, a monkeypox image dataset including a pre-trained model and a multilayer perceptron is constructed. The similarity of feature map vectors is calculated using a distance function to predict the positive probability of samples, thereby achieving accurate classification of monkeypox images.
It improves the accuracy of monkeypox image classification and the distribution balance of the dataset when dealing with small sample data and class imbalance, and can effectively predict the class label probability of samples.
Smart Images

Figure CN118918368A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a monkeypox image classification method, device, system and storage medium. Background Art
[0002] Convolutional Neural Networks (CNN) is considered to be an efficient deep learning tool, and its structure has many similarities with the multilayer perceptron in artificial neural networks. CNN is a method to improve and expand the traditional neural network structure. Although it is still a layered network, it has made subtle adjustments to its functions and structure. Convolutional neural network is like a smart student, it can learn a lot of relationships between inputs and outputs. Moreover, it does not need to remember any formulas or rules to describe this relationship. It only needs to train itself through known examples, and then it can predict the output based on the input. In this way, there is no need to use traditional complex calculation methods for recognition. As a powerful tool in the field of image processing, the core functions of convolutional neural network cover image classification, detection of target objects in images, image segmentation and other aspects, which effectively improves the accuracy and efficiency of image processing. The basic structure of the model includes input layer, convolution layer, pooling layer, fully connected layer and output layer.
[0003] After extracting image features from an image based on a convolutional neural network, in the prior art, a twin neural network is used to classify the image. Currently, the twin neural network can only determine whether the input paired samples belong to the same category, but cannot give the category label to which the sample belongs and the probability of each label. Summary of the invention
[0004] The present invention is provided to solve the above problems existing in the prior art. Therefore, a monkeypox image classification method, device, system and storage medium are needed, which combines the twin neural network with the similarity prediction idea to not only determine whether the samples belong to the same category, but also predict the probability of their respective labels.
[0005] According to a first embodiment of the present invention, a method for classifying monkeypox images is provided, the method comprising:
[0006] Acquire a plurality of monkeypox images, and construct a monkeypox image dataset based on the plurality of monkeypox images;
[0007] Constructing a twin neural network, the twin neural network comprising a pre-trained model, a multi-layer perceptron and a distance function, the pre-trained model is trained based on the monkeypox image dataset, the twin neural network is trained using the monkeypox image dataset to obtain a monkeypox image classification model, and monkeypox image classification is implemented based on the monkeypox image classification model;
[0008] When the twin neural network is trained using the monkeypox image dataset, a sample to be predicted, a negative sample and a positive sample are obtained from the monkeypox image dataset, and the sample to be predicted is respectively combined with the negative sample and the positive sample to form a first sample pair and a second sample pair, and the pre-trained model generates a first feature map group and a second feature map group corresponding to the first sample pair and the second sample pair in response to the input first sample pair and second sample pair, respectively, and the first feature map group and the second feature map group are fed to the multi-layer perceptron and output a first feature map vector group and a second feature map vector group, wherein the first feature map vector group includes feature map vectors corresponding to the predicted sample and the negative sample, and the second feature map vector group includes feature map vectors corresponding to the predicted sample and the positive sample, and the similarity of two feature map vectors in the first feature map vector group and the second feature map vector group is calculated respectively by using the distance function set at the output end of the multi-layer perceptron, and a first similarity p between the sample to be predicted and the negative sample and a second similarity n between the sample to be predicted and the positive sample are obtained, and the probability that the sample to be predicted is predicted to be positive is obtained according to the first similarity p and the second similarity n, and the probability is p / (p+n).
[0009] According to a second technical solution of the present invention, a monkeypox image classification device is provided, the device comprising:
[0010] a data acquisition module, configured to acquire a plurality of monkeypox images and construct a monkeypox image dataset according to the plurality of monkeypox images;
[0011] An image classification module is configured to construct a twin neural network, wherein the twin neural network includes a pre-trained model, a multi-layer perceptron, and a distance function, wherein the pre-trained model is trained based on the monkeypox image dataset, the twin neural network is trained using the monkeypox image dataset to obtain a monkeypox image classification model, and monkeypox image classification is implemented based on the monkeypox image classification model;
[0012] When the twin neural network is trained using the monkeypox image dataset, a sample to be predicted, a negative sample and a positive sample are obtained from the monkeypox image dataset, and the sample to be predicted is respectively combined with the negative sample and the positive sample to form a first sample pair and a second sample pair, and the pre-trained model generates a first feature map group and a second feature map group corresponding to the first sample pair and the second sample pair in response to the input first sample pair and second sample pair, respectively, and the first feature map group and the second feature map group are fed to the multi-layer perceptron and output a first feature map vector group and a second feature map vector group, wherein the first feature map vector group includes feature map vectors corresponding to the predicted sample and the negative sample, and the second feature map vector group includes feature map vectors corresponding to the predicted sample and the positive sample, and the similarity of two feature map vectors in the first feature map vector group and the second feature map vector group is calculated respectively by using the distance function set at the output end of the multi-layer perceptron, and a first similarity p between the sample to be predicted and the negative sample and a second similarity n between the sample to be predicted and the positive sample are obtained, and the probability that the sample to be predicted is predicted to be positive is obtained according to the first similarity p and the second similarity n, and the probability is p / (p+n).
[0013] According to a third technical solution of the present invention, a monkeypox image classification system is provided, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method described above.
[0014] According to a fourth technical solution of the present invention, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor, the method described above is executed.
[0015] The monkeypox image classification method, device, system and storage medium according to each solution of the present invention have at least the following technical effects:
[0016] In constructing the data set, the present invention comprehensively considers small sample data and sample category imbalance, so that more and more accurate monkeypox image samples can be obtained, and the distribution balance of the data set can be guaranteed. At the same time, the present invention introduces a method of fusing a pre-trained model with a multi-layer perceptron (MLP) to improve the scalability of the twin neural network, so that the twin neural network can predict the probability that the input features belong to their respective labels, thereby improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flowchart of a monkeypox image classification method according to an embodiment of the present invention is shown.
[0018] Figure 2 A flowchart of constructing a monkeypox image dataset according to an embodiment of the present invention is shown.
[0019] Figure 3 A schematic diagram of a small sample learning model training and testing process according to an embodiment of the present invention is shown.
[0020] Figure 4 A diagram showing a small sample monkeypox image classification training process according to an embodiment of the present invention is shown.
[0021] Figure 5 A sample image histogram according to an embodiment of the present invention is shown.
[0022] Figure 6 A schematic diagram of a twin neural network training framework according to an embodiment of the present invention is shown.
[0023] Figure 7 A structural diagram of a multi-layer perceptron according to an embodiment of the present invention is shown.
[0024] Figure 8 A schematic diagram of a similarity-based twin neural network prediction framework according to an embodiment of the present invention is shown.
[0025] Fig. 9 A DenseNet network structure diagram according to an embodiment of the present invention is shown.
[0026] Fig.10 A dense connection schematic diagram according to an embodiment of the present invention is shown.
[0027] Fig.11 The Transition layer structure according to an embodiment of the present invention is shown.
[0028] Fig.12 The Inception network structure according to an embodiment of the present invention is shown.
[0029] Fig.13 A depthwise separable convolution process according to an embodiment of the present invention is shown.
[0030] Fig.14 An Xception architecture diagram according to an embodiment of the present invention is shown.
[0031] Fig.15 A schematic diagram of loss curves of different loss functions and accuracy curves of different loss functions of the DenseNet model according to an embodiment of the present invention is shown.
[0032] Fig.16 A schematic diagram of loss curves of different loss functions of the Xception model and accuracy curves of different loss functions according to an embodiment of the present invention is shown.
[0033] Fig.17A schematic diagram of loss curves of different loss functions of the Inception model and accuracy curves of different loss functions according to an embodiment of the present invention is shown.
[0034] Fig.18 ROC curves of different loss functions of the DenseNet model according to an embodiment of the present invention are shown.
[0035] Fig.19 The ROC curves of different loss functions of the Xception model according to an embodiment of the present invention are shown.
[0036] Fig. 20 The ROC curves of different loss functions of the Inception model according to an embodiment of the present invention are shown.
[0037] Fig.21 The test set confusion matrix of different loss functions of the DenseNet model according to an embodiment of the present invention is shown.
[0038] Fig. 22 The test set confusion matrix of different loss functions of the Xception model according to an embodiment of the present invention is shown.
[0039] Fig.23 The test set confusion matrix of different loss functions of the Inception model according to an embodiment of the present invention is shown.
[0040] Fig.24 The predicted probability distribution of different loss functions of the DenseNet model according to an embodiment of the present invention is shown.
[0041] Fig.25 The predicted probability distribution of different loss functions of the Xception model according to an embodiment of the present invention is shown.
[0042] Fig.26 The predicted probability distribution of different loss functions of the Inception model according to an embodiment of the present invention is shown.
[0043] Fig. 27 A schematic diagram showing the training accuracy of each model for different data sets according to an embodiment of the present invention is shown.
[0044] Fig.28 A schematic diagram of the validation set accuracy of different data sets according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0046] The present invention provides a method for classifying monkeypox images. Figure 1 , is a flow chart of a method for classifying monkeypox images according to an embodiment of the present invention. The method comprises the following steps:
[0047] Step S100, obtaining a plurality of monkeypox images, and constructing a monkeypox image dataset according to the plurality of monkeypox images.
[0048] In some embodiments, considering that most previous medical imaging studies only considered one of the situations of small samples or class imbalance, and there are few studies on monkeypox images at present; this embodiment combines the actual situation and comprehensively considers the small sample data and sample class imbalance to construct a monkeypox image dataset with balanced sample classification.
[0049] like Figure 2 As shown in the figure, it is a flowchart for constructing the monkeypox image dataset. The monkeypox image dataset is constructed through the following steps:
[0050] Step S101 , selecting images with labels from the plurality of monkeypox images to construct a first data set.
[0051] Step S102: construct a first training set and a query set based on the first data set, wherein the first training set and the query set are disjoint, and the query set is composed of at least one monkeypox image randomly selected from various labels in the first data set.
[0052] Step S103, constructing a small sample monkeypox image classification model, and training the small sample monkeypox image classification model using the first training set and the query set to obtain a trained small sample monkeypox image classification model.
[0053] In a training cycle, a support sample in the first training set and all query samples in the query set are used as input, and an embedding module is used to extract a support sample and all query samples to obtain a corresponding first image feature and multiple second image features. The metric function is used to evaluate the similarity between the first image feature and each second image feature, and the label corresponding to the second image feature most similar to the first image feature is used as the predicted value. The loss function is used to calculate the difference between the actual value and the predicted value of the query sample in the current training cycle, and the actual value is determined according to the label corresponding to the query sample. The gradient is calculated using the back propagation algorithm, and the model parameters of the small sample monkeypox image classification model are continuously updated by the gradient descent method. The trained small sample monkeypox image classification model can label images that do not have labels in multiple monkeypox images.
[0054] Specifically, in real life, there is often an imbalance in data categories. Imbalance is mainly manifested in the large difference between the number of samples in one or several categories and the number of samples in other categories in the data set. In addition, there is a difference in the cost of misclassification between classes with fewer samples and classes with more samples, where the cost of misclassification of classes with fewer samples is usually more expensive. Classes with fewer samples are called minority classes, while classes with more samples are called majority classes.
[0055] In imbalanced data classification, the ratio between the number of samples in the majority class and the number of samples in the minority class is the imbalance ratio. According to the numerical value of the imbalance ratio, imbalanced classification can be refined into two types: mildly imbalanced data classification and highly imbalanced data classification. Generally, when the imbalance ratio exceeds 9, it is classified as highly imbalanced data classification. Such a division helps to more accurately describe and handle the imbalance problem in the data set.
[0056] Machine learning requires a large amount of training data, but in real life, we often encounter situations where certain data categories contain only a small amount of data, especially in the field of medical imaging research, where the problem of small amount of data and uneven distribution of categories is becoming increasingly prominent. However, humans can learn quickly with a small amount of data. For example, a six-year-old child had never seen a real sloth before going to the zoo, but after showing him a photo of a sloth, he was able to quickly identify the real sloth species when he saw one in the zoo. Inspired by the human learning mechanism, researchers proposed small sample learning, with the goal of enabling machine learning models to learn to solve problems and make the models more human-like.
[0057] The small sample in Few-shot Learning refers to the relatively small number of training samples contained in each category. In image classification, the goal of small-shot learning is to make the model class human, understand the similarities and differences of things, and learn to solve problems.
[0058] The small sample problem is also called the N-way K-shot problem, where N refers to the number of categories and K refers to the number of samples in each category. When only one sample is selected from each category (K = 1), the task is defined as a One-shot task; when the K value is set to 0, it is called a Zero-shot task. The specific learning process is as follows Figure 3As shown in the figure, in the training phase, N different categories are randomly selected from the original training set containing a large number of labeled samples, and K different images are randomly selected from various categories to form a support set (Support Set), which contains images and image label information. The support set is usually a smaller data set used to provide additional information. Then, some images are randomly selected from the remaining training samples in each category to form a query set, which also contains images and image label information. The support set and the query set do not intersect. The parameters of the model are not updated in the test phase, and the rest are similar to the training phase. For example, 10 classes are randomly selected, C samples are extracted from each class, 5 samples are randomly selected from the C samples as support samples, and the remaining C-5 samples are used as query samples. This is called the 10-way 5-shot problem. The fewer the number of categories, the higher the accuracy of small sample image classification; the more samples in each category, the higher the classification accuracy.
[0059] The main goal of training is to enable the neural network to effectively map various types of data into similar label spaces, and to accurately classify each item of data in the query set after comparing the query set with the sample set. As the training process progresses, the model gradually masters the ability to accurately identify new sample categories using the characteristics of known samples.
[0060] Metric-based small sample learning calculates the similarity between samples through a certain established function, and then classifies the samples of unclear categories. This method generally adopts an episodic training method, that is, decomposing the overall classification problem into a series of subtasks. In a complete training cycle (i.e., an epoch), it involves performing a forward propagation and a corresponding backpropagation process on all samples in the training set. In this process, each training stage (Episode) contains a support set and a query set. More specifically, during the training process, an embedding module is first used to extract the feature representations of the support samples and the query samples. Subsequently, the metric function is used to evaluate the similarity between these features, so as to determine the category with the highest similarity to the query sample category and use it as the prediction result. In order to measure the accuracy of the prediction, the loss function is used to calculate the difference between the actual value of the query sample and the predicted value. Then, the gradient is calculated using the backpropagation algorithm, and the model parameters are continuously updated through the gradient descent method to optimize the performance of the model. Figure 4 Demonstrates the training process for the few-shot monkeypox image classification task.
[0061] Exemplarily, the data used in this embodiment is derived from a publicly available monkeypox dataset. The dataset is divided into positive (positive) and negative (negative). Only the skin lesion image data of monkeypox patients are positive samples or positive samples, with a total of 117 images; negative samples or negative samples are non-monkeypox skin images, with a total of 577 images, mainly including chickenpox, cowpox, smallpox and measles, which are similar to monkeypox skin lesions. The dataset is subdivided into a training set and a test set. There are 520 images in the training set, of which 80 are positive (monkeypox) samples and 440 are negative (other) samples. Among the 174 test images used, 37 contain positive (monkeypox) images and 137 contain negative (other) images, as shown in Table 1.
[0062] Table 1 Dataset distribution
[0063] Monkeypox Other categories total Training set 80 440 520 Test Set 37 137 174 total 117 577 694
[0064] Image histogram analysis is a statistical method based on grayscale values. It is one of the best ways to identify pixel intensities and differences between various images. Specifically, the grayscale histogram is used to count the distribution probabilities of different grayscale levels. It can provide important information about the overall brightness distribution of the image, and can also reflect the brightness differences between different image categories. This paper selects samples with the negative class (non-monkeypox) name "ch_2" and the positive class (monkeypox) name "mo_27" in the dataset for image histogram analysis. Using grayscale histogram technology, the brightness and darkness of the image can be clearly displayed. When the grayscale value of an image is more likely to be 0, it means that the brightness of the image is lower; when the probability of the grayscale value being 255 is higher, it indicates that the image is brighter. According to Figure 5 The sample image histogram visualization results show that the grayscale distribution of positive samples is mainly concentrated around 50, while the grayscale distribution of negative samples is more concentrated around 200. This phenomenon shows that the images of negative samples are generally brighter, while the images of positive samples are relatively darker.
[0065] Data augmentation is one of the commonly used methods to improve model performance in image classification tasks. It can effectively improve the coverage, diversity and generalization ability of training datasets. It helps models perform better in real-world environments. Data augmentation is based on image geometric operations (rotation, translation, noise injection, color space transformation), and photometric transformation is not used for quality control purposes.
[0066] This paper uses image enhancement technology to process data. Specifically, it adopts a strategy of randomly rotating images to enhance the generalization ability of the model and improve its recognition performance of images at different angles. In addition, the range, direction, and step size of image translation are specified in a random manner with a given probability, further enriching the diversity of training samples. The relevant detailed configurations are listed in Table 2.
[0067] Table 2 Image enhancement parameter configuration
[0068]
[0069]
[0070] This example uses sampling to construct training samples, selects appropriate loss functions and pre-training models, and builds a small sample learning framework of a twin neural network to complete the classification of monkeypox images. This article selects indicators such as confusion matrix, accuracy, recall rate, F1-Score, and AUC to evaluate the classification ability of the model.
[0071] When evaluating imbalanced classification, the confusion matrix is often used. Its core function is to compare the model's predicted output for new samples with their true labels to quantify the accuracy of the model. The confusion matrix is presented in a matrix form, which contains all possible classification results and the corresponding number of instances. The confusion matrix is shown in Table 3.
[0072] Table 3. Classification results confusion matrix
[0073]
[0074] In monkeypox image classification, TP represents the number of samples that the model can accurately classify as monkeypox images; TN represents the number of samples that the model can accurately classify as non-monkeypox images; FP represents the number of samples that the model classifies as non-monkeypox images as monkeypox; and FN represents the number of samples that the model classifies as non-monkeypox images.
[0075] In the binary classification problem, classification accuracy is considered to be one of the most commonly used evaluation criteria. It helps to assess the overall effectiveness of the classifier and is calculated using the following formula:
[0076]
[0077] When evaluating tilt sensitivity, precision is a key indicator, which represents the proportion of samples that are truly positive among all predictions classified as positive. A classifier with high precision can usually accurately identify most of the positive samples, thereby ensuring the validity and reliability of the prediction results, which is calculated using the following formula:
[0078]
[0079] Recall is also widely used, which represents the proportion of the total number of true positive examples in the test set that are correctly predicted as positive examples, also known as the true positive rate (TPR). A classifier with a high recall rate shows a stronger capture ability when facing positive samples, and can therefore more accurately classify them into the correct category, which can be calculated using the following formula:
[0080]
[0081] F1-Score refers to the harmonic mean of precision and recall. In the field of statistics, F1-Score is used as a key indicator to evaluate the accuracy of binary classification models. It not only considers the accuracy of the classification model, but also its sensitivity. This indicator is regarded as a weighted average of the precision and sensitivity of the model. Its value range is between 0 and 1. The closer the value is to 1, the better the performance of the model. It is calculated using the following formula:
[0082]
[0083] The false positive rate is calculated using the following formula:
[0084]
[0085] Using the false positive rate as the horizontal axis and the true positive rate as the vertical axis, the ROC curve can be accurately depicted by collecting the values of the false positive rate and the true positive rate under different thresholds. Each point on the ROC curve corresponds to a specific threshold. When training, it is necessary to select each sample point to improve the classification accuracy. Since the ROC curve cannot clearly indicate which model performs better, the AUC value is selected as the evaluation criterion for the prediction model. The value of AUC (Area Under Curve) represents the area covered by the ROC curve. The larger the value of AUC, the better the classification performance of the model. To a certain extent, it can reflect the model's ability to classify the data set. When AUC is 1, the model shows the best classification performance, and accurate classification can be achieved regardless of the threshold setting of the model. When AUC exceeds 0.5, it means that after setting an appropriate threshold for the model, a more accurate prediction effect can be obtained. When AUC reaches 0.5, the prediction effect of the model is similar to random guessing and does not have the value of prediction. When AUC is lower than 0.5, the prediction ability of the model is even worse than random prediction.
[0086] Since the ROC curve cannot clearly indicate which classifier performs better in many cases, but when it is used as a numerical value, it means that the classifier with a larger AUC is more effective, so the AUC value is used as the evaluation criterion.
[0087] This example introduces the concepts of small samples and imbalance and introduces the dataset of this article. In order to further explore the characteristic distribution of sample data of different categories, the image histogram visualization method is used to provide a theoretical basis for subsequent experiments. Subsequently, for the training samples, the parameter settings of data enhancement are clarified and the methods used are explained. Finally, some commonly used model evaluation indicators are outlined, which lays a solid foundation for the subsequent in-depth discussion of the model.
[0088] Step S200, constructing a twin neural network, the twin neural network including a pre-trained model, a multi-layer perceptron and a distance function, the pre-trained model is trained based on the monkeypox image dataset, the twin neural network is trained using the monkeypox image dataset to obtain a monkeypox image classification model, and monkeypox image classification is implemented based on the monkeypox image classification model.
[0089] When the twin neural network is trained using the monkeypox image dataset, a sample to be predicted, a negative sample and a positive sample are obtained from the monkeypox image dataset, and the sample to be predicted is respectively combined with the negative sample and the positive sample to form a first sample pair and a second sample pair, and the pre-trained model generates a first feature map group and a second feature map group corresponding to the first sample pair and the second sample pair in response to the input first sample pair and second sample pair, respectively, and the first feature map group and the second feature map group are fed to the multi-layer perceptron and output a first feature map vector group and a second feature map vector group, wherein the first feature map vector group includes feature map vectors corresponding to the predicted sample and the negative sample, and the second feature map vector group includes feature map vectors corresponding to the predicted sample and the positive sample, and the similarity of two feature map vectors in the first feature map vector group and the second feature map vector group is calculated respectively by using the distance function set at the output end of the multi-layer perceptron, and a first similarity p between the sample to be predicted and the negative sample and a second similarity n between the sample to be predicted and the positive sample are obtained, and the probability that the sample to be predicted is predicted to be positive is obtained according to the first similarity p and the second similarity n, and the probability is p / (p+n).
[0090] For example, this embodiment uses a twin neural network to train data, and the training framework is as follows: Figure 6 As shown in Figure 1. First, the sample pairs in the training set are input into the pre-trained model (G(X)) to generate a pair of tensors. Then, a multi-layer perceptron (MLP F(X)) with shared weights is used to extract features from the input images and convert them into vector form. The structure of the multi-layer perceptron is as follows: Figure 7 As shown in the figure, it consists of two fully connected layers and one Gelu activation function layer, where the number of hidden nodes is set to 128 and 64 respectively. Finally, the distance function is used to calculate the similarity between two one-dimensional vectors. In order to reduce the risk of overfitting, a Dropout layer is introduced for each fully connected layer during training, and its probability is set to 20%. This strategy helps reduce the model's over-reliance on specific training samples and improves the generalization ability on unseen data.
[0091] In the prediction stage, the twin neural network cannot directly predict the category to which the sample belongs, but can only determine whether the input paired samples belong to the same category. Therefore, it is necessary to calculate the probability that the predicted sample is positive, such as Figure 8 As shown. First, a sample is randomly selected from the test set. Then, two samples are extracted from the training set by random stratified sampling. Subsequently, the selected samples from the test set are paired with the samples from the two training sets, and passed together as input to the pre-trained neural network model. After that, the similarity function is used to calculate the similarity n between the predicted sample and the negative sample, and the similarity p between the predicted sample and the positive sample. Finally, the probability that the predicted sample belongs to the positive category is calculated based on the similarity value p / (p+n). The specific training process is shown in the following formula:
[0092]
[0093] The predicted probability is calculated using the following formula:
[0094]
[0095] In the formula, output p Represents the similarity between the test sample and the positive sample, which reflects the similarity between the test sample and the positive sample. Output n It represents the similarity between the test sample and the negative sample, which measures the similarity between the test sample and the negative category sample.
[0096] In some embodiments, the pre-trained model can be a first neural network convolution model, a second neural network convolution model, or a third neural network convolution model. The specific composition of the above three models and the loss functions they use will be introduced in detail below.
[0097] In this embodiment, the first neural network convolution model uses DenseNet. In the evolution of CNN, a common practice is to enhance its performance by increasing the number of layers or expanding the width. However, DenseNet takes a different approach and proposes a new network structure that achieves evolution by reusing features. This method can not only alleviate the problem of gradient disappearance, but also reduce the number of parameters.
[0098] The network structure of DenseNet is as follows Fig. 9 As shown. Each node has a fixed connection and has the same data access method. The model consists of multiple dense blocks and transition layers alternating. Each dense block is composed of a series of convolutional layers with the same feature map size but different kernel sizes, and the input of each convolutional layer is the concatenation of the outputs of all previous layers. Each standard convolutional layer consists of two components: BN+ReLU+1×1Conv and BN+ReLU+3×3Conv. The main function of the first 1×1 convolutional layer is to control the number of input feature maps, that is, channels. The second convolutional layer can process different types of images as needed. A notable feature of DenseNet is that it ensures a close connection between the previous and next layers by establishing dense connections between all convolutional layers, thereby reusing features more effectively. In this way, the error signal can be easily propagated to the earlier layers, allowing these layers to be directly supervised by the final classification layer. This design allows the gradient to flow more smoothly, thereby improving the performance of the network. The schematic diagram of the dense connection is shown in Fig.10 shown.
[0099] The dense block can be specifically expressed by the following formula.
[0100]
[0101] In the formula, x k is the output of the kth layer in the first neural network convolutional model, H k is the nonlinear transformation function of the kth layer, {x0,x1,…,x k-1} is the output set of layers 0 to k-1, L is the number of convolutional layers in the dense block, k is the sequence number of the convolutional layer in the dense block, x0 is the output of layer 0, and inputfeature is the input feature.
[0102] In addition to extracting only the highest-level features from the standard convolutional network, DenseNet introduces multiple levels of features and ensures that low-dimensional features are preserved. This approach makes the decision boundary smoother and easier to make accurate decisions.
[0103] In order to effectively control the network computational complexity and resource requirements, DenseNet introduces a transition layer, whose core function is to reduce the spatial dimension and total amount of feature maps, thereby improving the overall performance of the network. Fig.11 shown.
[0104] Fig.11In the example, the transition layer is composed of BN layer, Relu, Conv2D and AvgPool layers, and the size of the convolution kernel of the convolution layer (Conv2D) is usually 1×1, which is to reduce the number of channels of the input feature map. The size of the feature map can be reduced by using the average pooling layer (AvgPool). Through the Transition technology, large multi-channel feature maps can be converted into smaller low-channel feature maps and can be imported into the next DenseBlock.
[0105] Specifically, it can be expressed by the following formula.
[0106] x″=avgpool(x′)=avgpool(BN(Conv(1×1)(x))), for1≤k≤L.
[0107] In the formula, x is the input feature map, Conv(1×1)(x) represents the convolution operation on the input feature map, followed by normalization operation, represented by BN. The output of x after the transition layer is x′, and the final output of x′ after average pooling avgpool is x″.
[0108] This example uses a DenseNet-121 network, which consists of 120 convolutional layers and 4 average pooling layers.
[0109] In this embodiment, the second neural network convolution model adopts the Inception network, and the Inception network mainly includes Inceptionv1, Inceptionv2, Inceptionv3 and Inceptionv4, which have the fastest operation speed and the best parameter efficiency. In the traditional convolutional neural network, the network structure adopts a stacked form, and a very complex nonlinear simulation system is formed through the combination of multiple convolutional layers. It is worth noting that in the same convolutional layer, all convolution kernels share the same hyperparameter settings. This design ensures that the features extracted from each layer are consistent in dimension, thereby effectively extracting the key information in the image. The Inception architecture has made innovative improvements to the convolutional layer, allowing multiple paths to be set. Each path can perform different operations independently, and even different kernel sizes and step sizes can be set as needed. This design expands the width and depth of the convolution operation, links information across channels, and thus mines richer and deeper features. This not only improves the performance of the network, but also improves the utilization efficiency of computing resources. Specifically, in the architecture design of the Inception network, four parallel convolutional layers with different kernel sizes are used to extract image features of various scales and pass these features to the next level. The basic structure of Inception is as follows Fig.12 shown.
[0110] according to Fig.12 As shown in the figure, the core structure of Inception is to combine multiple convolution and pooling operations of different sizes into a unified module. After being processed by the Inception structure, the input image data will be extracted with various unique features and integrated on the calculation results of each branch as the final output of the layer. Since each branch has consistent width and height dimension outputs, in the process of network integration, it is only necessary to accumulate the number of channels of each branch, thereby improving the adaptability of the network at different scales. Figure 2-6 In the Inception module shown, the size of the convolution kernel is not fixed to 1×1, 3×3 or 5×5, but can be flexibly adjusted according to the needs and results of the specific experiment.
[0111] In summary, the innovation of the Inception architecture lies in the application of global average pooling to replace the fully connected layer, which greatly reduces the overall parameter size of the model. Based on this, Inceptionv2 further incorporates the BN layer and uses two sets of parallel 3×3 convolution blocks to replace the original 5×5 convolution blocks. Such structural adjustments not only effectively speed up the convergence speed during network training, but also improve the computational efficiency of the overall model during training. In addition, the BN layer can enhance the generalization performance of the model on unseen test data sets by standardizing the network layer input, which is expressed by the following formula.
[0112]
[0113] In the formula, Represents the average value of each neuron in the input data, Var(x (k) ) represents the standard deviation of the neuron input value in a batch of data, γ and β are parameters, represents the normalized sample, x (k) represents the data of the kth sample, γ (k) represents the scaling parameter, β (k) represents the translation parameter, and y(k) represents the output of the normalization layer.
[0114] In the design of the Inception v3 architecture, a new strategy called Factorization Convolutions was introduced. The basic idea is to use a pair of serial one-dimensional convolution operations to simulate the functional effects of traditional two-dimensional convolution layers. For example, the original single 7×7 convolution can be split into 1×7 convolution and 7×1 convolution to achieve functional equivalence. This is aimed at reducing the number of parameters required for model training and can potentially improve the model's adaptability to training data and its ability to express complex patterns. In addition, the Inception V3 structure also cleverly integrates a nonlinear transformation layer. This design helps to overcome the gradient vanishing or gradient exploding problems that may occur during training, thereby enhancing the stability and training efficiency of the network. The network structure is shown in Table 4.
[0115] Table 4 InceptionV3 network structure
[0116]
[0117] In this embodiment, the third neural network convolution model uses Xception, which is an improvement on the InceptionV3 structure. The main idea implemented in Xception is to make the convolution operation more effective in the Inception module by introducing more efficient depthwise separable convolution. Specifically, depthwise separable convolution is used to replace the Inception module in InceptionV3, and the connection between each part of the Xception model uses the residual connection method instead of the concat in the Inception network structure. The specific process of depthwise separable convolution is as follows: Fig.13 shown.
[0118] Compared with the traditional convolution calculation method, the depthwise separable convolution has a different calculation mechanism, but its performance is better. In terms of performance, the depthwise separable convolution shows higher computational efficiency; and in terms of accuracy, its loss is also kept at a relatively small level. The convolution is divided into two stages: first, pointwise convolution (PW), followed by depthwise convolution (DW). In this process, features are first extracted from each channel, and then the features between channels are connected through point-by-point convolution. Point-by-point convolution changes the dimension, while depth-by-depth convolution represents channel-by-channel spatial convolution. In a typical convolution calculation structure, if the length and width of the output feature map are the same, and the size of the input feature map is M×D F ×D F The input feature map length and width are D F, M represents the number of input channels; if we assume that the output is D k ×D k × N,
[0119] Middle D k Represents the length and width of the output feature map, and N represents the number of output channels; at this time, for a typical convolution D k ×D k The amount of calculation required can be expressed as D k ×D k ×M×N×D F ×D F Indicates; in the depth-level separable convolution calculation structure, the depth convolution calculation amount is expressed as D k ×D k ×M×D F ×D F Indicates that the point-by-point convolution calculation amount is M×N×D F ×D F , so the total computational cost of depth-level separable convolution is D k ×D k ×M×D F ×D F +M×N×D F ×D F By comparing the computational complexity of depth-level separable convolution and standard convolution, we can obtain the result calculated by the following formula.
[0120]
[0121] It can be easily observed from the above formula that in most cases, the number of output channels is relatively large. k ×D k If the convolution kernel is , then the depthwise separable convolution can reduce about times the amount of computation, thus significantly improving performance.
[0122] The construction of the Xception model is based on a premise that in the feature graph of the convolutional neural network, the relationship between channel correlation and spatial correlation can be completely decoupled. Fig.14 The network architecture of the model is shown, which consists of 36 convolutional layers and is divided into 14 parts, all of which are connected by linear residual connections, with the exception of the first and last parts. Overall, the Xception architecture is a linear stack of depth-separable convolutional layers with residual connections, which makes it easier to define and modify the architecture. This design not only speeds up the convergence of the Xception model, but also improves the accuracy of the model.
[0123] This embodiment also exemplifies the loss functions and reasoning processes that can be used in the neural convolutional network model described above. A total of three categories are introduced, namely, cross entropy loss function, contrast loss function, and binary cross entropy loss function.
[0124] The cross-entropy loss function (Cross-EntropyLoss) is defined by the following formula.
[0125]
[0126] In the formula, y i is the label value, y′ i is the predicted value.
[0127] For all samples, the loss function is defined as follows.
[0128]
[0129] In the process of binary classification, the model needs to predict only two possible scenarios, namely yes and no. For example, in this article, the model predicts two possible results: one is a monkeypox image and the other is a non-monkeypox image. For each category, a prediction probability can be obtained, denoted as p and 1-p respectively. At this time, the cross entropy is calculated using the following formula.
[0130]
[0131] In the formula, the category of sample i is represented by y i To represent, 1 represents the positive class and 0 represents the negative class. The probability that sample i is predicted as a positive class is p i .
[0132] For the case where label y is 1, that is, y i =1, 1-y i =1-1=0, then the binary cross entropy loss function L=-log(p i ). From the formula, we can know that if the positive sample is predicted to be positive, then its probability is close to 1 and the loss function is close to 0; if the probability of the positive sample being predicted to be positive is close to 0, then the loss function becomes infinite.
[0133] For the case where label y is 0, that is, y i = 0, then the binary cross entropy loss function L = -log(1-p i ), if the negative sample is predicted to be positive, then its probability is close to 1, and the loss function will increase significantly; if the probability of the negative sample being predicted to be positive is close to 0, then the loss function is close to 0.
[0134] In summary, cross entropy is used to describe the difference between two distributions. When the predicted value is close to the label value, the loss is small, and when the predicted value is far from the label value, the loss is large. This feature is conducive to model learning. The smaller the cross entropy, the closer the assumed distribution is to the true distribution, and the better the model.
[0135] As one of the dimensionality reduction learning methods, contrast loss can learn a mapping relationship so that samples of the same class but far away from the high-dimensional space are as close as possible after entering the low-dimensional space through function mapping, and samples of different classes but closer are farther away from the mapped low-dimensional space. This loss function has been widely used in the dimensionality reduction task of deep learning. Specifically, samples that are close to each other in the original space can still maintain relative proximity in the newly constructed feature space after feature extraction and dimensionality reduction. On the contrary, for samples that do not have similarity in the initial state, after completing the dimensionality reduction process, the distance between these samples in the new feature space should continue to show significant differentiation.
[0136] Contrastive Loss is calculated using the following formula.
[0137]
[0138] In the formula, It represents the Euclidean distance between the features of two samples, and P refers to the dimension of the feature. Y is the label of whether the two samples match. Y = 1 means that the two samples are similar or matched; Y = 0 means that the two samples are not similar or matched. M is the set threshold, and N is the number of samples.
[0139] As can be seen from the above formula, Contrastive Loss can also effectively measure the degree of matching between samples and can be used to train feature extraction models.
[0140] Specifically:
[0141] When y=1, it means that the samples are similar, and the loss function is That is, when samples are similar, if the feature Euclidean distance is large, it means that the model does not match similar samples well, so the loss value will be large.
[0142] When y=0, it means that the samples are not similar, and the loss function is L s =(1-Y)max(mD w ,0) 2 That is, when the samples are not similar, the smaller the Euclidean distance of the samples, the greater the loss value, so a larger parameter penalty is imposed on the model to reduce the loss value. This meets the requirements of this paper for the model.
[0143] FocalLoss is a dynamic scaling method for the binary cross entropy loss function. Its main purpose is to use the dynamic scaling factor to weaken the weight of easy-to-classify samples, thereby prompting the learning process to focus more on samples that are more difficult to identify, so as to improve the model's recognition ability for complex samples more quickly. These difficult-to-distinguish samples may be positive samples or negative samples, but they are helpful for training the network. During the training process, if there are a large number of easily distinguishable negative samples, the entire training process may revolve around these negative samples, thereby ignoring the importance of positive samples and resulting in large losses. In order to solve this problem, a specific modulation factor is introduced to specifically process those difficult-to-distinguish sample data.
[0144] FocalLoss is an improvement on cross entropy and is calculated using the following formula.
[0145] FL(p t )=-(1-p t ) γ log(p t ).
[0146] In the formula (1-p t ) γ It is called the modulation coefficient, γ is a parameter, γ≥0.
[0147] The core idea of Focalloss is to use an appropriate function to measure the contribution of difficult-to-classify and easy-to-classify samples to the overall loss, and to reduce the weight of easy-to-classify samples by adjusting the coefficients, so that the model can focus more on difficult-to-classify samples during training. It has the following two properties:
[0148] (1) When a sample is misclassified, the probability p of its true category t is small, which means that the modulation factor (1-p t ) is close to 1, the loss will not be affected; when p t When it is close to 1, it means that the samples are well classified and easy to classify. The modulation factor (1-p t ) is close to 0, resulting in a decrease in the weight of well-classified samples. Therefore, when the modulation coefficient is close to 1, the difference from the initial loss is not significant. t When it approaches 1 (which represents a correctly classified and easily classified sample), the modulation coefficient approaches 0 and has a very small impact on the overall loss.
[0149] (2) When γ = 0, Focal Loss is equivalent to the traditional cross entropy loss; as γ increases, the modulation factor will also increase accordingly. Increasing γ can enhance the influence of the modulation factor. Experimental results show that the effect is most significant when γ is 2. From an intuitive point of view, the modulation factor reduces the loss of easy-to-classify samples, expands the range of samples with low loss values, and increases the importance of misclassified samples.
[0150] The following embodiments of the present invention will further illustrate the feasibility and advancement of the present invention in conjunction with specific implementation cases.
[0151] The experimental steps can be divided into the following steps: (1) construct paired samples; (2) train the model; (3) predict unknown samples. Specifically, when constructing paired samples, sample pairs of the same category are marked as 1, and sample pairs of different categories are marked as 0. During the training process, the original training set is divided into a new training set and a validation set in a ratio of 8:2. Subsequently, the divided training set and validation set are further processed to construct paired samples. Ideally, for each non-monkeypox sample in the original training set (a total of 440 samples), there should be a paired monkeypox sample, and a total of 220 pairs of sample pairs of different categories need to be constructed. Table 5 lists the distribution of the paired data set in detail. There are 440 pairs of sample data in the training set, including 220 pairs of sample pairs belonging to the same category and 220 pairs of sample pairs of different categories. Similarly, the validation set contains 110 pairs of sample data, of which 55 pairs are sample pairs of the same category and the other 55 pairs are sample pairs of different categories. In the prediction phase, the weights of the trained model are first loaded, and an image is randomly selected from the test data set. Then, stratified sampling is performed from the original data, and two pairs of paired samples are extracted, one monkeypox image and one non-monkeypox image. Finally, these paired samples are input into the model for prediction.
[0152] Table 5 Distribution of paired datasets
[0153] Sample pairs of the same category Different categories of samples Training set 220 220 Validation set 55 55
[0154] The specific experimental settings are as follows: the training epoch parameter is set to 5 rounds, the batch size is set to 16, and the adaptive momentum estimation (Adam) is selected to optimize the model loss because it has a robust performance compared with other existing optimization algorithms. One potential advantage of Adam is that it often shows excellent performance in binary image classification. To avoid overfitting, an early termination condition is set (if the accuracy of the validation set does not increase by 0.01 for 4 rounds, it is terminated).
[0155] For the DenseNet model, three different loss functions are used: Binary Cross-Entropy Loss (BCE), Contrastive Loss (CL), and Focal Loss (FL), which have good results in training and verification. These three loss functions are trained on the same training and verification datasets. Fig.15 The loss curves and accuracy change curves of the training process show that all three loss functions have achieved satisfactory training results.
[0156] Specifically, when BCE is used as the loss function, the training loss value is 0.25 and the validation loss value is 0.33. At the same time, the training accuracy is as high as 0.99, while the accuracy on the validation set is 0.91. When the loss function is replaced by Contrastive Loss, the training loss value is reduced to 0.05 and the validation loss value is 0.10. In this case, the training accuracy is also as high as 0.99, while the validation accuracy is 0.87. When Focal Loss is used as the loss function, the training loss value is further reduced to only 0.02 and the validation loss value is 0.03. In this case, the training accuracy is 0.98 and the validation accuracy is 0.92. The results show that it is effective to choose the DenseNet model for training with these three loss functions. These loss functions each achieve competitive results in terms of loss value and accuracy indicators, demonstrating the successful performance of the model in both the training set and the validation set.
[0157] Using Xception as the pre-trained model, three loss functions, Binary Cross-Entropy Loss (BCE), Contrastive Loss (CL) and Focal Loss (FL), are used to train and verify the same training set and validation set. The results of the loss curve and accuracy change curve are as follows: Fig.16 shown.
[0158] When the loss function is BCE, the training loss value is 0.41 and the validation loss value is 0.50. The training accuracy rate reaches 0.98, while the validation accuracy rate is 0.85. When the loss function is Contrastive Loss, the training loss value is 0.16 and the validation loss value is 0.20. The training accuracy rate remains at 0.98, but the validation accuracy rate drops to 0.69. When the loss function is Focal Loss, the training loss value is only 0.02 and the validation loss value is 0.07. The training accuracy rate reaches 0.99, while the validation accuracy rate is 0.86. It is worth mentioning that the validation accuracy rates of the three loss functions all show a trend of first rising and then falling, that is, as the number of training rounds increases, the performance of the model on the validation set may decline to a certain extent. In summary, when using Xception as a pre-training model, these three different loss functions have achieved certain effects on the training and validation of the model. The BCE loss function performs well in terms of validation accuracy, while the Focal Loss loss function performs well in terms of training and validation loss values, indicating that it can effectively reduce the error. However, the Contrastive Loss loss function has a low validation accuracy and may need further optimization and adjustment.
[0159] For the experiment with the Inception pre-trained model, the experimental settings are the same as those of the first two models. Fig.17 The loss curves and accuracy change curves of different loss functions are shown. When BCE is applied as the loss function, the training loss value is 0.55, and the validation loss value is 0.54. At this time, the training accuracy reaches the highest value of 1, and the validation accuracy is 0.85. However, when Contrastive Loss is used as the loss function, the training loss value drops significantly to 0.04, and the validation loss value also drops to 0.09. The training accuracy reaches 1, and the validation accuracy increases to 0.90. When Focal Loss is used as the loss function, the training loss value is only 0.05, and the validation loss value is 0.06. The training accuracy is 0.97, and the validation accuracy is stable at 0.90.
[0160] In summary, for the Inception pre-trained model, these three different loss functions have achieved certain results in model training and verification. The BCE loss function performed well during training, but was slightly insufficient in verification accuracy. In contrast, the Contrastive Loss and Focal Loss loss functions both showed high accuracy during verification, especially the Focal Loss loss function, which achieved better results in both loss value and accuracy. It should be further noted that regardless of the loss function, the loss function value and accuracy during training showed a stable trend, which also verified the good performance of the model and the effective training effect.
[0161] In this embodiment, without determining the optimal threshold, the prediction performance of various models and loss functions on the test set is compared and analyzed using the ROC curve. Fig.18 , Fig.19 and Fig. 20 It can be seen that no matter whether DenseNet, Xception or Inception is used as the pre-training model, the three different loss functions all show excellent performance. The ROC curve shows that these models and loss functions can help the model locate a specific threshold, so that the model can achieve the best effect in distinguishing positive and negative samples, which shows that the model can accurately classify the category of a given sample. It is worth noting that when the pre-training model is DenseNet and Xception, the ROC curves of the BCE loss function and the Focal Loss loss function are closer to the upper left corner than the CL loss function, indicating that in the DenseNet and Xception models, the BCE loss function and the Focal Loss loss function are better than the CL function.
[0162] By maximizing the area under the ROC curve, this embodiment determines the optimal classification threshold of the model, as shown in Table 6. When the pre-training model is DenseNet, the AUC values obtained by the three different loss functions are all over 0.9, indicating that the classification effect of the model is good. Specifically, the AUC values obtained by the models using the BCE loss function, the Contrastive Loss loss function, and the Focal Loss function are 0.95, 0.93, and 0.95, respectively. As mentioned above, when the pre-training model is DenseNet, the models of the BCE loss function and the Focal Loss have better classification capabilities.
[0163] Table 6 AUC values and optimal thresholds of different models
[0164]
[0165] When the pre-trained model is Xception, the AUC values of the three loss functions, BCE loss function, Contrastive Loss and Focal Loss, are 0.96, 0.94 and 0.97 respectively, indicating good performance on the test set. Among them, the model with Focal Loss loss function performs better on the test set.
[0166] When the pre-trained model is Inception, the AUC values of the three loss functions are all above 0.95, indicating that the model has shown excellent performance in the unbalanced small sample monkeypox classification task in this paper. Among the three loss functions, the AUC values of the BCE loss function, Contrastive Loss, and Focal Loss are 0.96, 0.97, and 0.97, respectively, showing the efficient classification ability of the Inception model.
[0167] In general, the three pre-trained models, DenseNet, Xception, and Inception, can achieve satisfactory classification results when used with three different loss functions.
[0168] Fig.21 , Fig. 22 and Fig.23 The confusion matrices of the nine models are presented, showing the classification performance of each model when dealing with small sample class imbalanced monkeypox data. Since the number of negative case samples is larger than the number of positive case samples, the test set contains 137 negative sample images and 37 positive sample images. In the context of using DenseNet as the feature decoder, three different loss functions, BCE Loss, Contrastive Loss and Focal Loss, are used respectively. The classification results are shown in Figure 2. Fig.21 Specifically, the number of correctly classified negative samples is 129, 122, and 120, respectively, while the number of correctly classified positive samples is 33, 30, and 36, respectively. In addition, the number of false negative samples of BCE Loss, Contrastive Loss, and Focal Los is 4, 7, and 1, respectively, while the number of false positive samples is 8, 15, and 17, respectively.
[0169] Comprehensive analysis of the above results shows that when processing small sample class imbalanced monkeypox data, the BCE Loss model performs better on negative samples and produces the least false positive samples. The Contrastive Loss model has relatively poor classification performance on positive samples and produces more false negative and false positive samples. The Focal Loss model performs well on positive samples and also has good results on negative samples, producing the least false negative samples.
[0170] When Xception is used as the feature decoder, three different loss functions, BCE Loss, Contrastive Loss, and Focal Loss, are used for classification. The classification results are shown in Figure 2. Fig. 22 As shown. Specifically, for negative samples, the number of correctly classified samples of BCE Loss is 123, that of Contrastive Loss is 113, and that of Focal Loss is 123. For the classification of positive samples, when BCE Loss is used, the number of correctly classified samples is 35. When Contrastive Loss is used as the loss function, the number of correctly classified samples increases to 36, and the number of correctly classified samples using Focal Loss is also 36. In addition, the number of false negative samples of BCE Loss, Contrastive Loss, and Focal Loss are 2, 1, and 1, respectively, and the number of false positive samples are 14, 24, and 14, respectively. The number of false negative and false positive samples of Contrastive Loss is higher than that of BCE Loss and Focal Loss. BCE Loss and Focal Loss perform similarly in terms of the number of overall correct classifications, while the number of false negative and false positive samples of Contrastive Loss is higher than that of BCE Loss and Focal Loss.
[0171] In the case of Inception as the feature decoder, three different loss functions are used: BCE Loss, Contrastive Loss, and Focal Loss. The classification results are as follows: Fig.23 As shown. Specifically, the number of correctly classified negative samples is 127, 129, 123, respectively, while the number of correctly classified positive samples is 35, 33, 35, respectively. In addition, the number of false negative samples of BCELoss, Contrastive Loss, and Focal Loss are 2, 4, 2, respectively, while the number of false positive samples is 10, 8, 14, respectively. The results show that Focal Loss has the highest number of false negative samples, and it is easy to misclassify some negative samples as positive samples to a certain extent. The number of false negative and false positive samples of BCE Loss and Contrastive Loss is equal, which is lower than that of Focal Loss loss function. The overall performance of Focal Loss is relatively poor.
[0172] In order to more clearly compare the performance of the nine models on the test set, the three key indicators of accuracy, recall and F1 score are used to evaluate the classification ability of different models. The specific data are listed in Table 7.
[0173] Table 7 Performance of each model on the test set
[0174]
[0175]
[0176] It is worth noting that when the pre-training model selects DenseNet, the BCE Loss loss function can correctly classify 93% of the samples in general, with a high recall rate and F1 score. Compared with BCE Loss, the accuracy and recall rates of Contrastive Loss have decreased, and the F1 score has also decreased significantly. This may indicate that Contrastive Loss is not as effective as BCE Loss in classifying the data set. Compared with BCE Loss, Focal Loss has a slightly lower accuracy, but a higher recall rate and a relatively low F1 score, indicating that the pre-training model DenseNet has a lower predictive ability for true positive samples when using Focal Loss, which may lead to some misclassification. In summary, on the pre-training model DenseNet, BCE Loss performs better in accuracy, recall, and F1 score, while Contrastive Loss and Focal Loss perform weaker in some aspects.
[0177] When Xception is selected as the pre-trained model, the accuracy and recall of BCE Loss loss function are high, indicating that the model does a good job in finding positive samples. However, the low F1 score may indicate that the model has some problems with the classification boundary of the samples. BCE Loss has an accuracy of 0.90, a recall of 0.95, and an F1 score of 0.81. Contrastive Loss has an accuracy of 0.86, a recall of 0.97, and an F1 score of 0.74. Focal Loss has an accuracy of 0.91, a recall of 0.97, and an F1 score of 0.83. Compared with BCE Loss, the accuracy and F1 score of Contrastive Loss have decreased, while the recall rate has increased. This may mean that Contrastive Loss is better in finding true positive samples, but the overall classification accuracy of the model is relatively low. Compared with the other two loss functions, Focal Loss performs better in accuracy, recall and F1 score, which shows that on the pre-trained model Xception, Focal Loss can better handle the problem of sample imbalance in the dataset and improve the overall classification accuracy and F1 score.
[0178] When Inception is selected as the pre-trained model, the accuracy, recall, and F1 scores of BCE Loss are all high, indicating that the model performs well in the classification of both positive and negative samples. Compared with BCE Loss, the accuracy and F1 scores of Contrastive Loss are basically the same, while the recall rate is slightly lower; compared with BCE Loss, the accuracy of Focal Loss is slightly lower, and the recall rate and F1 score also decrease, indicating that Focal Loss's prediction ability for positive samples is weaker than BCE Loss. In summary, using BCE Loss on the pre-trained model Inception can obtain higher accuracy, recall, and F1 scores, and performs better than the other two loss functions on the test set.
[0179] In summary, when DenseNet is selected as the pre-training model, the BCE Loss loss function shows the best performance on the test set. Specifically, its accuracy is as high as 0.93, the recall rate is 0.89, and the F1 score is 0.85. When the pre-training model is Xception, the Focal Loss loss function has an overall classification accuracy, recall rate, and F1 score of 0.91, 0.97, and 0.83, which exceeds the other two loss functions. When the pre-training model is Inception, BCE Loss has an accuracy, recall rate, and F1 score of 0.93, 0.95, and 0.85, respectively, showing excellent performance. Comprehensively comparing different models, the Inception model using the BCE Loss loss function has the highest accuracy, recall rate, and F1 score, and performs best.
[0180] In the previous paragraph, this paper discussed the performance of twin networks using different pre-trained models on classification tasks, and pointed out that the model proposed in this paper is closely related to the selection of pre-trained models. The subsequent research of this paper will further explore the impact of different loss functions on the predicted probability distribution of twin networks based on different pre-trained models (DenseNet, Xception and Inception). Figure 24 to Figure 26 As shown in the figure, these charts use the model's prediction results as the horizontal axis, while the vertical axis reflects the proportion of samples with positive labels in the observed samples. Figure 24 to Figure 26 It clearly depicts the distribution range of the model's predicted values for different types of samples, thereby intuitively revealing the model's ability to distinguish different samples and reflecting the model's discrimination.
[0181] according to Fig.24The predicted probability distribution of different loss functions of the DenseNet model shown shows that when using the DenseNet model, when the BCE Loss loss function is used, the positive samples are concentrated in the probability range between 0.5 and 0.6, while the predicted probability interval of negative samples is between 0.2 and 0.5. This shows that the model can distinguish the two types of samples well, showing high reliability. When the Contrastive Loss loss function is used, the predicted probabilities of some positive and negative samples are concentrated around 0.5, indicating that the model cannot distinguish negative and positive samples well. When the Focal Loss loss function is used, the positive samples are concentrated in the probability range between 0.5-0.8, and the predicted probability of negative samples is between 0.0 and 0.5. The model can distinguish the two types of samples well. The predicted probability of positive samples is mainly concentrated in the area close to 0.5, reflecting that the imbalance of categories and the scarcity of data samples still have an adverse effect on the performance of the model.
[0182] according to Fig.25 It can be seen that when the pre-trained model is Xception, the prediction probability of positive samples is generally concentrated between 0.5 and 0.6; the prediction probability of negative samples is mainly distributed in the range of 0.1 to 0.4. This shows that the model can better identify and distinguish positive and negative samples and has a high degree of credibility. It should be noted that when the loss function is BCE Loss, the optimal threshold of the model is 0.46. Compared with the case where the loss function is Contrastive Loss and Focal Loss, when BCE Loss is selected, the prediction probability of positive samples is more concentrated between 0.5 and 0.6, while the prediction probability of negative samples is more distributed between 0.1 and 0.4, showing better model resolution.
[0183] according to Fig.26It shows that when using the Inception model, when the BCE Loss loss function is used, the predicted probability of positive samples is generally concentrated between 0.5 and 0.8, while the predicted probability of negative samples is concentrated between 0.0 and 0.5. This means that the model can effectively divide positive and negative samples and has great credibility. When the Contrastive Loss loss function is used, the predicted probability of positive samples is mainly concentrated in the range of 0.5 to 0.6, while the predicted probability of negative samples is mainly concentrated in the range of 0.1 to 0.5. When the Focal Loss loss function is used, the predicted probability of positive samples is concentrated in the range of 0.5 to 0.6, while the predicted probability of negative samples is concentrated between 0.4-0.5. In particular, when BCE Loss is used as the loss function, the optimal threshold is 0.48. Compared with the Contrastive Loss and Focal Loss loss functions, the predicted probability of positive samples is more distributed between 0.5 and 0.8, while the predicted probability of negative samples is more distributed between 0.0 and 0.4, showing better model discrimination ability.
[0184] In image classification tasks, the training quality of the model is often affected by the number of samples. This embodiment will explore the specific impact of using data sets of different sizes on the training quality of the proposed method to reveal the intrinsic connection between the sample size and the performance of the method of the present invention. According to the results of the previous article, when the pre-trained model is DenseNet, the classification effect is better when the loss function selects the BCE loss function; when the pre-trained model is Xception, the classification effect is better when the loss function selects Focal Loss than the other two loss functions; when the pre-trained model is Inception, the classification effect is better when the loss function is BCE. Therefore, this embodiment will study the sensitivity of different data amounts under these three model combinations.
[0185] Based on 220 pairs of samples of different categories and the same category under ideal conditions, the number of pairs of samples of the same category and different categories in the training set is reduced by 40 pairs each time, and the ratio of the training set to the validation set is always 8:2. Table 8 lists the distribution of each data set. Fig. 27 and Fig.28 The training accuracy and validation accuracy of each model for different dataset numbers are shown. As the number of samples gradually decreases, each model generally shows different degrees of overfitting, and the validation accuracy gradually decreases. It is worth noting that the Xception model shows competitive performance when the number of samples is relatively scarce, which is likely due to the efficient and streamlined design of its parameters.
[0186] Table 7 Comparison of the best performance model with other image classification models
[0187] Model Accuracy Recall F1 ResNet50 0.73 0.83 0.84 EfficientNetB0 0.75 0.88 0.85 MobileNet 0.77 0.87 0.85 Ours 0.93 0.95 0.85
[0188] In image classification with imbalanced classes, most existing technologies use undersampling or oversampling to deal with the imbalanced class problem. Undersampling can effectively reduce the adverse effects of uneven sample distribution on model learning, but its biggest drawback is that it fails to effectively utilize majority class samples; while oversampling is another extreme. Due to the small number of samples and the large similarity between data, the robustness of the model is poor.
[0189] In order to solve the problem of monkeypox image classification under the background of limited sample data and imbalanced categories, the present invention takes the small sample learning technology based on metric learning as the core, introduces transfer learning into the small sample learning framework, so that the model can effectively learn the difference information of samples of different categories and the common information of samples of the same category. In the training stage, the sample pairs in the training set are first input into the pre-training model to obtain a pair of tensors. Then a multi-layer perceptron with shared weights is used to extract the features of the input image and convert these images into vector form. Next, the similarity between the two one-dimensional vectors is calculated. In the prediction stage, a sample is first randomly selected from the test set and two samples are obtained by random stratified sampling from the training set. Then the samples from the test set are paired with the samples from the training set, and the paired sample pairs are passed to the trained network model. Finally, the similarity function is used to output the probability value of the predicted sample belonging to the positive category according to a specific calculation formula.
[0190] This example uses DenseNet, Xception and Inception models for transfer learning to automatically and effectively extract image features and combine with multi-layer perceptrons to enhance the high-dimensional information representation capabilities. In addition, under the twin neural network framework, three different loss functions, namely cross entropy loss function, Contrastive Loss and Focal Loss, are combined with three different pre-training models, and common model evaluation methods are used to compare and analyze the classification effects of different models on the unbalanced small sample monkeypox dataset.
[0191] The key to classifying monkeypox images is to avoid misjudgment as negative results, because misjudgment as negative increases the risk of infected people spreading the disease to others, leading to serious consequences. Therefore, when comparing and analyzing models, the recall rate should be given priority while ensuring the accuracy. Based on the comparative analysis of multiple classification indicators, the experimental results show that when the pre-trained model is the Inception network, the model with the binary cross entropy loss function as the pre-training loss function has the best overall performance. On the test set, the accuracy of the model reached 0.93, the recall rate was 0.95, and the F1 score was 0.85. Compared with other commonly used image classification models, the model in this paper shows better classification performance and effectively solves the classification problem of monkeypox images in unbalanced small sample backgrounds. Finally, the Grad-CAM technology is used to outline the key areas activated by the model during the classification process, which enhances the interpretability of the model in this paper.
[0192] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.
Claims
1. A method for classifying monkeypox images, characterized in that: The method comprises: Acquire a plurality of monkeypox images, and construct a monkeypox image dataset based on the plurality of monkeypox images; Constructing a twin neural network, the twin neural network comprising a pre-trained model, a multi-layer perceptron and a distance function, the pre-trained model is trained based on the monkeypox image dataset, the twin neural network is trained using the monkeypox image dataset to obtain a monkeypox image classification model, and monkeypox image classification is implemented based on the monkeypox image classification model; When the twin neural network is trained using the monkeypox image dataset, a sample to be predicted, a negative sample and a positive sample are obtained from the monkeypox image dataset, and the sample to be predicted is respectively combined with the negative sample and the positive sample to form a first sample pair and a second sample pair, and the pre-trained model generates a first feature map group and a second feature map group corresponding to the first sample pair and the second sample pair in response to the input first sample pair and second sample pair, respectively, and the first feature map group and the second feature map group are fed to the multi-layer perceptron and output a first feature map vector group and a second feature map vector group, wherein the first feature map vector group includes feature map vectors corresponding to the predicted sample and the negative sample, and the second feature map vector group includes feature map vectors corresponding to the predicted sample and the positive sample, and the similarity of two feature map vectors in the first feature map vector group and the second feature map vector group is calculated respectively by using the distance function set at the output end of the multi-layer perceptron, and a first similarity p between the sample to be predicted and the negative sample and a second similarity n between the sample to be predicted and the positive sample are obtained, and the probability that the sample to be predicted is predicted to be positive is obtained according to the first similarity p and the second similarity n, and the probability is p / (p+n).
2. The method according to claim 1, characterized in that Acquiring a plurality of monkeypox images and constructing a monkeypox image dataset according to the plurality of monkeypox images comprises: Filtering images with labels from the plurality of monkeypox images to construct a first data set; Based on the first data set, construct a first training set and a query set, wherein the first training set and the query set are disjoint, and the query set is composed of at least one monkeypox image randomly selected from various labels in the first data set; A small sample monkeypox image classification model is constructed, and the small sample monkeypox image classification model is trained using the first training set and the query set. In a training cycle, a support sample in the first training set and all query samples in the query set are used as input, and an embedding module is used to extract a support sample and all query samples to obtain a corresponding first image feature and multiple second image features. A metric function is used to evaluate the similarity between the first image feature and each second image feature, and a label corresponding to the second image feature most similar to the first image feature is used as a predicted value. A loss function is used to calculate the difference between the actual value and the predicted value of the query sample in the current training cycle, and the actual value is determined according to the label corresponding to the query sample. The gradient is calculated using a back propagation algorithm, and the model parameters of the small sample monkeypox image classification model are continuously updated by a gradient descent method to obtain a trained small sample monkeypox image classification model.
3. The method according to claim 1, characterized in that The pre-trained model is a first neural network convolution model, which includes a plurality of dense blocks and a transition layer, wherein two dense blocks are connected by one of the transition layers; The data processing flow of the dense block is shown in the following formula: In the formula, x k is the output of the kth layer in the first neural network convolutional model, H k is the nonlinear transformation function of the kth layer, {x0,x1,...,x k-1 } is the output set of layers 0 to k-1, L is the number of convolutional layers in a dense block, k is the sequence number of the convolutional layer in a dense block, x0 is the output of layer 0, and inputfeature is the input feature; The data processing flow of the transition layer is shown in the following formula: x″=avgpool(x′)=avgpool(BN(Conv(1×1)(x))), for1≤k≤L. In the formula, x is the input feature map, Conv(1×1)(x) represents the convolution operation on the input feature map, and BN represents the normalization operation. x′ is the intermediate output of the input feature map x after the transition layer, avgpool represents the average pooling operation, and x″ is the final output of the intermediate output x′ after average pooling.
4. The method according to claim 1, characterized in that The pre-trained model is a second neural network convolution model. The second neural network convolution model extracts at least two image features from the image data in response to the input image data, integrates the at least two image features, and outputs an image feature map. The data processing flow of the second neural network convolution model is expressed as follows: In the formula, Represents the average value of each neuron in the input data, Var(x (k) ) represents the standard deviation of the neuron input value in a batch of data, γ and β are parameters, represents the normalized sample, x (k) represents the data of the kth sample, γ (k) represents the scaling parameter, β (k) represents the translation parameter, and y(k) represents the output of the normalization layer.
5. The method according to claim 1, characterized in that The pre-trained model is a third neural network convolution model, which is based on an input feature map, obtains a first output feature map through DW convolution, and uses PW convolution to process the first output feature to obtain a final output feature map.
6. The method according to any one of claims 1 to 5, characterized in that The loss function used when the pre-trained model is obtained by training based on the monkeypox image dataset is a first loss function, a second loss function or a third loss function, and the first loss function is expressed as: In the formula, Loss represents the loss function value, i represents the sequence number of the sample, yi represents the category of sample i, N represents the total number of samples, and p i represents the probability that sample i is predicted to be positive, L i Represents the loss function value of sample i.
7. The method according to claim 6, characterized in that The second loss function is expressed as: In the formula, represents the Euclidean distance between sample features X1 and X2, P represents the dimension of sample features, Y represents the label of whether sample features X1 and X2 match, Y=1 represents that sample features X1 and X2 match; Y=0 represents that sample features X1 and X2 do not match, m represents the set threshold, N represents the number of samples, n represents the sample number, max represents the maximum value function, and W represents the parameters of the model; The third loss function is expressed as: FL(p t )=-(1-p t ) γ log(p t ). In the formula, (1-p t ) γ is the modulation coefficient, γ is a parameter, γ ≥ 0, p t Represents the predicted probability.
8. A monkeypox image classification device, characterized in that: The device comprises: a data acquisition module, configured to acquire a plurality of monkeypox images and construct a monkeypox image dataset according to the plurality of monkeypox images; An image classification module is configured to construct a twin neural network, wherein the twin neural network includes a pre-trained model, a multi-layer perceptron, and a distance function, wherein the pre-trained model is trained based on the monkeypox image dataset, the twin neural network is trained using the monkeypox image dataset to obtain a monkeypox image classification model, and monkeypox image classification is implemented based on the monkeypox image classification model; When the twin neural network is trained using the monkeypox image dataset, a sample to be predicted, a negative sample and a positive sample are obtained from the monkeypox image dataset, and the sample to be predicted is respectively combined with the negative sample and the positive sample to form a first sample pair and a second sample pair, and the pre-trained model generates a first feature map group and a second feature map group corresponding to the first sample pair and the second sample pair in response to the input first sample pair and second sample pair, respectively, and the first feature map group and the second feature map group are fed to the multi-layer perceptron and output a first feature map vector group and a second feature map vector group, wherein the first feature map vector group includes feature map vectors corresponding to the predicted sample and the negative sample, and the second feature map vector group includes feature map vectors corresponding to the predicted sample and the positive sample, and the similarity of two feature map vectors in the first feature map vector group and the second feature map vector group is calculated respectively by using the distance function set at the output end of the multi-layer perceptron, and a first similarity p between the sample to be predicted and the negative sample and a second similarity n between the sample to be predicted and the positive sample are obtained, and the probability that the sample to be predicted is predicted to be positive is obtained according to the first similarity p and the second similarity n, and the probability is p / (p+n).
9. A monkeypox image classification system, characterized in that: The system comprises: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 7. 10 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, perform the method according to claim 1 .
Citation Information
Patent Citations
Traffic anomaly detection method and system based on twin neural network
CN117113246A
Early Parkinson's disease intelligent screening method and system based on behavioral analysis
CN117694836A
