A method for identifying oral mucosal diseases based on deep learning and multi-feature fusion
Through deep learning multi-feature fusion method, combined with traditional feature extraction and Efficientnet model, the problems of high misdiagnosis rate and long diagnosis time in the diagnosis of oral mucosal diseases are solved, and the accurate identification of oral white spots, lichen planus, oral cancer, etc. are achieved. , improves diagnostic efficiency and accuracy.
Patent Information
- Application Number
- CN202111300552.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-11-04
AI Technical Summary
The existing diagnostic methods for oral mucosal diseases have problems with high misdiagnosis rate, long diagnosis time and low efficiency. Especially oral white spots, lichen planus and oral cancer are similar in color and characteristics, and are difficult to accurately identify.
Using a multi-feature fusion method based on deep learning, the feature extraction of grayscale symbiosis matrix (GLCM), HOG, HSV combined with Efficientnet network model, combined with traditional classifiers such as SVM and random forests for recognition, feature selection and dimensionality reduction are performed to improve recognition Accuracy and efficiency.
Accurate identification of oral white spots, lichen planus, oral cancer and recurrent oral ulcers has been achieved, the diagnostic efficiency and accuracy have been improved, the shortcomings of traditional methods have been overcome, and the misdiagnosis rate has been reduced.
Smart Images

Figure CN114494106B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition, classification and counting, and in particular to an oral mucosal disease recognition method based on deep learning and multi-feature fusion. Background Art
[0002] With the development of modern society and the improvement of people's living standards, oral health has received more and more attention from the public. Oral disease is one of the most common diseases in my country. With the increase in population and the aging of the population, the number of patients is also increasing.
[0003] Oral mucosal diseases have the characteristics of large intra-class differences and high inter-class similarities. They are usually difficult to distinguish during disease diagnosis, and even professional doctors sometimes cannot make an accurate diagnosis. Traditional oral disease diagnosis methods rely on doctors' accumulated experience to make preliminary judgments, and then perform biopsies based on different situations, extract samples and send them to the pathology department for further diagnosis. This method takes a long time to diagnose and is inefficient, greatly increasing the length of treatment for patients and the economic burden on patients. It is greatly affected by doctors' subjective factors and is prone to misdiagnosis.
[0004] Therefore, it is urgent to study a method to identify oral mucosal diseases. With the rapid development of deep learning, the combination of medical and artificial intelligence has been widely used. The widespread application of medical and artificial intelligence can greatly reduce the workload of hospitals, reduce the work pressure of doctors in diagnosing a large number of patients, shorten the diagnosis time, improve the efficiency of clinical diagnosis and make the diagnosis accuracy more stable, reduce misdiagnosis and mistreatment, increase the enthusiasm of patients for treatment, detect and treat the disease in the early stage, and reduce the burden of personal, family and social medical insurance. In addition, since oral leukoplakia and oral lichen planus are precancerous lesions, they are similar to oral cancer in color, characteristics, and site of onset, which brings greater challenges to identification.
[0005] Because oral cancer is similar to oral leukoplakia and lichen planus, and their features are difficult to distinguish, it is difficult to achieve good results using only deep learning methods. Therefore, the research plan was redesigned for the identification of oral leukoplakia, oral lichen planus, oral cancer, and recurrent oral ulcers. Summary of the invention
[0006] In view of this, the present invention provides a method for identifying oral mucosal diseases based on deep learning multi-feature fusion, which utilizes an improved deep learning multi-feature fusion algorithm to achieve more accurate classification of oral mucosal diseases and faster classification speed.
[0007] In order to solve the problems existing in the prior art, the technical solution adopted by the present invention is:
[0008] A method for identifying oral mucosal diseases based on deep learning multi-feature fusion, characterized in that the method steps are:
[0009] 1) Professional doctors use cameras to take pictures under white light to collect images of oral mucosal diseases to form a training data set;
[0010] 2) Preprocess the collected images, perform data enhancement on the images, use five data enhancement algorithms, and divide the amplified data set into training set and test set in a ratio of 7:3;
[0011] 3) Traditional feature extraction: Gray-level co-occurrence matrix (GLCM) algorithm is used to extract texture features of oral diseases, and the underlying features of shape and color are extracted from HOG and HSV images using a neural network model;
[0012] 4) Use the Efficientnet network model to extract high-level features of RGB images;
[0013] 5) Feature selection: The low-level features of color, shape, and texture features obtained are combined with the high-level features of deep learning to select representative feature vectors and reduce the dimension of the features;
[0014] 6) Add traditional classifiers such as SVM, random forest, naive Bayes, etc. to the features selected in step 5) for classification, identification and test verification;
[0015] 7) Use the test set images divided in step 2) to test the models trained by different classifiers in step 6).
[0016] Further, the specific method of step 3 is:
[0017] 3.1) Gray-level co-occurrence matrix: Four feature quantities, namely contrast, energy, entropy and correlation, are used to describe the texture characteristics of the image.
[0018] Contrast expression:
[0019] Energy expression:
[0020] Entropy expression:
[0021] Correlation expression:
[0022] The gray-level co-occurrence matrix is used to extract four feature quantities, namely, texture energy, contrast, texture entropy and correlation, at four directions of 0°, 45°, 90° and 135°. In addition, a total of 40 feature vectors, including difference, homogeneity, covariance, texture mean and inverse difference moment, are also extracted.
[0023] 3.2) HOG shape features: Build a neural network model to extract HOG shape features. The HOG feature calculation formula is:
[0024] G x (x,y)=H(x+1,y)-H(x-1,y)
[0025] G y (x,y)=H(x,y+1)-H(x,y-1)
[0026] 3.3) HSV color feature: The oral image is converted from RGB color space to HSV color space. The color feature vector is obtained through the convolutional neural network constructed in step 3.2. The HSV conversion formula is:
[0027]
[0028]
[0029] V=MAX
[0030] Furthermore, the specific method of step 4 is: deep learning feature extraction: use RGB images to extract and represent color images through different color components. The preprocessed oral data set, that is, the RGB image, is directly used as the input image of the convolutional neural network, and the EfficientnetB0 network structure is used to obtain the deep learning feature vector.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] 1) The method of the present invention can identify oral leukoplakia, oral lichen planus, oral cancer and recurrent oral ulcers. The network uses the Efficientnet model to extract high-level features, and uses HSV, Hog and gray-level co-occurrence matrix GLCM to extract the underlying features of color, shape and texture of oral diseases respectively. The Pearson coefficient is combined with the random forest algorithm for feature selection, and the features with a greater relationship with the target value are selected. At the same time, feature selection can also reduce the dimension of the feature to prevent model overfitting. Finally, traditional classifiers such as SVM, RF, NB are used for classification and recognition.
[0033] 2) The method of the present invention combines two different types of features to greatly enrich the representativeness of oral features, which not only overcomes the shortcoming that traditional image features cannot fully reflect the information of the lesion area, but also makes up for the disadvantage that the convolutional neural network classification effect is poor when the sample data volume is small and insufficient.
[0034] 3) The present invention combines deep learning methods with traditional feature extraction methods to improve the efficiency of diagnosis and the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 The sample image extracted using GLCM texture features in the present invention;
[0036] Figure 2 It is the GLCM features of the four oral images at 0° direction of the present invention;
[0037] Figure 3 These are the four HOG shape features of oral images of the present invention;
[0038] Figure 4 The sample image extracted using HOG shape features in the present invention;
[0039] Figure 5 These are four oral cavity image HSV images of the present invention;
[0040] Figure 6 This is a sample image extracted using HSV color features in the present invention;
[0041] Figure 7 Sample graph for extracting features using deep learning for the present invention;
[0042] Figure 8 Two structural diagrams of feature selection are used for the present invention;
[0043] Fig. 9 It is the Pearson correlation coefficient diagram calculated in the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0045] This embodiment provides a method for identifying oral mucosal diseases based on deep learning multi-feature fusion, the steps of which are:
[0046] Step 1. A professional doctor took photos with a Canon EOS60D camera under white light, and collected four types of oral mucosal disease images, including oral leukoplakia, oral lichen planus, oral cancer, and recurrent oral ulcers. The total number of collected images was 1,125, including 271 oral leukoplakia, 387 oral lichen planus, 255 oral cancer, and 212 recurrent oral ulcers. The format was PNG, and the resolution was 5184×3456. All image data were collected after the case was confirmed by the dentist, and these image data were used for subsequent model training and testing;
[0047] Step 2. Preprocess the collected images, including removing noise data and abnormal data, performing data enhancement on the images, using five data enhancement algorithms, and dividing the amplified data set into a training set and a test set in a ratio of 7:3;
[0048] Step 2.1: Clean the data and delete the data with unclear pathological features and a lot of noise in the oral cavity, including the doctor's medical instruments, saliva and blood in the patient's oral cavity, etc.
[0049] Step 2.2 In order to balance the data samples, oral cancer, recurrent oral ulcers and oral leukoplakia were randomly oversampled, oral lichen planus was partially downsampled, and some similar data were deleted;
[0050] Step 2.3 uses the Opencv open source library to perform five data augmentations, including rotating the image by 90°, increasing contrast, random cropping, horizontal flipping, and adding Gaussian noise. All the augmented data are scaled to 224×224, and then the data is randomly divided into training and test sets according to a 7:3 ratio. The training set is used to train the model parameters, and the test set is used to test and verify the model.
[0051] Step 3. Traditional feature extraction: Gray-level co-occurrence matrix (GLCM) algorithm is used to extract texture features of four oral diseases, and the underlying features of shape and color are extracted from HOG and HSV images using a neural network model.
[0052] Step 3.1 Figure 1 As shown, GLCM texture features are used to extract a 40-dimensional feature vector;
[0053] Gray-level co-occurrence matrix: Four characteristic quantities, namely contrast, energy, entropy and correlation, are commonly used to describe the texture characteristics of an image. In the formula, P(i, j) is the element value of the gray-level co-occurrence matrix.
[0054] Contrast expression:
[0055] Energy expression:
[0056] Entropy expression:
[0057] Correlation expression:
[0058] The gray-level co-occurrence matrix is used to extract the above four feature quantities, texture energy, contrast, texture entropy and correlation, at four directions of 0°, 45°, 90° and 135°. In addition, a total of 40 feature vectors including difference, homogeneity, covariance, texture mean and inverse difference moment are extracted. Figure 2Shown are the GLCM features of four oral images at 0° direction.
[0059] Step 3.2 calculates the HOG features and converts them into images, such as Figure 3 As shown, the HOG feature calculation formula is as follows: x (x, y) represents the horizontal gradient of the pixel point (x, y), G x (x, y) represents the vertical gradient of the pixel point (x, y), and H(x, y) represents the pixel value.
[0060] G x (x,y)=H(x+1,y)-H(x-1,y)
[0061] G x (x,y)=H(x,y+1)-H(x,y-1)
[0062] Step 3.3 uses a neural network model to extract a 128-dimensional feature vector from the HOG shape feature, as shown in Figure 4, which is the HOG shape feature of four oral images of the present invention. The first layer of the self-built neural network model is a convolution layer: 64 convolution kernels are set, the size is 5×5, the step size is 1, the padding parameter is set to the same, and the activation function is ReLU; the second layer adopts a maximum pooling layer, and the pooling window size is 2×2; the third layer is connected to the convolution layer and 128 convolution kernels are set, the size is 5×5, the step size is 1, the padding parameter is set to the same, and the activation function is ReLU; the fourth layer is connected to the maximum pooling layer again, and the window size is set to 2×2; the fifth layer is a Flatten layer, which flattens the feature parameters; the sixth layer is a fully connected layer, and the number of neurons is set to 128 to obtain a 128-dimensional feature vector.
[0063] Step 3.4 HSV color feature: Convert the oral image from RGB color space to HSV color space, such as Figure 6 As shown, four oral images of the present invention and the corresponding HSV images, the HSV conversion formula is, red (R), green (G), blue (B) represent the size of the pixel values of the three channels, MAX is the size of the maximum value of the three channels, MIN is the size of the minimum value of the three channels.
[0064]
[0065]
[0066] V=MAX
[0067] Step 3.5: The self-constructed convolutional neural network is obtained through step 3.3. From the HSV image obtained in step 3.4, a 128-dimensional color feature vector can be obtained from the HSV image using the convolutional neural network model, as shown in Figure 6 shown.
[0068] Step 4. Deep learning feature extraction: RGB images are used for extraction. RGB images represent the colors of three channels: red (R), green, and blue (B). Color images are represented by different color components. The preprocessed oral data set, namely RGB images, is directly used as the input image of the convolutional neural network. As shown in Figure 7, the EfficientnetB0 network structure is used. The first layer is the convolution layer, and 64 convolution kernels are set; the second layer is the EfficientnetBO layer, which extracts deep features; the third layer is the convolution layer, and 128 convolution kernels are set; the fourth layer is the maximum pooling layer; the fifth layer is the Flatten layer, which flattens the feature parameters; a total of 512-dimensional feature vectors can be obtained.
[0069] Table 1 is the network structure of EfficientNetB0
[0070]
[0071] Step 5. Figure 8 As shown, the obtained 808-dimensional feature vector is first subjected to Pearson coefficient feature selection, and then to random forest feature selection.
[0072] Step 5.1 Pearson correlation coefficient: It is a method to measure the correlation between random variables X and Y. The range of the correlation coefficient is [-1, 1]. The larger the absolute value of the correlation coefficient, the higher the correlation between X and Y. When X and Y are linearly correlated, the correlation coefficient takes the value of 1 (positive linear correlation) or -1 (negative linear correlation). The formula is as follows:
[0073]
[0074] Where Cov represents covariance, E is the mathematical expectation, and D is the variance.
[0075] Usually, the correlation strength of variables is judged by the following value range:
[0076] Correlation coefficient: 0.8-1.0 is a very strong correlation;
[0077] 0.6-0.8 is a strong correlation;
[0078] 0.4-0.6 is considered a moderate correlation;
[0079] 0.2-0.4 is a weak correlation;
[0080] 0.0-0.2 is very weakly correlated or has no correlation;
[0081] The Pearson correlation coefficient of the 808-dimensional feature vector is first calculated, and then the threshold is set to 0.4 to filter out feature vectors with medium correlation or above. Fig. 9 It calculates the Pearson coefficient between 10 random eigenvectors and the target value target.
[0082] Step 5.2 Random Forest: It is an ensemble learning algorithm based on decision trees. It is easy to implement and has low computational overhead. It performs well in both classification and regression problems. When there are too many features in a data set, the model is prone to overfitting. In this case, a random forest can be used to generate a data set of the importance of each feature after training. Using this data set, a threshold is determined to select some features that are most helpful for model training. After screening out important variables, the model is trained to improve the generalization ability of the model.
[0083] For each decision tree, select the corresponding out-of-bag data (OOB) to calculate the out-of-bag data error, denoted as err OOB1 The so-called out-of-bag data refers to the fact that each time a decision tree is built, a data set is obtained through repeated sampling for training the decision tree. At this time, about 1 / 3 of the data is not used and does not participate in the establishment of the decision tree. This part of the data can be used to evaluate the performance of the decision tree and calculate the prediction error rate of the model, which is called out-of-bag data.
[0084] Step 5.3 randomly adds noise interference to the feature X of all samples of the out-of-bag data OOB in step 5.2 (which can randomly change the value of the sample at feature X), and calculates the out-of-bag data error again, which is recorded as
[0085] err OOB2 .
[0086] Step 5.4 uses the out-of-bag data errors of steps 5.2 and 5.3 to calculate the importance of the feature, assuming that there are
[0087]
[0088] If there are N trees, the importance formula of feature X is as follows:
[0089] The reason why this formula can explain the importance of the feature is that if random noise is added, the accuracy of the out-of-bag data will drop significantly, indicating that this feature has a great impact on the prediction results of the sample, and thus indicates that the importance is relatively high. After feature selection, a total of 478 representative features were selected.
[0090] Step 6. Add traditional classifiers such as SVM, random forest, and naive Bayes to the features selected in step 5, and use the training set images divided in step 2.3 to train the model;
[0091] Step 7. Use the test set images divided in step 2.3 to test and verify the models trained by different classifiers in step 6.
[0092] Use indicators such as:
[0093] The evaluation indicators include accuracy (Acc), sensitivity (Sen), and specificity.
[0094] Four different performance indicators, including Specificity, Spe, and Area Under The Curve (AUC), are used to fully evaluate the model as a whole.
[0095]
[0096]
[0097]
[0098] Wherein, TP is the number of cases correctly identified as positive examples, FP is the number of cases incorrectly identified as positive examples, TN is the number of cases correctly identified as negative examples, and FN is the number of cases incorrectly identified as negative examples.
[0099] Table 2 shows the experimental results of oral recognition comparison between traditional features and deep learning features.
[0100] Training Method Accuracy / % Sensitivity / % Specificity / % AUC / % Traditional features + NB 87.69 83.18 89.44 94.19 Traditional features + RF 89.53 87.42 91.06 94.89 Traditional features + KNN 88.75 82.32 92.03 95.04 Traditional features + LDA 89.72 86.22 92.12 95.25 Traditional features + SVM 89.95 86.67 92.59 96.02 <![CDATA[ EfficientNet ]]> 89.35 91.87 89.79 96.79 Combined features + NB 90.35 84.89 92.11 94.99 Combined features + RF 91.75 90.12 91.35 96.53 Combined features + KNN 89.81 87.62 92.58 95.11 Combined features + LDA 89.88 87.46 89.55 95.46 Combined features + SVM 92.98 90.02 96.06 98.09
[0101] In summary, NB is a naive Bayes classifier, KNN is a K nearest neighbor classifier, LDA is a linear discriminant classifier, and the traditional features are a fusion of color, shape, and texture features, totaling 296 dimensions. The combined features are features that are a fusion of traditional features and deep learning features and have undergone feature selection. The combined features perform well in the random forest RF classifier and SVM, achieving classification accuracies of 91.75% and 94.98% respectively. Compared with the traditional image feature classification method, the various evaluation indicators of the classification method that integrates deep learning high-level features are significantly improved, proving that the Efficientnet model extracts high-level features that can effectively reflect the information of the lesion area, thereby further improving the classification accuracy of the four oral diseases. The present invention combines traditional features and deep learning features, which greatly enriches the representativeness of oral features and significantly improves the accuracy of the classification of the four oral diseases.
[0102] The content of the present invention is not limited to the embodiments listed, and any equivalent changes made to the technical solution of the present invention by ordinary technicians in this field after reading the description of the present invention are covered by the claims of the present invention.
Claims
1. A method for identifying oral mucosal diseases based on deep learning multi-feature fusion, characterized by: The method steps are: 1) Professional doctors use cameras to take photos under white light to collect images of oral mucosal diseases to form a training data set; 2) Preprocess the collected images, perform data enhancement on the images, use five data enhancement algorithms, and divide the amplified data set into a training set and a test set in a ratio of 7:3; 3) Traditional feature extraction: Gray-level co-occurrence matrix (GLCM) algorithm is used to extract texture features of oral diseases, and the underlying features of shape and color are extracted from HOG and HSV images using a neural network model; 4) Use the Efficientnet network model to extract high-level features of RGB images; 5) Feature selection: The low-level features of color, shape, and texture features obtained are combined with the high-level features of deep learning to select representative feature vectors and reduce the dimension of the features; 6) Add SVM, random forest, and naive Bayes traditional classifiers to the features selected in step 5) for classification, identification, and test verification; 7) Using the test set images divided in step 2) to test the models trained by different classifiers in step 6); The specific method of step 3) is: 3.1) Gray-level co-occurrence matrix: Contrast, energy, entropy, and correlation are used to describe the texture characteristics of the image. Contrast expression: Energy expression: Entropy expression: Correlation expression: The gray-level co-occurrence matrix is used to extract four feature quantities, namely, texture energy, contrast, texture entropy and correlation, at four directions of 0°, 45°, 90° and 135°. In addition, a total of 40 feature vectors, including difference, homogeneity, covariance, texture mean and inverse difference moment, are also extracted. 3.2) HOG shape features: Build a neural network model to extract HOG shape features. The HOG feature calculation formula is: G x (x,y)=H(x+1,y)-H(x-1,y) G y (x,y)=H(x,y+1)-H(x,y-1) 3.3) HSV color feature: The oral image is converted from RGB color space to HSV color space. The color feature vector is obtained through the convolutional neural network constructed in step 3.
2. The HSV conversion formula is:
2. The method for identifying oral mucosal diseases based on deep learning and multi-feature fusion according to claim 1, characterized in that: The specific method of step 4) is: deep learning feature extraction: use RGB images to extract and represent color images through different color components. The preprocessed oral data set, i.e., RGB images, is directly used as the input image of the convolutional neural network, and the EfficientnetB0 network structure is used to obtain the deep learning feature vector.
Citation Information
Patent Citations
The invention also discloses application of saliva metabolism marker in early diagnosis of oral lichen planus
CN112201344A
Oral cavity epithelium abnormal hyperplasia distinguishing and grading equipment and system based on deep learning
CN113095376A
Oral mucosal disease diagnostic support system, method, and program
JP6935663B1
