Deep learning training and identification method

By designing deep learning image classification and recognition network models, the problem that deep learning technology is difficult to deal with complex image recognition is solved, and the accurate recognition and classification of multi-label images is achieved, and the advanced intelligent capabilities of artificial intelligence are given.

CN119091243APending Publication Date: 2024-12-06BEIJING UNIV OF TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410078313.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-01-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Deep learning technology is difficult to have the ability to imagine and think like humans, especially in dealing with complex, untrained image classification and recognition tasks, and lacks advanced intelligent performance.

Method used

A deep learning image classification and recognition network model is designed, and pre-training and fine-tuning is performed based on advanced deep neural networks, combining transfer learning and data set adjustment, softmax layer reconstruction and sigmoid activation function are used, using Adam optimizer and Early stopping techniques are used to train high-performance image classification and recognition models to accurately identify complex multi-label images.

Benefits of technology

It realizes accurate classification and recognition of complex and untrained images, improves the intelligence and generalization capabilities of artificial intelligence, and enables it to have advanced intelligence unique to humans, such as association and thinking, breaking through the existing scope of deep learning capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure FT_1
    Figure FT_1
  • Figure FT_2
    Figure FT_2
  • Figure FT_3
    Figure FT_3
Patent Text Reader

Abstract

The invention discloses a deep learning training and identification method. An image classification and recognition network model is designed, pre-training is carried out on a big data set, network weight is obtained, transfer learning is carried out, the network model is initialized, and a pre-trained network model is generated. A training set, an evaluation set and a test set are constructed, the training set and the evaluation set only contain single-label images, and the test set only contains multi-label images. And carrying out training, evaluation and optimization on the training set and the evaluation set according to the loss function and the accuracy to obtain the network model of the invention. Calculation of the class performance of the network model is superior to experts in the image field or has statistical significance, and the performance is excellent. The multi-label images in the test set are input, a plurality of probability values are output, the network model is triggered to accurately identify and classify complex and untrained multi-label images with a certain probability, the imagination and thinking ability of the network model is endowed, and the intellectualization and generalization ability is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology and invents a deep learning training and recognition method. Background Art

[0002] For a long time, humans have been committed to developing artificial intelligence to assist or even replace humans in a large number of complicated tasks, liberating humans from complicated tasks so that they can devote themselves to more meaningful and challenging tasks and solve the problems of survival, development and self-improvement faced by humans. The development of algorithms and the accumulation of data, as well as the rapid development of computer and network hardware (the development of computing power, image processing capabilities, and massive storage) have given birth to deep learning. The emergence and development of deep learning has brought a ray of hope to artificial intelligence and solved a series of scientific, technological and engineering problems. Deep learning has made important progress in image processing, speech recognition, visual object recognition, object detection, image processing and other fields. At the same time, this technology is also booming in many basic disciplines, such as mechanics (rotating machinery fault diagnosis), physics (realistic and high-dimensional multi-physics problems and inverse problems), geophysics (shear wave splitting measurement), bioinformatics (obtaining knowledge and actionable insights from complex, high-dimensional and heterogeneous biomedical data), biology and medicine (patient classification, basic biological processes and patient treatment), chemistry and materials science (inverse molecular design), optics (object recognition), pharmaceutical research (drug discovery), neuroscience (emotion, learning and memory) and genomics (chromatin interactions). However, deep learning still faces a fundamental challenge: how to work as smartly as humans, such as having the ability to imagine and think. Summary of the invention

[0003] The present invention trains and triggers a deep learning image classification and recognition network model to imagine and think, and can accurately classify and recognize complex, untrained (unencountered) images. First, based on the advanced, optimized image processing deep neural network as the prototype of the network model of the present invention, the image classification and recognition network model of the present invention is designed. The prototype of the network model of the present invention is pre-trained on a large data set to obtain the network weights of the prototype of the image classification and recognition network model of the present invention, and transfer learning is performed. Except for the last convolution layer, batch normalization layer and fully connected layer, the other network weights of the image classification and recognition network model of the present invention are initialized using the obtained network weights to generate the pre-trained network model of image classification and recognition of the present invention, that is, fine-tuning the structure of the prototype of the image classification and recognition network model of the present invention and its network parameters. Specifically, since the softmax layer of the prototype of the image classification and recognition network model of the present invention contains an output quantity, that is, the number of predicted image classification probabilities is different from the number of classifications of the image classification and recognition network model of the present invention. Therefore, the image classification and recognition network model according to the present invention needs to adjust the number of outputs in the softmax layer according to the number of classified images, and reconstruct the softmax layer. A public image dataset or a self-collected image dataset is selected as the target image dataset, and after image processing and data cleaning, the dataset of the present invention is formed: a training set, a validation set and a test set. The data volume of the images in the training set and the validation set is distributed according to a certain ratio, and the training set and the validation set only contain single-label image data; the test set only contains multi-label image data. The pre-trained network model of the present invention is trained and fine-tuned using the training set and the validation set to obtain a trained image classification and recognition network model of the present invention, that is, the image classification and recognition network model of the present invention. The pre-trained network model of image classification and recognition of the present invention is trained and fine-tuned using the training set and the validation set: the last fully connected layer of the pre-trained network model of image classification and recognition of the present invention is used as a classifier, and sigmoid is used as the activation function of the last fully connected layer. The loss function uses binary cross entropy. Adam is selected as the optimizer. The training cycle, batch size and learning rate are set. During the learning process, due to the change in the parameters of the network model, it becomes the learning network model of image classification and recognition of the present invention. In order to avoid overfitting and save training time, an early stopping technique is implemented. After a certain period, the training stops because there is no further improvement in loss and accuracy. At this time, the image classification and recognition learning network model of the present invention has been trained and optimized, and the trained image classification and recognition network model of the present invention, that is, the image classification and recognition network model of the present invention, is obtained.

[0004] On the one hand, the class performance of the image classification and recognition network model of the present invention is calculated, such as: area under the curve, F1 number, accuracy, precision, recall rate, Kappa coefficient. When these parameters are better than the recognition level of experts in the data field or have statistical significance, it shows that the image classification and recognition network model of the present invention has very high performance. On the other hand, the image classification threshold of the image classification and recognition network model of the present invention is set to be statistically significant. If an output probability value is greater than the set image classification threshold, the image classification and recognition network model of the present invention is triggered to accurately recognize and classify such images. Input a multi-label image in the test set, the image classification and recognition network model of the present invention outputs multiple probability values, among which there are two or more probability values ​​greater than the set image classification threshold. The image classification and recognition network model of the present invention accurately recognizes the multi-label image in the test set with a certain probability, and accurately classifies and recognizes different multi-label images in the test set with different but consistent probabilities, or accurately imagines, thinks, identifies, classifies and recognizes complex, untrained multi-label images in the test set with a certain probability, which enables the imagination and thinking ability of the image classification and recognition network model of the present invention, and improves the intelligence and generalization ability of the image classification and recognition network model of the present invention. This is a significant result, as it breaks through the capabilities of existing deep learning (the forefront of artificial intelligence), which can only automatically extract and generate features, but cannot complete advanced human intelligence such as imagination and thinking. Imagination and thinking are a characteristic of humans, and even only belong to humans, and this characteristic occurs occasionally. How do you view this result?

[0005] In the embodiment, the class activation map is calculated to visualize the experimental results of the embodiment. From both positive and negative aspects, a typical case of the four cases in which the embodiment is accurately identified with a certain probability is analyzed. When accurately identifying and classifying such images, the image classification and recognition network model of the invention accurately extracts and learns the features of the two types of single-label images in the training set, and after artificially overlapping them, a synthetic image is formed, which is completely matched in space with the original image that is accurately identified in the test set; on the contrary, when such images cannot be accurately identified and classified, although the image classification and recognition network model of the invention still accurately extracts and learns the features of the two types of single-label images in the training set, after overlapping them, a synthetic image is formed, which is not matched in space with the original image that cannot be accurately identified in the test set, or is blocked from each other, or the features of one type of single-label image are obvious and the area is large, while the features of the other type of single-label image are not obvious and the area is small. This semi-quantitatively reveals that the image classification and recognition network model of the present invention can only accurately classify and recognize complex, untrained (unencountered) multi-label images in the test set with a certain probability.

[0006] Since the present invention is a major breakthrough, there is no precedent and it can only be compared to the most similar human processing of images. When humans are faced with a complex image that they have never seen before, if they are not trained by simple images that make up the complex image, it is very likely that this complex image will be recognized and classified as the image itself. Even if it has been trained with these simple images, extracting and learning (imagining) the features in the training images, it is necessary to be triggered to recall and imagine the possible components of this complex image, and further accurately identify this complex image (thinking); otherwise, the test image will be recognized as its own image or another image with a high probability (misrecognition). Compared with the situation where images are distributed in space, in the time domain, when encountering an old friend that has not been seen for many years, the brain will search, recall, and imagine frame after frame of images until an image suddenly appears, triggering the classification and recognition mechanism, and recognizing the old friend (the current image) through thinking.

[0007] In short, the class activation map and the human working mechanism can be summarized into the following conclusions. Based on the existing scientific knowledge, the only scientific and reasonable explanation is that the deep learning model of the present invention not only learns the features of a certain number of images with single labels in the training process, but also imagines and thinks about these learned features in the classification and recognition process, imagines that the complex images in the test set are composed of simple images in the training set and the validation set, and accurately thinks, identifies, classifies and recognizes the complex, untrained multi-label images in the test set with a certain probability. Therefore, the present invention enables artificial intelligence to truly possess the advanced intelligence unique to humans, such as association and thinking. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 .The present invention proposes a process of deep learning training and recognition method.

[0009] Figure 2 .Recognition results of two disease images. a, the number of images with two disease labels G&A, D&H, D&A and D&G that are accurately identified. b, the probability of images with two disease labels G&A, D&H, D&A, and D&G being accurately identified. The two disease labels G&A, D&H, D&A, and D&G represent glaucoma and age-related macular degeneration, diabetic retinopathy and hypertensive retinopathy, diabetic retinopathy and age-related macular degeneration, and diabetic retinopathy and glaucoma, respectively.

[0010] Figure 3.Accurately identify images with two disease labels, glaucoma and age-related macular degeneration, showing the original images and their class activation maps. a, Inaccurately identified series of images, image 1881_right in the first column is the original image with Glaucoma & AMD labels, the images in the second and third columns are the class activation maps of Glaucoma and AMD respectively, and the fourth column is the superimposed composite image of the second and third columns. b, Accurately identified series of images, image 2029_left in the first column is the original image with Glaucoma & AMD labels, the images in the second and third columns are the class activation maps of Glaucoma and AMD respectively, and the fourth column is the superimposed composite image of the second and third columns. c, Class activation maps of Glaucoma and AMD, and the superimposed image of these two class activation maps. Glaucoma, abbreviated G, represents glaucoma, and AMD, or abbreviated A, represents age-related macular degeneration. DETAILED DESCRIPTION

[0011] The following will be combined with the attached embodiment of the present invention Figure 1 , the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0012] like Figure 1 As shown, the present invention proposes a deep learning training and recognition method, which specifically includes the following steps: Step 1 is based on an advanced, image processing optimized deep neural network as a prototype of the network model of the present invention, and designs the image classification and recognition network model of the present invention.

[0013] Step 1 specifically includes designing the image classification and recognition network model of the present invention based on the most advanced convolutional neural network EfficientNet and transfer learning. EfficientNet is a light and efficient neural network architecture optimized for image classification. It uses efficient and direct compound coefficients to uniformly scale all depth, width and resolution dimensions within a limited resource budget, enabling the network to achieve higher performance under a fixed resource budget. There are eight basic EfficientNet networks, encoded from EfficientNet-B0 to EfficientNet-B7, suitable for most image resolutions and parameters. The depth, width and resolution increase from Efficientnet-B0 to B7 at a fixed ratio, and the number and complexity of parameters also increase. In order to balance the performance and computational budget of the actual network, the present invention selects EfficientNet-B3 as the core structure of the two-category disease classification model of the present invention to classify normal and seven types of retinal diseases.

[0014] Step 2, pre-training of the image classification and recognition network model of the present invention.

[0015] Step 2 specifically includes that training a deep convolutional neural network from scratch usually requires a large amount of labeled data. On the ImageNet dataset consisting of 1.2 million training natural images, the prototype EfficientNet-B3 of the image classification and recognition network model of the present invention is pre-trained to obtain network weights, transfer learning, and except for the last convolutional layer, batch normalization layer and fully connected layer, the other network weights of the image classification and recognition network model of the present invention are initialized using the weights obtained by pre-training to generate the pre-trained network model of image classification and recognition of the present invention, that is, fine-tuning the structure of the prototype of the image classification and recognition network model of the present invention and its network parameters. Specifically, since the softmax layer of EfficientNet-B3 contains 1000 outputs to predict the probability of 1000 image classifications. Therefore, the image classification and recognition network model according to the present invention needs to classify eight images, adjust the number of outputs in the softmax layer to eight categories, and reconstruct the softmax layer of the image classification and recognition network model of the present invention to replace the 1000 outputs in the EfficientNet-B3 network;

[0016] Step 3: Image processing and data cleaning in the target image dataset.

[0017] Step 3 specifically includes selecting the public database provided by Peking University in the global grand challenge of "International Competition on Ocular Disease Intelligent Recognition, ODIR" as the data source of the present invention. Seven types of diseases and one normal disease label are provided for each patient to be labeled, namely normal, diabetic retinopathy (DR), glaucoma, cataract, age-related macular degeneration (AMD), hypertensive retinopathy, myopia and other abnormal diseases (abnormal). The fundus image is labeled with a single disease label or two disease labels in the seven categories of diseases other than normal. The ODIR database only provides patient-level disease labels, left eye diagnostic keywords and right eye diagnostic keywords. According to the diagnostic keywords of the left eye and the right eye, eight types of diseases are assigned to each fundus image to reduce the complexity of model construction, so that the model can classify fundus diseases of a single eye. This operation modifies the patient-level diagnosis to the image-level diagnosis. Then, the fundus images with large black backgrounds were cropped to remove the black background areas by automatically reading the upper, lower, left, and right boundaries of the images. Due to the different sizes of fundus images taken by various cameras on the market, the retinal area was extracted to 299 × 299 pixels according to the detected retinal circle. Here, the Hough circle transform was used to detect the circle of the retina. In order to reduce the impact of uneven illumination on the image, the retinal fundus images were subjected to contrast-limited adaptive histogram equalization. It was ensured that the fundus images in the ODIR database met the experimental requirements. Low-quality images of the left or right eye were removed. According to the diagnostic keywords of the left and right eyes, the exclusion criteria of the images followed the following rules: (1) images with lens dust; or (2) images with low image quality; or (3) images with image offset; or (4) images with only other abnormal markers.

[0018] Step 4, constructing the data sets of the present invention: training set, validation set and test set.

[0019] Step 4 specifically includes, by manually browsing the images in the fundus image dataset, discarding 2824 completely invisible images from the 10,000 fundus images in the ODIR database, leaving 7176 images, forming an off-site test set of 996 images and a training set and field test set of 6210 images. In order to identify and classify single-class disease images and two-class disease images, training sets and test sets were constructed. 639 multi-disease images were separated from the 6210 images in the training set and field test set, and 5571 single-class disease images were retained. Images marked as other diseases were deleted, and 4429 single-class disease images were retained. There are relatively more images in the normal, diabetic retinopathy and other abnormal categories, while there are fewer images in the glaucoma, cataract, hypertensive retinopathy and myopia categories, resulting in a class imbalance problem. The number of image samples was enlarged by using oversampling technology, image rotation and technology for changing the position of lesion features on the image, generating 9704 single-class disease images, constructing training sets and validation sets with a ratio of 9:1 or 8734:970. At the same time, 996 images with single-class disease labels in the off-site test set were deleted, and 66 images with multiple diseases were retained. These 66 images and the separated 639 images together constituted 705 images with two-class disease labels, and further abandoned images labeled with other diseases to form the test set of the present invention, of which 261 images were labeled with two-class disease labels.

[0020] Step 5, using the training set and the validation set to train and fine-tune the pre-trained network model for image classification and recognition of the present invention. The image classification and recognition network model of the present invention is trained using a training set consisting of 9704 images. The input image of 299×299 pixels passes through red, green and blue channels. The last fully connected layer of the model is used as a classifier, and the sigmoid function is used as the activation function of the last fully connected layer. The loss function adopts binary cross entropy. The image classification and recognition network model of the present invention selects Adam as the optimizer of the image classification and recognition network model of the present invention. Dropout is applied to prevent overfitting. 200 epochs are designed to train the pre-trained network model for image classification and recognition of the present invention, and the batch size is set to 32. In order to avoid overfitting and save training time, the learning network model for image classification and recognition of the present invention implements an early stopping technique. After 30 epochs, the training stopped because there was no further improvement in the loss function and accuracy. The learning rate was initially set to 0.0001, and the learning rate decay strategy was when the verification accuracy was stable within 4 epochs. During the learning process, the parameters of the network model change, and the network model for image classification and recognition of the present invention becomes a learning network model. At this time, the learning network model for image classification and recognition of the present invention has completed learning and optimization, and a trained network model for image classification and recognition of the present invention is obtained, that is, the network model for image classification and recognition of the present invention.

[0021] Step 6, class performance of the image classification and recognition network model of the present invention. The area under the curve, F1 number, accuracy, precision, recall rate, and Kappa coefficient of the image classification and recognition network model of the present invention are calculated, and the area under the curve is 78.47%, the F1 score is 78.22%, the accuracy is 78.21%, the precision is 74.8%, the recall rate is 0.358, and the Kappa coefficient is 0.367. The accuracy of the single-class disease classification and recognition of the image classification and recognition network model of the present invention is higher than 70%, indicating that the classification and recognition of fundus diseases are completed with high precision; the F1 value is 78.22%, indicating that the overall performance of the image classification and recognition network model of the present invention is relatively stable; the area under the curve is 78.47%, indicating that the image classification and recognition network model of the present invention has a high classification performance. All evaluation results prove that these parameters are better than the recognition level of experts in the data field or are statistically significant, indicating that the trained image classification and recognition network model of the present invention already has very high performance.

[0022] Step 7, the imagination and thinking ability of the image classification and recognition network model of the present invention. The image classification threshold of the image classification and recognition network model of the present invention is set to be statistically significant. If an output probability value is greater than the set image classification threshold, the image classification and recognition network model of the present invention is triggered to accurately identify and classify such images. The test set consists of 261 images marked with two disease labels, which are input into the image classification and recognition network model of the present invention, and seven probability values ​​are output (a total of eight categories, in which normal conditions are not trained or output), corresponding to the seven categories of disease results in the data set of the present invention. As a result, 8 images with two diseases were successfully identified and classified. Among all 19 images with glaucoma and age-related macular degeneration labels in the test set, 1 image was successfully identified with a recognition probability of 5.26%. Among all 95 images with diabetic and hypertensive retinopathy labels in the test set, 5 images were successfully identified with a recognition probability of 5.26%. From the 25 images with diabetic and age-related macular degeneration labels in the test set, 1 image was successfully identified with a recognition probability of 4%. Of the 44 images in the test set labeled with diabetes and glaucoma, 1 image was successfully found with a recognition probability of 2.27%. Of the remaining 78 images in the test set, no images with the two disease labels could be distinguished and recognized. These recognition counts and probability results are shown in Figure 2. Therefore, the image classification and recognition network model of the present invention accurately recognizes the multi-label images in the test set with a certain probability, and accurately classifies and recognizes different multi-label images in the test set with different but consistent probabilities, or accurately imagines, thinks, identifies, classifies and recognizes the complex, untrained multi-label images in the test set with a certain probability. How do you view this result?

[0023] On the one hand, we calculated the class activation map to visualize our experimental results, where the most important feature areas are highlighted in red and yellow. From both positive and negative aspects, we analyzed a typical case of the four cases in which the embodiment accurately identified images with a certain probability, that is, accurately identifying images with two disease labels, glaucoma and age-related macular degeneration, such as Figure 3 As shown, Figure 3 a shows another series of images in misidentification, 1 image out of 18 images misidentified out of 19 images and its class activation map, Figure 3 b shows a series of images that are accurately identified, 1 image out of 19 images that is accurately identified and its class activation map. Figure 3 In a, the image in the first column is the original image where glaucoma and age-related macular degeneration are misidentified; the second and third columns are their class activation maps; the features of age-related macular degeneration and glaucoma are extracted respectively and overlapped to form the fourth column. The features of age-related macular degeneration are large and prominent, causing the model to recognize the image as a single age-related macular degeneration disease image, but unable to detect glaucoma lesions. In contrast, in Figure 3 In b, the first column gives the original classic images of glaucoma and age-related macular degeneration that are accurately identified; the second and third columns are their class activation maps; the features of age-related macular degeneration and glaucoma are extracted respectively and overlapped to form the fourth column. The features of glaucoma and age-related macular degeneration are clear, matched, and separated in space, ensuring that the model recognizes the image as an image of two diseases, glaucoma and age-related macular degeneration. Figure 3In c, the important feature areas of glaucoma and age-related macular degeneration show the patterns and textures of the activation category map in detail, where the top image is generated by the overlap of the lower left and lower right maps, and the overlap of the most important features of glaucoma and age-related macular degeneration, where the features of age-related macular degeneration and glaucoma are clear, textured and precise at separate spaces in the class activation map. As expected, if the most important features of glaucoma and age-related macular degeneration are separated, clear and textured in the class activation map, these features will be extracted and learned. In fact, the image classification and recognition network model of the present invention has discovered and recognized the features of age-related macular degeneration and glaucoma on the original test image. This result intuitively shows that the image classification and recognition network model of the present invention not only extracts and learns the disease features on the training images, as shown in the lower left and lower right images of 3c, but also recognizes two disease features on the test image, as shown in the top image.

[0024] On the other hand, the present invention is a major breakthrough, without precedent, and can only be compared to the most similar human processing of images. When humans are faced with a complex image that they have never seen before, if they are not trained by simple images that make up this complex image, it is very likely that this complex image will be recognized and classified as the image itself. Even if they have been trained with these simple images, extracting and learning (imagining) the features in the training images, they must be triggered to recall, imagine the possible components of this complex image, and further accurately recognize this complex image (thinking); in addition, the test image will be recognized as its own image or another image with a high probability (misrecognition). Compared with the situation where images are distributed in space, in the time domain, when encountering an old friend who has not seen for many years, frame by frame of images will be searched, recalled, and imagined in the brain until an image suddenly appears, triggering the classification and recognition mechanism, and recognizing the old friend (the current image) through thinking. In short, the class activation map and the human working mechanism can be attributed to the following conclusion. Based on the existing scientific knowledge, the only scientific and reasonable explanation is that the deep learning model of the present invention not only learns the features of a certain number of images with single labels in the training process, but also imagines and thinks about these learned features in the classification and recognition process, imagines that the complex images in the test set are composed of simple images in the training set and the validation set, and accurately thinks, identifies, classifies and recognizes the complex, untrained multi-label images in the test set with a certain probability. Therefore, the present invention enables artificial intelligence to truly possess the advanced intelligence unique to humans, such as association and thinking.

Claims

1. A deep learning training and recognition method, characterized in that: Step 1, based on an advanced deep neural network that optimizes image processing as a prototype of the network model of the present invention, the image classification and recognition network model of the present invention is designed. The advanced deep neural network that optimizes image processing is an efficient neural network optimized for image classification; Step 2, pre-training of the image classification and recognition network model of the present invention. The prototype of the image classification and recognition network model of the present invention is pre-trained on a large data set to obtain the network weights of the prototype of the image classification and recognition network model of the present invention, and transfer learning is performed. Except for the last convolutional layer, batch normalization layer and fully connected layer, the obtained network weights are used to initialize the other network weights of the image classification and recognition network model of the present invention to generate the pre-trained network model of image classification and recognition of the present invention, that is, fine-tune the structure of the prototype of the image classification and recognition network model of the present invention and its network parameters. Specifically, since the softmax layer of the prototype of the image classification and recognition network model of the present invention contains an output quantity, that is, the number of predicted image classification probabilities is different from the number of classifications of the image classification and recognition network model of the present invention. Therefore, the image classification and recognition network model according to the present invention needs to adjust the number of outputs in the softmax layer according to the number of classified images, and reconstruct the softmax layer; Step 3, image processing and data cleaning in the target image data set. A public image data set or a self-collected image data set is selected as the target image data set, and the data set of the present invention is formed after image processing and data cleaning. The target image with a large area of ​​black background is cropped to remove the black background area by automatically reading the upper, lower, left and right boundaries of the image. Since the image sizes in the target image data set are different, the target image is scaled with the region of interest as the center according to the characteristic structure of the target image detected by mathematical transformation to match the image classification and recognition network model of the present invention for the input image size. In order to reduce the impact of uneven lighting on the image, the target image is subjected to adaptive histogram equalization with limited contrast; Step 4, construct the data sets of the present invention: training set, validation set and test set. Manually browse the images in the target image data set, and distribute the data volume of the images in the training set and validation set according to a certain ratio, and the training set and validation set only contain single-label image data. Single-label images refer to images that only contain one type of features. Single-label images are simple images relative to multi-label images. The test set only contains multi-label image data. Multi-label images refer to images that contain two or more types of features. Multi-label images are complex images relative to single-label images, and multiple labels are composed of single labels. A large amount of target image data directly distributes the data volume of the images in the training set and validation set according to the ratio, but the number of target images is insufficient or the image data volume of each category is inconsistent, that is, the categories are unbalanced. The oversampling technology, image rotation and image translation technology are used to amplify the number of image samples, so that the number of images of various categories in the target data set is increased and the categories are balanced, and then the data volume of the images in the training set and validation set is distributed according to the ratio; Step 5, using the training set and the validation set to train and fine-tune the pre-trained network model for image classification and recognition of the present invention. The last fully connected layer of the pre-trained network model for image classification and recognition of the present invention is used as a classifier, and sigmoid is used as the activation function of the last fully connected layer. Binary cross entropy is used as the loss function. Adam is selected as the optimizer. Set the training cycle, batch size and learning rate. During the learning process, due to the change in the parameters of the network model, it becomes the learning network model for image classification and recognition of the present invention. In order to avoid overfitting and save training time, an early stopping technique is implemented. After a certain period, since there is no further improvement in loss and accuracy, the training stops. At this time, the learning network model for image classification and recognition of the present invention has completed training and optimization, and the trained image classification and recognition network model of the present invention is obtained, that is, the image classification and recognition network model of the present invention; Step 6, class performance of the image classification and recognition network model of the present invention. The area under the curve, F1 number, accuracy, precision, recall rate, and Kappa coefficient of the image classification and recognition network model of the present invention are calculated. These parameters are better than the recognition level of experts in the data field or have statistical significance, indicating that the trained image classification and recognition network model of the present invention has very high performance; Step 7, imagination and thinking ability of the image classification and recognition network model of the present invention. Set the image classification threshold of the image classification and recognition network model of the present invention to make it statistically significant. If an output probability value is greater than the set image classification threshold, the image classification and recognition network model of the present invention is triggered to accurately recognize and classify such images. Input a multi-label image in the test set, the image classification and recognition network model of the present invention outputs multiple probability values, of which there are two or more probability values ​​greater than the set image classification threshold. The image classification and recognition network model of the present invention accurately recognizes the multi-label images in the test set with a certain probability, and accurately classifies and recognizes different multi-label images in the test set with different but consistent probabilities, or accurately imagines, thinks, identifies, classifies and recognizes complex, untrained multi-label images in the test set with a certain probability.

2. A deep learning training and recognition method according to claim 1, characterized in that: Step 1 specifically includes designing the image classification and recognition network model of the present invention based on an advanced deep neural network that optimizes image processing as a prototype of the network model of the present invention. The advanced deep neural network that optimizes image processing is an efficient neural network optimized for image classification.

3. A deep learning training and recognition method according to claim 1, characterized in that: Step 2 specifically includes pre-training of the image classification and recognition network model of the present invention. The prototype of the image classification and recognition network model of the present invention is pre-trained on a large data set to obtain the network weights of the prototype of the image classification and recognition network model of the present invention, and transfer learning is performed. Except for the last convolutional layer, batch normalization layer and fully connected layer, the obtained network weights are used to initialize the other network weights of the image classification and recognition network model of the present invention to generate the pre-trained network model of image classification and recognition of the present invention, that is, fine-tune the structure of the prototype of the image classification and recognition network model of the present invention and its network parameters. Specifically, since the softmax layer of the prototype of the image classification and recognition network model of the present invention contains an output quantity, that is, the number of predicted image classification probabilities is different from the number of classifications of the image classification and recognition network model of the present invention. Therefore, the image classification and recognition network model according to the present invention needs to adjust the number of outputs in the softmax layer according to the number of classified images, and reconstruct the softmax layer.

4. A deep learning training and recognition method according to claim 1, characterized in that: Step 3 specifically includes image processing and data cleaning in the target image data set. A public image data set or a self-collected image data set is selected as the target image data set, and the data set of the present invention is formed after image processing and data cleaning. By automatically reading the upper, lower, left and right boundaries of the image, the target image with a large area of ​​black background is cropped to remove the black background area. Since the image sizes in the target image data set are different, the characteristic structure of the target image detected by mathematical transformation is used as the region of interest, and the target image is scaled with the region of interest as the center to match the image classification and recognition network model of the present invention for the input image size. In order to reduce the impact of uneven lighting on the image, the target image is subjected to adaptive histogram equalization with limited contrast.

5. A deep learning training and recognition method according to claim 1, characterized in that: Step 4 specifically includes constructing the data sets of the present invention: training set, validation set and test set. Manually browse the images in the target image data set, distribute the data volume of the images in the training set and validation set according to a certain ratio, and the training set and validation set images only contain single-label image data. Single-label images refer to images containing only one type of features. Single-label images are simple images relative to multi-label images.

6. A deep learning training and recognition method according to claim 1, characterized in that: Step 4 specifically includes that the test set only contains multi-label image data. Multi-label images refer to images containing two or more types of features. Multi-label images are complex images relative to single-label images, and multi-label images are composed of single labels.

7. A deep learning training and recognition method according to claim 1, characterized in that: Step 4 specifically includes directly allocating a large amount of target image data to the data volume of the training set and the validation set images in proportion.

8. A deep learning training and recognition method according to claim 1, characterized in that: Step 4 specifically includes: if the number of target images is insufficient or the amount of image data of each category is inconsistent, that is, the categories are unbalanced, the number of image samples is enlarged by using oversampling technology, image rotation and image translation technology, so that the number of images of various categories in the target data set is increased and the categories are balanced, and then the data volume of the training set and the verification set images is allocated in proportion.

9. A deep learning training and recognition method according to claim 1, characterized in that: Step 5 specifically includes training and fine-tuning the pre-trained network model of image classification and recognition of the present invention using a training set and a validation set. The last fully connected layer of the pre-trained network model of image classification and recognition of the present invention is used as a classifier, and sigmoid is used as the activation function of the last fully connected layer. The loss function adopts binary cross entropy. Adam is selected as the optimizer. Set the training cycle, batch size and learning rate. During the learning process, due to the change in the parameters of the network model, it becomes the learning network model of image classification and recognition of the present invention. In order to avoid overfitting and save training time, an early stopping technique is implemented. After a certain period, due to no further improvement in loss and accuracy, the training stops. At this time, the learning network model of image classification and recognition of the present invention has completed training and optimization, and the trained image classification and recognition network model of the present invention, that is, the image classification and recognition network model of the present invention, is obtained.

10. A deep learning training and recognition method according to claim 1, characterized in that: Step 6 specifically includes the class performance of the image classification and recognition network model of the present invention. The area under the curve, F1 number, accuracy, precision, recall rate, and Kappa coefficient of the image classification and recognition network model of the present invention are calculated. These parameters are better than the recognition level of experts in the data field or are statistically significant, indicating that the trained image classification and recognition network model of the present invention has very high performance.

11. A deep learning training and recognition method according to claim 1, characterized in that: Step 7 specifically includes the imagination and thinking ability of the image classification and recognition network model of the present invention. The image classification threshold of the image classification and recognition network model of the present invention is set to be statistically significant. If an output probability value is greater than the set image classification threshold, the image classification and recognition network model of the present invention is triggered to accurately recognize and classify such images. A multi-label image in the test set is input, and the image classification and recognition network model of the present invention outputs multiple probability values, of which two or more probability values ​​are greater than the set image classification threshold. The image classification and recognition network model of the present invention accurately recognizes the multi-label images in the test set with a certain probability, and accurately classifies and recognizes different multi-label images in the test set with different but consistent probabilities, or accurately imagines, thinks, identifies, classifies and recognizes complex, untrained multi-label images in the test set with a certain probability.

Citation Information

Cited By

  • Evaluating and executing method for model and electronic device

    TWI934668B