Coral classification method fusing depth data augmentation and transfer learning

Abstract

The present application relates to underwater scene recognition and classification technical field, specifically to a kind of coral classification method of fusing depth data enhancement and transfer learning, comprising the following steps: network structure building;Image preprocessing;Calculate the F1-score of each category on original training set;Original training set is deepened data enhanced using DeepSMOTE-F1 to obtain new sample;Using transfer learning to initialize network parameters, obtain the pre-training model of coral classification;The parameters of pre-training model are retrained on original sample and new generation sample, and the final model of coral classification is obtained;Model test and result analysis.The present application realizes the targeted data enhancement of each category sample, so that the model can learn the characteristics of each class more fully, thereby solving the problem of data set imbalance in coral classification task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater scene recognition and classification, in particular to a coral classification method fusing deep data enhancement and transfer learning. BACKGROUND

[0002] Coral reef ecosystems play a key role in maintaining the health and stability of marine ecosystems, achieving sustainable development of marine resources, protecting coastlines, providing resources, and supporting national defense construction. As the main builders of coral reef ecosystems, the survival of reef-building corals is closely related to the health of coral reef ecosystems. However, due to climate change, human activities, and threats from coral diseases and pests, about 36% of reef-building coral species are at risk of extinction. Therefore, accurately understanding the species of reef-building corals is crucial for monitoring the health of coral reef ecosystems. In recent years, with the development of image acquisition technology, such as Remotely Operated Vehicle (ROV) and Autonomous Underwater Vehicles (AUV), the number of in-situ collected coral images has greatly increased, providing a data basis for coral classification research. Traditional coral classification methods mainly rely on coral ecologists to manually complete the classification based on local ecosystem knowledge and related software. First, this method is tedious and time-consuming, requiring a large amount of human resources and time. Second, subjective factors are inevitably introduced in the identification and classification process, lacking consistency and objectivity. Finally, a large amount of coral image data is not utilized, resulting in information loss. Therefore, there is an increasing demand for automated coral classification and identification technology.

[0003] Traditional coral classification methods usually use image classification methods based on shallow machine learning. This method designs a feature extraction function suitable for the characteristics of the data set, uses the extracted features to train the classifier, and realizes image classification. However, this method has several problems: first, the feature extraction function is only suitable for a specific data set, resulting in weak generalization ability and poor robustness of the classification model; second, the extracted features can only represent specific physical meanings such as color, texture, and shape, and cannot represent higher-level semantic information; finally, although the designed function is based on visual neural theory, it is still limited by human experience. Therefore, the automatic method based on traditional shallow machine learning performs poorly in coral classification.

[0004] Although deep learning has shown superior performance in image classification, there are still some challenges in the study of coral classification. On the one hand, there is an imbalance in the number of samples of different coral species in existing data sets, which has a great influence on the image classification effect. The classification model tries to minimize the overall classification error during training and implicitly assigns the same error classification cost to all classes, causing the model to tend to correctly classify classes with more samples. In this case, the characteristics of classes with fewer samples cannot be fully learned, which can easily lead to misclassification. However, in the real world, more attention is paid to classes with fewer samples in some cases. For example, in the Caribbean region, the ecological role of the endangered species Acropora palmata (a species of Acropora) is difficult to replace. Due to the relatively low diversity of Acropora in this region, when the number of Acropora palmata decreases, it is difficult for other species to quickly reproduce and regrow to restore the ecological balance. On the other hand, the existing deep learning algorithm needs to be improved for coral classification and identification. First, corals have intraclass diversity and interclass similarity. This is a manifestation of corals adapting to different environments and ecological pressures. Through intraclass diversity, the same coral can develop multiple forms and characteristics to enhance its adaptability according to the needs of different environments; interclass similarity allows different species of corals to compete and coexist in environments that share resources. Second, researchers take photos of corals from different angles (elevation, depression, left, right, etc.) and different distances between the camera lens and the coral, resulting in differences in the angle and outline of the coral presented in the photos. Finally, due to environmental factors such as seawater turbidity, light intensity, and seawater refraction, as well as the properties of the camera itself, the quality of the photos is not high, and the details of the corals in the images are not clear enough. These factors make the existing coral data set insufficient to fully cover the diversity of corals in the real world. Therefore, the deep learning algorithm cannot fully learn the various characteristics of corals, resulting in insufficient classification and identification accuracy. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a coral classification method combining deep data enhancement and transfer learning.

[0006] To achieve the above object, the present application is implemented by the following technical scheme: a coral classification method combining deep data enhancement and transfer learning, comprising the following steps: S01: network structure building, modifying the ResNet-50 structure and using it as the backbone network of the model;

[0007] S02: image preprocessing, adjusting the size of the coral image to 128x128 pixels and normalizing the image;

[0008] S03: Calculate the F1-score of each class on the original training set,

[0009] Train the network without parameter initialization on the training set and calculate the F1-score of each class;

[0010]

[0011] S04: Deep data augmentation of the original training set using DeepSMOTE-F1 to obtain new samples;

[0012] S05: Use transfer learning to initialize network parameters to obtain a pre-trained model for coral classification;

[0013] S06: Re-train the parameters of the pre-trained model on the original samples and newly generated samples to obtain the final model for coral classification;

[0014] S07: Model testing and result analysis.

[0015] Preferably, in S01, the original fully connected layer is replaced by two fully connected layers with different structures, and an activation function is connected after each of them. The specific structure is that the first fully connected layer contains 512 neurons and is connected with a ReLU activation function, and the second fully connected layer has the same number of neurons as the number of classes in the dataset, and finally, this layer is connected with a Softmax activation function. The introduction of activation function can increase the nonlinear fitting ability of the network.

[0016] Preferably, in S04, DeepSMOTE-F1 uses the backbone network of Generative Adversarial Networks (GAN) as the encoder and decoder, and combines with Synthetic Minority Over-sampling Technique (SMOTE) to realize deep enhancement of image data.

[0017] Preferably, in S04, the image is first mapped to a high-dimensional feature space by the encoder, then the features of the image are sampled by SMOTE, and finally the new feature vectors are decoded into new images by the decoder; the specific steps of image feature sampling are as follows: first, for each sample x in a certain class, calculate the distance from it to all samples in the class sample set based on the Euclidean distance, and obtain its k-nearest neighbors; then, for sample x, randomly select 1 sample from its k-nearest neighbors as the nearest neighbor Finally, according to formula (2), a new sample x is constructed new where the coefficient of formula (2) is called the scaling factor, and its value is random and between 0 and 1, which is used to determine the proportion of the new sample x newWith x and similarity

[0018]

[0019] Preferably, in S04, to avoid misclassifying this type of coral as another type (FN) and misclassifying other types of coral as this type (FP), the F1-score takes into account both FN and FP cases by assigning equal weight to recall and precision, as shown in formulas (2)-(3).

[0020]

[0021]

[0022] Recall represents the proportion of correctly classified positive samples out of the actual positive samples, while Precision represents the proportion of correctly classified positive samples out of the predicted positive samples. Both describe the model's performance from different dimensions.

[0023] To address the difficulty in determining the number of samples for DeepSMOTE, an F1-score-sensitive formula for calculating the number of samples is proposed (5). This formula can 'tailor' the number of additional samples needed for each class, thereby maximizing the efficiency while taking into account the classification performance of each class.

[0024]

[0025] Where num represents the number of samples to be taken from each class, i represents the class, and n is the number of classes in the dataset. For each class's F1 score, Then, it represents the error rate of class i, and N is the total number of samples in the original dataset. In formula (5), the coefficient in front of N represents the ratio of the error rate of class i to the sum of the error rates of all classes. By comparing the error rate of class i with the total error rate, the proportion of the number of samples that need to be adjusted for class i can be determined. Multiplying this ratio by the total number of samples N in the original dataset, the number of samples that need to be sampled for class i can be obtained. Since all terms in the formula are positive, there is no undersampling that leads to information loss. Combining formula (5) with DeepSMOT is called DeepSMOT-F1, which is used to perform deep data augmentation on each class in the training set in a targeted manner.

[0026] Preferably, in S05, the pre-trained model for coral classification is obtained by using transfer learning to initialize network parameters, using the parameters trained on ImageNet to initialize the network parameters of the ResNet-50 with a more complex structure, not initializing the full connection layer, and then retraining the parameters of the network on new samples and original samples, including two full connection layers, so that the network is more suitable for the coral classification task.

[0027] Preferably, in S06, transfer learning from the large public dataset ImageNet is used to transfer the parameters of other layers except the full connection layer to the ResNet-50 with a more complex structure, to initialize the network parameters and obtain the pre-trained model for coral classification; then, the parameters of the pre-trained model are trained on the expanded training set, including two full connection layers, to obtain the final coral classification model.

[0028] The beneficial effects of the present application are: for the problem of unbalanced data set, the present application proposes a deep data enhancement method DeepSMOTE-F1 according to the F1-score of each category on the original data set by modifying the sampling quantity definition formula. DeepSMOTE-F1 considers the similarity between samples and automatically generates new samples using a deep learning model, ensuring that the generated images not only retain the attributes of the original images, but also have high visual quality and rich information, meeting the needs of deep learning models for diversified data. At the same time, DeepSMOTE-F1 'tailors' the number of samples that need to be increased for each class according to the comprehensive index F1-score in the classification algorithm, so that the model can fully learn the features of each class, thereby maximizing the classification effect of each class. In addition, for the problem of low model classification accuracy caused by small data set and insufficient sample diversity, the present application uses transfer learning from ImageNet to initialize network parameters. By using the large scale and diversity of the ImageNet dataset, the pre-trained network parameters can be applied to the coral classification task, thereby accelerating the convergence speed of the model and alleviating the problem of low model classification recognition ability caused by small data set and diversity. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The technical roadmap of the present application is shown in the figure;

[0030] Figure 2 The network structure diagram of the ResNet-50 with a more complex structure is shown in the figure;

[0031] Figure 3 The image feature sampling diagram is shown in the figure;

[0032] Figure 4 The DeepSMOTE-F1 structure diagram is shown in the figure;

[0033] Figure 5 For transfer learning. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0035] The following will be described in conjunction with Figure 1 The algorithm process and principles are described.

[0036] A coral classification method fusing deep data enhancement and transfer learning, comprising the following steps:

[0037] S01: Network structure building.

[0038] The ResNet-50 structure is modified and used as the backbone network of the model. The original fully connected layer is replaced by two fully connected layers with different structures, and an activation function is connected after each of them, as shown in Figure 2 The specific structure is that the first fully connected layer contains 512 neurons and is connected with a ReLU activation function. The number of neurons of the second fully connected layer is the same as the number of classes of the data set. Finally, a Softmax activation function is connected after the layer. The introduction of the activation function can increase the nonlinear fitting ability of the network.

[0039] S02: Image preprocessing.

[0040] In order to avoid the problems of large model, high memory occupation and increased training time, the size of the coral image is adjusted to 128x128 pixels, and the image is normalized to improve the convergence speed and robustness of the model.

[0041] S03: Calculate the F1-score of each category.

[0042] The network without parameter initialization is trained on the training set, and the F1-score of each class is calculated. Figure 2

[0043]

[0044] S04: DeepSMOTE-F1 deep data enhancement.

[0045] ​DeepSMOTE adopts the backbone network of Generative Adversarial Networks (GAN) as the encoder and decoder, and combines with the Synthetic Minority Over-sampling Technique (SMOTE) to realize the deep enhancement of image data. It first maps the image to a high-dimensional feature space through the encoder, then samples the features of the image using SMOTE, and finally decodes these new feature vectors into new images using the decoder. The specific steps of image feature sampling are as follows: first, for each sample x in a certain class, calculate the distance from it to all samples in the sample set of the class based on the Euclidean distance, and get its k-nearest neighbors. Then, for sample x, randomly select 1 sample from its k-nearest neighbors as a neighbor Finally, according to formula (2), a new sample x new is constructed, where the coefficient of formula (2) is called the scaling factor, its value is random and between 0 and 1, which is used to determine the similarity degree of new sample x new and x and . The specific process is shown in Figure 3 . This method considers the similarity between samples and automatically generates new samples using deep learning models, ensuring that the generated images not only retain the attributes of the original images, but also have high visual quality and rich information, meeting the needs of deep learning models for diversified data.

[0046]

[0047] Although DeepSMOTE is an effective data augmentation method for deep learning models, it still faces an important problem, that is, how to determine the number of samples to be expanded for each class. In the training process of a classification model, the classification algorithm takes the balance of sample quantity as a prerequisite and takes the overall classification accuracy as a goal, which is not applicable to unbalanced data sets. For example, given an unbalanced data set with only 1% positive samples, when the classifier labels all examples as negative samples, its classification accuracy can be as high as 99%, but this value does not truly reflect the value of the classifier. Therefore, when the data set is unbalanced, it is more meaningful to improve the classification effect of each class than to improve the overall classification accuracy while ensuring the overall classification accuracy. Therefore, the present application focuses on the ability of the model to correctly classify each coral class, while avoiding the misclassification of corals of this class as other classes (FN) and the misclassification of corals of other classes as this class (FP) as much as possible. F1-score considers both FN and FP by giving equal weight to Recall and Precision, as shown in equations (2) and (3). Recall represents the proportion of correctly classified positive samples to actual positive samples, and Precision represents the proportion of correctly classified positive samples to predicted positive samples, both of which describe the performance of the model from different dimensions.

[0048] We evaluate the classification of each class by F1-score to adjust the data distribution of training samples, thereby indirectly affecting the model training process. A low F1-score indicates poor performance of the model in this class, and to improve the performance of this class, we increase the number of training samples so that the model has more opportunities to learn its features. Conversely, if the F1-score is high, the model can identify the coral class well, so there is no need to oversample the coral class in the data augmentation phase so that the model can focus more on other classes with poor classification results.

[0049]

[0050]

[0051] Based on the above idea, to solve the problem of the difficulty of determining the number of samples for DeepSMOTE, we use the important comprehensive indicator F1-score in the classification algorithm to propose a sampling number calculation formula (equation 5) sensitive to F1-score. This formula can 'tailor' the number of samples to be added for each class, thereby maximizing the classification effect of each class.

[0052]

[0053] where num represents the number of samples needed for each class, i represents the class, and n is the number of classes in the dataset, F1-score for each class, where e i represents the error rate of class i, and N is the total number of samples in the original dataset. In formula 5, the coefficient in front of N represents the ratio of the error rate of class i to the sum of error rates of all classes. By comparing the error rate of class i with the total error rate, we can determine the proportion of the number of samples that class i needs to adjust. Multiply this ratio by the total number of samples N in the original dataset, and we get the number of samples that class i needs to sample. Since all terms in the formula are positive, there is no information loss caused by under-sampling. Combine formula (5) with DeepSMOT, called DeepSMOT-F1, and use it to perform deep data augmentation on each class in the training set. The specific process is shown in Figure 4 First, map the image data in the training set to a high-dimensional feature space through the encoder; then, use the SMOTE algorithm to select the corresponding number of samples and their nearest neighbors according to the results of formula (5) to synthesize new samples; finally, use the decoder to decode them into new image data. Add the newly generated images to the original training set to adjust the data distribution of the training samples, thereby affecting the training process of the model and enabling the model to more targetedly learn the features of each class, improving the performance of the model in the coral classification task.

[0054] S05: Transfer learning from ImageNet.

[0055] For hierarchical convolutional neural networks, different layers of neurons learn different types of features. Bottom-layer neurons learn low-level features such as edges, lines, and colors that are common in image domains, while high-layer neurons learn high-level semantic features related to the target task. Therefore, the relevance of the features extracted from bottom to top to the target task increases level by level.

[0056] Based on this, the present application uses transfer learning from the large public dataset ImageNet to transfer the parameters of all layers except the fully connected layers to the modified ResNet-50( Figure 2 ) after the structure change, to initialize the network parameters. Figure 5), to obtain a pre-trained model of coral classification. Then, the parameters of the pre-trained model are trained on the expanded training set, including two fully connected layers, to obtain a final coral classification model. ImageNet is a super large data set containing more than one million images covering 1000 categories. Although it has some differences from the coral image data set, the low-level features such as edges, lines and colors of the images have certain universality. By using the pre-trained parameters on ImageNet, the learned rich semantic features can be applied to the coral data set, solving the difficulties faced by small data sets in training deep neural networks from scratch and the problem of poor model performance caused by insufficient data diversity. At the same time, transfer learning can also speed up the training speed, reduce the risk of overfitting and improve the generalization ability of the model.

[0057] S06: Model testing and result analysis.

[0058] This part mainly uses test samples to verify the accuracy of the trained model. The test samples of corals are used to test the model, and compared with other classic coral classification methods for analysis.

[0059] The present application proposes a deep data enhancement method based on DeepSMOTE-F1, which realizes targeted data enhancement for samples of various categories, so that the model can learn the features of each class more fully, thereby solving the problem of data imbalance in the coral classification task. The present application uses the huge scale and diversity of the ImageNet data set to apply the pre-trained network parameters to the coral classification task, thereby improving the problem that the performance of deep learning in the coral classification and recognition task needs to be improved due to small data set and insufficient diversity.

[0060] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A coral classification method integrating deep data augmentation and transfer learning, characterized in that: Includes the following steps: S01: Network structure construction, modifying the ResNet-50 structure and using it as the backbone network of the model; S02: Image preprocessing, resizing the coral image to 128×128 pixels and normalizing the image; S03: Calculate the F1-score for each category on the original training set. Train the network on the training set without parameter initialization and compute the values ​​for each class. ; （1） S04: Use DeepSMOTE-F1 to perform deep data augmentation on the original training set to obtain new samples; S05: Initialize network parameters using transfer learning to obtain a pre-trained model for coral classification; S06: Retrain the parameters of the pre-trained model on the original samples and the new samples to obtain the final model for coral classification; S07: Model testing and results analysis; In S01, the original fully connected layer is replaced with two fully connected layers with different structures, and activation functions are connected after them respectively. Specifically, the first fully connected layer contains 512 neurons and is connected to a ReLU activation function. The number of neurons in the second fully connected layer is the same as the number of classes in the dataset. Finally, this layer is connected to a Softmax activation function. The introduction of activation functions can increase the non-linear fitting ability of the network. To address the problem of the difficulty in determining the number of samples in DeepSMOTE, an F1-score-sensitive formula for calculating the number of samples is proposed (5). This formula tailors the number of additional samples required for each class, thereby maximizing the efficiency of the classification performance of each class. （5） in, This represents the number of samples to be taken from each category, where i represents the category and n is the number of categories in the dataset. For the F1 score of each class, 1- Then, N represents the error rate of class i, and N is the total number of samples in the original dataset. In formula (5), the coefficient in front of N represents the ratio of the error rate of class i to the sum of the error rates of all classes. By comparing the error rate of a class with the total error rate, the proportion of the number of samples that need to be adjusted for class i can be determined. Multiplying this ratio by the total number of samples N in the original dataset will give the number of samples that need to be sampled for class i. Since all terms in the formula are positive, there is no undersampling that leads to information loss. Combining formula (5) with DeepSMOT is called DeepSMOT-F1, which is used to perform deep data augmentation on each class in the training set in a targeted manner.

2. The coral classification method integrating deep data augmentation and transfer learning according to claim 1, characterized in that: In S04, DeepSMOTE-F1 uses a Generative Adversarial Network (GAN) backbone network as the encoder and decoder, combined with the Synthetic Minority Over-sampling Technique (SMOTE) to achieve depth enhancement of image data.

3. The coral classification method integrating deep data augmentation and transfer learning according to claim 1, characterized in that: In S04, the image is first mapped to a high-dimensional feature space by the encoder, then the features of the image are sampled using SMOTE, and finally the new feature vectors are decoded into a new image by the decoder. The specific steps of image feature sampling are as follows: First, for each sample x in a certain class, the distance from it to all samples in the sample set of that class is calculated using Euclidean distance as the standard, and its k nearest neighbors are obtained; then, for sample x, one sample is randomly selected from its k nearest neighbors as its nearest neighbor. Finally, construct new samples according to formula (2). The coefficient in formula (2) is called the scaling factor, and its value is random, ranging from 0 to 1, and is used to determine the new sample. With x and similarity （2）。 4. The coral classification method integrating deep data augmentation and transfer learning according to claim 1, characterized in that: In S04, the goal is to avoid misclassifying this type of coral as another type (FN) and misclassifying other types of coral as this type (FP) simultaneously. The F1-score takes into account both FN and FP cases by assigning equal weight to recall and precision, as shown in formulas (3)-(4). （3） （4） Recall represents the proportion of correctly classified positive samples out of the actual positive samples, while Precision represents the proportion of correctly classified positive samples out of the predicted positive samples. Both describe the model's performance from different dimensions.

5. The coral classification method integrating deep data augmentation and transfer learning according to claim 1, characterized in that: In S05, during the process of initializing network parameters using transfer learning to obtain the pre-trained model for coral classification, the parameters trained on ImageNet are used to initialize the network parameters of the ResNet-50 after the structural modification. The fully connected layers are not initialized here.

6. The coral classification method integrating deep data augmentation and transfer learning according to claim 1, characterized in that: In S06, the parameters of the pre-trained model are trained on the expanded training set, including two fully connected layers, to obtain the final coral classification model.

Images