An Improved Visual Transformer Underwater Sediment Sonar Image Classification Method Based on Transfer Learning

Abstract

The present invention discloses an improved vision Transformer subsea sediment sonar image classification method based on transfer learning. The improved vision Transformer classification method belongs to a deep learning method. It forms patches by preprocessing the image through a convolutional layer, and then obtains the classification result after passing through patch embedding, positional embedding, the Transformer encoding layer, and the multi-layer perceptron output layer in sequence. The classifier is trained by minimizing the network residual through the backpropagation method. The purpose of deep learning is to learn the internal laws and representation levels of sample data, and it has a strong non-linear fitting learning ability, which can effectively discover the common features among images of the same category. Through the stacking of multiple encoding layers, the network will gradually learn global and local features and will actively focus on the "important" parts of the image.

Description

Technical Field

[0001] The present invention belongs to the fields of underwater sediment acoustic exploration, inversion, image classification technology and artificial intelligence, and particularly relates to an improved visual Transformer underwater sediment sonar image classification method based on transfer learning. Background Technique

[0002] The goal of underwater detection is to use certain technical means to obtain the spatial distribution characteristics of underwater sediments and their variation laws over time, which is a pattern study based on a continuous spatio-temporal system. Sound waves have very superior propagation characteristics in seawater. With the rapid development of sonar technology, underwater sediment sonar images can already contain very rich underwater topography and sediment feature information. Inverting underwater topography from underwater sediment sonar images is of great significance for underwater exploration, underwater shipwreck salvage and deep-sea protection. However, due to the limitations of the complex underwater acoustic field environment and sonar equipment performance, sonar images may have problems such as speckle noise and blurred edges, which impose more stringent requirements on the performance of the classifier.

[0003] In underwater scientific research, a variety of underwater acoustic detection technologies have been developed, and underwater acoustic technology has played an important role in detecting shallow subsurface profiles and even deep formation structures. Since parameters such as the sound absorption coefficient, reflection coefficient, and surface roughness of different types of sediments have significant differences, traditional acoustic underwater inversion uses the underwater sound reflection characteristics to identify the types of underwater sediments. Most of this method is based on some acoustic indicators and empirical formulas. In fact, although this inversion method is computationally simple, due to the complex underwater environment, the classification accuracy is not very ideal.

[0004] Modern acoustic seabed sediment classification is based on sonar images obtained by sonar systems, combined with real seabed sediment sample data obtained by mechanical sampling methods. Classification algorithms are used to establish an automatic analysis model for sediment types, thereby achieving efficient and accurate seabed sediment classification. Diesing used the random forest algorithm to classify sonar images of 4 sediment types, and the results were better than those of traditional target analysis methods and geostatistical analysis methods. Many domestic and foreign scholars have also used various artificial neural network methods to achieve the classification and recognition of seabed sediments. Chakraborty et al. studied the classification and recognition of seabed sediments using self-organizing feature mapping (SOFM) and learning vector quantization (LVQ) neural network methods. Tang Qiuhua effectively combined GA and BP neural networks to achieve automatic classification and recognition of sediment types such as seabed bedrock, gravel, sand, fine sand, and mud. The classification accuracies of bedrock, gravel, sand, fine sand, and mud were 92.2%, 81.9%, 89.3%, 85.9%, and 88.2% respectively. However, the LVQ neural network cannot fully utilize neurons and is sensitive to initial values. Moreover, the BP neural network has problems such as slow convergence speed, relatively random initialization parameters, and being easily trapped in local optima. The algorithm operation efficiency is not high, especially when the number of network layers is large and the complexity is high, the training time is long. Subsequently, convolutional neural networks were used, combining the texture features and statistical features of sonar images with CNN to construct a classifier. This classifier can achieve the function of automatic identification and classification of sediments. However, ordinary convolutional neural networks have overfitting problems during training, and due to the limitations of the principle, convolutional neural networks cannot effectively obtain global information and position information of feature maps within a large field of view. That is, they cannot learn long-distance dependencies of pictures, thus affecting the classification performance of the network. In addition, deep learning methods require a large number of training samples to obtain common features of images, and the insufficient number of seabed sediment sonar images also limits the improvement of the classification accuracy of deep learning seabed sediment sonar images. Summary of the Invention

[0005] The purpose of the present invention is to provide an improved vision Transformer seabed sediment sonar image classification method based on transfer learning.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] An improved vision Transformer seabed sediment sonar image classification method based on transfer learning, comprising the following steps:

[0008] Step 1: Obtain a dataset for training;

[0009] It is necessary to know the sediment type to which the image belongs and obtain n types of seabed sediment sonar images;

[0010] Step 2: Preprocess these images;

[0011] The preprocessing includes image denoising and image feature enhancement;

[0012] Step 3: Obtain source domain images for pre-training;

[0013] Step 4: Build an improved Vision Transformer network;

[0014] It includes a front convolutional transformation layer, a patch and position embedding layer, a Transformer encoding layer, and a multi-layer perceptron output layer; The layers are interconnected to form the entire improved Vision Transformer network;

[0015] Step 5: Train the network on the source domain texture images to obtain a pre-trained model;

[0016] Use the cross-entropy loss function to divide the texture images into a training set and a validation set and input them into the network for network training;

[0017] Step 6: According to Step 2, set the number of the training set and the validation set according to the ratio of 4:1 - 7:1;

[0018] Step 7: Input the images in the training set and their actual categories into the network for adjustment, and verify the classification accuracy on the validation set every round of training;

[0019] Step 8: Adjust the network parameters to obtain a trained Transformer classifier;

[0020] Step 9: Input the sonar images whose substrate types need to be judged into the classifier, and the classifier automatically outputs the classification results.

[0021] Furthermore, the images selected in Step 1 belong to the same substrate type, use sonar image slices of the same size, and the network adopts input images with a resolution of 100×100.

[0022] Furthermore, Step 2 specifically includes:

[0023] Step 2.1: Perform smoothing denoising and feature enhancement on the sonar images of the seabed substrate respectively;

[0024] Step 2.2: Use wavelet transform technology to filter the sonar images of the seabed substrate that have undergone smoothing denoising and feature enhancement;

[0025] Step 2.3: Use the multi-level median filtering method to strengthen the sonar images of the seabed substrate after filtering.

[0026] Furthermore, the specific structures of each layer in Step 4 are as follows:

[0027] The pre - convolutional transformation layer is as follows: It converts the input image into a feature map of 12×12×512, and the dimension of the output tensor is 12×12×512 for subsequent patch embedding operations;

[0028] The Patch and position embedding layer is as follows: It converts the tensor output by the pre - convolutional transformation layer into patches that can be used by the Transformer, records the position information of each patch, and adds a class token at the same time;

[0029] The Transformer encoding layer is as follows: It includes 12 encoding blocks and a multi - head attention structure, and does not change the dimension number of the tensor; Among them, the multi - head attention structure uses 12 heads, which contain 12 q, k, and v values, and the following calculations are performed:

[0030]

[0031] head i = Attention(QW i Q , KW i K , VW i V )

[0032] Multi - head attention(Q, K, V) = Concat(head1,..., head i )W O

[0033] The W matrix is a transformation parameter matrix. The Softmax function can map the input into probabilities of various categories. The function is as follows:

[0034]

[0035] where h(x, y j ) is the prediction result for the j - th item;

[0036] The multi - layer perceptron output layer is as follows: It uses a single fully - connected layer to achieve dimension transformation to obtain the class output; The output is an n - dimensional tensor, where n is the number of classes of the samples, and the type to which the input image belongs can be directly obtained.

[0037] Furthermore, step 5 takes the difference between the output of the network and the true output as the basis for network training, divides the training set images into multiple Batches, and inputs them into the network in batches for training; When the validation accuracy of the network no longer improves, the training is completed.

[0038] Furthermore, the adjustment method in step 7 is as follows:

[0039] Copy the parameters in the Transformer encoding layer of the pre-trained model to the new model to obtain the migrated model. Then, input the images in the training set into the network and also use the cross-entropy loss function for training. The training process is similar to that in step 5. After each iteration of the training set, it is necessary to verify the classification accuracy on the validation set. When the validation accuracy of the network no longer improves, the finally fine-tuned model is obtained.

[0040] Furthermore, the output result of step 9 will be presented in the form of an n-dimensional vector, where n is the number of image categories. The size of the number at each corresponding position for each category is its probability, and the number at the position corresponding to the maximum value in the result vector is the category result output by the classifier.

[0041] The beneficial effects of the present invention are as follows:

[0042] The present invention adopts the transfer learning method, which is a method of transferring existing prior knowledge to a new model. By adopting the basic idea of pre-training in the source domain and fine-tuning in the target domain, it effectively improves the training speed, classification accuracy, and convergence speed under small samples. The pre-trained model can be applied to other sonar images of seabed sediments. Using a transfer learning deep learning classifier can, through pre-training, apply the methods learned in a similar data domain to the new target domain. This transfer process only requires fine-tuning to obtain better classification effects and can greatly reduce the number of training samples required.

[0043] The Transformer of the present invention is a new type of neural network architecture, and its typical feature is the use of a large number of attention modules. The Transformer classifier adopts a multi-head attention mechanism, which can effectively capture the detailed information beneficial to classification in the image, suppress other useless information, and can learn the effective information within a large scale range. It is no longer limited by the window size of the convolutional neural network, can effectively grasp the global and local features of the sonar image of the seabed sediment, has strong generalization ability, retains the ability of the attention mechanism to grasp important features, and the introduction of multiple heads can also reduce overfitting, calculate and select multiple pieces of information from the input information in parallel, and has higher efficiency in a high-parallel environment such as GPU. It has the ability to quickly obtain the area that needs attention. The use of the multi-head attention mechanism enables the network to comprehensively integrate global and local features and effectively grasp the unique features of various images. Many regularization methods such as LayerNorm and DropPath are used in the network to reduce network overfitting. At the same time, in view of the characteristics of the seabed sonar image, the present invention improves and adjusts the network structure so that it can process single-channel images and solves the problem that the original vision Transformer cannot process grayscale images. By adjusting the construction method of the input patch, the convergence ability of the network during few-shot training is improved. The present invention can realize the fast and automatic recognition of sediment categories, realize a large number of sonar image classification operations, and can obtain a high classification accuracy by using only a small number of training samples. Description of the Drawings

[0044] Figure 1 It is a flowchart of the steps of the method for classifying sonar images of seabed sediment based on vision Transformer of the present invention;

[0045] Figure 2 It is a schematic diagram of transfer learning adopted by the present invention;

[0046] Figure 3 It is the overall network structure diagram of the improved Transformer network proposed by the present invention;

[0047] Figure 4 It is a detailed diagram of the pre-convolution transformation layer in the network of the present invention;

[0048] Figure 5 It is a detailed structure diagram of a single coding block of the present invention. Detailed Embodiment

[0049] The present invention will be further described below with reference to the accompanying drawings.

[0050] The improved Vision Transformer classification method adopted in the present invention belongs to a deep learning method. It forms patches by preprocessing the image through convolution, and obtains the classification result after passing through patch embedding, positional embedding, the transformer encoding layer, and the multi-layer perceptron output layer in sequence. The classifier is trained by minimizing the network residual through the backpropagation method. The purpose of deep learning is to learn the internal laws and representation levels of sample data, and it has a strong non-linear fitting learning ability, which can effectively discover the common features among similar images. Through the stacking of multiple encoding layers, the network will gradually learn global and local features and actively focus on the "important" parts of the image.

[0051] Next, according to the steps described in the technical solution, the present invention will be further described in detail with reference to the accompanying drawings.

[0052] Step 1: Obtain the dataset for training. It is necessary to know the substrate type to which the image belongs and obtain a certain number of sonar images of n types of seabed substrates. Since deep learning needs to learn the same features in the image to achieve classification, a certain number of sonar images of the same type are indispensable. In this step, it is necessary to ensure that the selected images belong to the same substrate type. To obtain consistent features, sonar image slices of the same size are also required. Our network uses input images with a resolution of 100×100.

[0053] Step 2: Preprocess the dataset images. Since sonar images may have characteristics such as low resolution, serious noise interference, and deteriorated edge textures, it is necessary to preprocess these seabed substrate sonar images. The preprocessing specifically includes: first, perform smoothing denoising and feature enhancement on the seabed substrate sonar images respectively, and the feature enhancement includes calibration enhancement and edge sharpening; then use wavelet transform technology to filter the seabed substrate sonar images that have undergone smoothing denoising and feature enhancement to obtain seabed substrate sonar images with higher signal-to-noise ratio and better denoising effect. The filtering uses wavelet band-pass filters with different scales, and the scale refers to that the wavelet function uses different thresholds for the low-frequency and high-frequency parts. Finally, use the multi-level median filtering method to strengthen the filtered seabed substrate sonar images, and the strengthening process includes the texture and edge shape information contained. Operations such as filtering and feature enhancement on the image reduce the influence of noise generated by the sound field environment and sonar equipment performance on blurring the image edge, and highlight the feature information of the sonar image.

[0054] Step 3: Obtain source domain images for pre-training. Transfer learning requires that the source domain images for pre-training have similar features and distributions to the target domain images for fine-tuning, so as to ensure that the patterns learned during pre-training can be used for target domain image classification. Most existing pre-trained models are based on ImageNet images, which have weak correlation with the features of sub-bottom sonar images. The present invention proposes using a texture dataset as the source domain images for pre-training. They have strong similarity and close data distribution, and are more suitable for model transfer. The selected data and transfer learning process are as Figure 2 shown.

[0055] Step 4: Establish an improved Vision Transformer network. The network used in the present invention adopts a pre-convolution transformation layer, a patch and position embedding layer, a Transformer encoding layer, and a Multilayer Perceptron output layer. The layers are interconnected to form the entire improved Vision Transformer network, as shown in the appendix Figure 2 shown. The dimensions of the current tensor are given on the right side of each layer. The structures of each layer are introduced in detail below.

[0056] Pre-convolution transformation layer: It converts the input image into a feature map of 12×12×512 for subsequent patch embedding operations. The detailed structure of the pre-convolution transformation layer is shown in the appendix Figure 3 shown. First, it passes through a 7×7 two-dimensional convolutional layer, followed by group normalization and dropout operations, which can prevent overfitting. Then, it passes through two residual modules. Residual connections help preserve information before and after. The final output tensor dimension is 12×12×512. The tensor dimensions obtained at each stage are listed in the figure. The improved network of the present invention adds a pre-convolution transformation module, which is optimized for sonar images, enabling the network to directly process single-channel sonar images and converge faster than the original network.

[0057] Patch and position embedding layer: Its function is to convert the pre-tensor into patches that can be used by the Transformer and record the position information of each patch. At the same time, a class token is added, which is a trainable parameter and will change with the training of the network. As shown in the corresponding structure in the appendix Figure 2 shown, use a 1×1 convolutional operation to increase the patch dimension to 768 and flatten the tensor to two dimensions. At this time, an additional position patch is spliced in, and a trainable parameter of the same size of 145×768 is added to it to achieve position embedding.

[0058] Transformer Encoding Layer: It is stacked by 12 encoding blocks. The structure of each encoding block and the multi-head attention is as Figure 4 shown. This part will not change the dimension of the tensor. The most important one is the multi-head attention module. In the present invention, 12 heads are used, that is, there are 12 q, k, and v values. The multi-head processing will execute multiple attentions with different parameter initializations in parallel to calculate the matching parameters between two elements, and splice the attention results of each element into an input of the subsequent projection network. The multi-head attention layer performs the following calculations:

[0059]

[0060] head i = Attention(QW i Q , KW i K , VW i V )

[0061] Multi-head Attention(Q, K, V) = Concat(head1,..., head i )W O

[0062] The W matrix is the transformation parameter matrix. The Softmax function can map the input into probabilities of various categories. The function is as follows

[0063]

[0064] where h(x, y j ) is the prediction result of the j-th item.

[0065] In addition, the detailed structure of the multi-layer perceptron in the structure is also given in Figure 4 . The main function of the Transformer encoding layer is to learn the key information in the input data and is the core part of the entire network. It does not change the dimension of the data.

[0066] Output Layer of the Multi-layer Perceptron: In the present invention, a fully connected layer is used to achieve dimension transformation to obtain the class output. The output is an n-dimensional tensor, where n is the number of classes of the samples. From this output, the type to which the input image belongs can be directly obtained.

[0067] The present invention selects the SGD optimizer for gradient descent. In view of the performance in the classification task of subsea sediment sonar images, the SGD optimizer can obtain higher accuracy.

[0068] Step Five: Train the network on the source domain texture images to obtain a pre-trained model.

[0069] Using the cross-entropy loss function, the texture images are divided into a training set and a validation set and input into the network for network training. Based on the difference between the output of the network and the output in the real situation, the network parameters are continuously learned and updated. The training set images are segmented into several Batches and input into the network in batches for training. When the validation accuracy of the network no longer improves, the training is completed. At this time, the network has mastered the data extraction ability of the source domain.

[0070] Step Six: Set the number of the training set and the validation set according to the ratio of 4:1 - 7:1 to ensure the correct class labels of the images. Divide the data set processed in Step Two into the training set and the validation set according to a specific ratio. The purpose is to make the training set and the validation set have reasonable and similar distributions, enabling the network to better generalize and learn the common features of the images.

[0071] Step Seven: Input the images in the training set and their actual classes into the network for fine-tuning, and verify the classification accuracy on the validation set for each round of training. The fine-tuning method is as follows: Copy the parameters in the Transformer encoding layer of the pre-trained model into the new model, which is the migrated model. Then input the images in the training set into the network and also use the cross-entropy loss function for training. The training process is similar to that in Step Five. At this time, since the network has learned the prior knowledge of the source domain, usually only a few rounds of training are required for the network to converge. Note that the classification accuracy needs to be verified on the validation set after each iteration of the training set. When the validation accuracy of the network no longer improves, the finally fine-tuned model is obtained.

[0072] Step Eight: Adjust parameters such as the learning rate and the momentum of the optimizer to make the classifier perform best on the validation set and obtain the trained Transformer classifier. To obtain the network with the best classification effect, parameters such as the learning rate and the batch size need to be adjusted to make the network reach the optimal solution. The larger the learning rate, the faster the network converges, but it will cause phenomena such as loss explosion and oscillation; if the learning rate is too small, the network is likely to fall into a local optimal solution, resulting in the training loss no longer decreasing and the validation accuracy no longer improving, and the network effect is limited. Therefore, the size of the learning rate needs to be adjusted according to experience.

[0073] Increasing the batch size can improve the stability of the convergence of the train loss, but the size of the batch size is restricted by the memory. If the batch size is too small, the network may not converge.

[0074] The present invention proposes to use a learning rate of 0.0001 - 0.001. As the number of iterations increases and the training loss decreases, the size of the learning rate can also be appropriately reduced to approach the optimal solution. A batch size of 16 - 64 is more suitable for the classification of sonar images of seabed sediment. The selected batch size should consider the limitations of memory, video memory, and computing resources simultaneously and should not exceed the maximum value.

[0075] Step Nine: Input the sonar image whose sediment type needs to be judged into the classifier, and the classifier will automatically output the classification result. The output result will be presented in the form of an n-dimensional vector, where n is the number of image categories. The size of the number at each corresponding position of each category is its possible probability. Therefore, the position corresponding to the maximum value in the result vector is the category given by the classifier.

[0076] The factors that will affect the accuracy of the classifier in the present invention are as follows:

[0077] 1. The images used as the training set of the classifier are preferably able to well reflect the characteristics of various sediments. Image quality problems such as noise and resolution will affect the generalization of the features extracted by the algorithm. Excellent training set images will bring the best classification performance. The present invention hopes to reduce the impact brought by the image quality of the training set through means such as image denoising and feature enhancement.

[0078] 2. The size of the learning rate should be set according to the situation of the training set to enable the network to obtain a better convergence speed and higher classification accuracy. If the learning rate is too large, the network will not be able to converge and will wander around the optimal value. If the learning rate is too small, the network will fall into a local optimum.

[0079] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An improved visual Transformer-based subsea sediment sonar image classification method using transfer learning, characterized by: It includes the following steps: Step 1: Obtain the dataset for training; It is necessary to know the substrate type to which the image belongs and obtain n types of sonar images of seabed substrates; Step 2: Perform preprocessing operations on these images; The preprocessing includes image denoising and image feature enhancement; Step 3: Obtain the source domain texture images for pre-training; Step 4: Establish an improved Vision Transformer network; It includes a pre-convolution transformation layer, a patch and position embedding layer, a Transformer encoding layer, and a multi-layer perceptron output layer; the layers are interconnected to form the entire improved Vision Transformer network; Step 5: Train the network on the source domain texture images to obtain a pre-trained model; Use the cross-entropy loss function to divide the texture images into a training set and a validation set and input them into the network for network training; Step 6: According to Step 2, set the number of the training set and the validation set according to the ratio of 4:1 - 7:1; Step 7: Input the images in the training set and their actual categories into the network for adjustment, and verify the classification accuracy on the validation set every time a round of training is completed; Copy the parameters in the Transformer encoding layer of the pre-trained model to a new model, which is the migrated model; then input the images in the training set into the network, and also use the cross-entropy loss function for training. The training process is similar to that in Step 5; Verify the classification accuracy on the validation set every time the training set iteration is completed. When the validation accuracy of the network no longer improves, obtain the finally fine-tuned model; Step 8: Adjust the network parameters to obtain a trained Transformer classifier; Step 9: Input the sonar image whose substrate type needs to be judged into the classifier, and the classifier automatically outputs the classification result.

2. An improved visual Transformer-based subsea sediment sonar image classification method based on transfer learning according to claim 1, characterized in that: The images selected in Step 1 belong to the same substrate type, and sonar image slices of the same size are used. The network uses input images with a resolution of 100×100.

3. An improved visual Transformer-based subsea sediment sonar image classification method based on transfer learning according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Perform smoothing denoising and feature enhancement on the sonar images of seabed substrates respectively; Step 2.2: Use wavelet transform technology to filter the sonar images of seabed substrates that have undergone smoothing denoising and feature enhancement; Step 2.3: Use the multi-level median filtering method to strengthen the filtered sonar images of seabed substrates.

4. An improved visual Transformer subsea sediment sonar image classification method based on transfer learning according to claim 1, characterized in that: The specific structures of each layer in Step 4 are as follows: The pre-convolution transformation layer is: Convert the input picture into a feature map of 12×12×512, and the dimension of the output tensor is 12×12×512 for subsequent patch embedding operations; The Patch and Position Embedding Layer is: Convert the tensor output by the pre-convolution transformation layer into patches that can be used by the Transformer, record the position information of each patch, and add a class token at the same time; The Transformer encoding layer is: It includes 12 encoding blocks and a multi-head attention structure, and does not change the dimension number of the tensor; among them, the multi-head attention structure uses 12 heads, corresponding to 12 Q, K, and V values, and the following calculations are performed: head i = Attention (QW i Q , KW i K , VW i V ) Multi-Head Attention(Q, K, V) = Concat(head1,..., head i )W O The W matrix is the transformation parameter matrix, and the Softmax function can map the input into probabilities of various categories. The function is as follows: where h(x, y j ) is the prediction result for the j-th item; The output layer of the multi-layer perceptron is as follows: A fully connected layer is used to implement dimensional transformation to obtain the class output; the output is an n-dimensional tensor, where n is the number of classes of the samples, and the type to which the input image belongs can be directly obtained.

5. An improved visual Transformer-based subsea sediment sonar image classification method based on transfer learning according to claim 1, characterized in that: In step 5, the difference between the network output and the true output is used as the basis for network training. The training set images are divided into multiple batches and input into the network in batches for training; when the validation accuracy of the network no longer improves, the training is completed.

6. An improved visual Transformer-based subsea sediment sonar image classification method based on transfer learning according to claim 1, characterized in that: The output result in step 9 will be presented in the form of an n-dimensional vector, where n is the number of image classes; the magnitude of the number at the corresponding position for each class is its probability, and the number at the position corresponding to the maximum value in the result vector is the class result output by the classifier.

Images