A sonar image texture feature removal method based on a deep neural network

Abstract

The application discloses a sonar image texture feature removing method based on a deep neural network and belongs to the technical field of image processing. The method comprises the following steps: constructing a multilayer feature extraction convolutional neural network, and extracting deep features of an input image by using the multilayer feature extraction convolutional neural network; performing feature transformation on the extracted deep features by using a feature transformation formula to obtain deep features after feature transformation; adding noise pollution to the deep features after transformation; constructing a multilayer decoding deep neural network, inputting the deep features after adding noise pollution into the multilayer decoding deep neural network for decoding to obtain image features after removing texture features. The source field and the target field are subjected to the same texture feature removing transformation, so that the source field and the target field after transformation are more similar, and the field gap between the source field and the target field is shortened.

Description

Technical Field

[0001] This invention relates to the field of sonar image processing technology, and more specifically to a method for removing texture features from sonar images based on deep neural networks. Background Technology

[0002] Sonar is the most widely used sensor in underwater detection, target search and other fields. In recent years, with the development of marine activities, the application of sonar equipment has been extended beyond military applications to include commercial and civilian applications, such as seabed resource detection, oil exploration, maritime rescue, automatic mapping of seabed topography and geomorphology, and fish detection.

[0003] However, most current underwater robots only have the function of acquiring sonar image data, and the interpretation and analysis of sonar images still mainly rely on manual labor. When the surveyed sea area is large and the number of characteristic landforms is huge, manual identification requires a lot of manpower and time. If underwater robots could have autonomous target recognition capabilities, this problem would be greatly improved.

[0004] Current supervised learning methods require a large number of labeled training samples to achieve good classification results. However, sonar image data is difficult and costly to acquire, resulting in a very small number of available training samples containing targets. Training deep neural networks on a small number of samples can easily lead to overfitting, making it impossible to directly apply deep neural networks for target detection and recognition.

[0005] Transfer learning is the most effective solution to overcome the problem of scarce samples. The essence of transfer learning is to use knowledge learned from similar domains to improve learning performance, thereby reducing the number of samples required during the learning process. This similar domain is called the source domain, and the domain to be improved is the target domain.

[0006] However, conventional transfer learning methods typically perform poorly when applied to sonar image classification and detection tasks. This is because there is a significant domain gap between the source domain used to extract knowledge and the sonar image domain, resulting in a large portion of the knowledge learned from the source domain being unapplicable to sonar image recognition. Summary of the Invention

[0007] In view of this, the present invention provides a sonar image texture feature removal method based on deep neural networks. By performing the same texture feature removal transformation on the source domain and the target domain, the transformed source domain and the target domain are more similar, which shortens the domain gap between the source domain and the target domain. This increases the proportion of knowledge learned in the source domain that can be used in the target domain, improves transfer learning performance, and overcomes the problem that deep neural networks cannot be applied due to insufficient samples in the target domain.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for sonar image texture feature removal based on deep neural networks, comprising:

[0010] S1. Construct a multi-layer feature extraction convolutional neural network, which includes a convolutional layer, a pooling layer and a first activation function layer, and use the multi-layer feature extraction convolutional neural network to extract the depth features of the input image;

[0011] S2. Use the feature transformation formula to perform feature transformation on the extracted depth features to obtain the feature-transformed depth features;

[0012] S3. Add noise contamination to the transformed depth features;

[0013] S4. Construct a multi-layer decoding deep neural network, which includes a deconvolution layer, an interpolation upsampling layer, an unpooling layer, and a second activation function layer. Input the depth features after adding noise pollution into the multi-layer decoding deep neural network for decoding to obtain the image features after removing texture features.

[0014] Preferably, in step S1, the multi-layer feature extraction convolutional neural network is composed of multiple stacked feature extraction units, and each feature extraction unit is formed by sequentially connecting a convolutional layer, a pooling layer and a corresponding first activation function layer.

[0015] Preferably, the feature transformation formula in step S2 includes In the formula f c For deep features extracted by a multi-layer feature extraction convolutional neural network, f t For the transformed depth features, D c Let f be a diagonal matrix, and let f be the eigenvalue of the diagonal matrix. t covariance matrix E c f t The covariance matrix is ​​an orthogonal matrix, and the covariance matrix satisfies the formula:

[0016]

[0017] Preferably, in step S3, noise contamination is added to the transformed depth features, specifically including f n =f t +λf noise In the formula, f n To add depth features after noise contamination, f noise For feature dimension and f t The same noise matrix, λ is the noise matrix fnoise The weight.

[0018] Preferably, the network structure of the multi-layer decoding deep neural network in step S4 is symmetrically set with the network structure of the multi-layer feature extraction convolutional neural network in step S1. It includes a first type of inverse transform group consisting of deconvolution layers, interpolation upsampling layers and corresponding second activation function layers, and a second type of inverse transform group consisting of deconvolution layers, unpooling layers and corresponding second activation functions. The first type of inverse transform group is used to remove texture features that are different between the source and target domains, and the second type of inverse transform group is used to retain texture features that are the same between the source and target domains. The first type of inverse transform group and the second type of inverse transform group are alternately linked.

[0019] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a sonar image texture feature removal method based on deep neural network. By performing the same texture feature removal transformation on the source domain and the target domain, the transformed source domain and the target domain are more similar, shortening the domain gap between the source domain and the target domain. This increases the proportion of knowledge learned in the source domain that can be used in the target domain, improves transfer learning performance, and overcomes the problem that deep neural networks cannot be applied due to insufficient samples in the target domain. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method flow provided by the present invention;

[0022] Figure 2 This is a block diagram of the network structure of the method provided by the present invention;

[0023] Figure 3 This is an example of a source domain remote sensing image dataset used in this invention to verify the effectiveness of the invention and to evaluate the side-scan sonar classification task.

[0024] Figure 4 This is an example of a target domain side-scan sonar image dataset sample constructed for evaluating the side-scan sonar classification task to verify the effectiveness of the present invention.

[0025] Figure 5This is an example of an image of a source domain sample and a target domain sample used to verify the effectiveness of the present invention for evaluating a side-scan sonar classification task, after the texture features have been removed by the method of the present invention.

[0026] Figure 6 This is an example of a source domain image dataset used in this invention to verify the effectiveness of the invention and to evaluate a forward-looking sonar target detection task.

[0027] Figure 7 This is an example of a target domain forward-looking sonar image dataset sample constructed for evaluating the effectiveness of the present invention in order to verify the effectiveness of the present invention.

[0028] Figure 8 This is an example of an image of a source region sample used to verify the effectiveness of the present invention for evaluating a forward-looking sonar target detection task, after the texture features have been removed by the method of the present invention.

[0029] Figure 9 This is an example of the results of a transfer learning experiment on sample target detection, used to verify the effectiveness of the present invention and constructed for evaluating the forward-looking sonar target detection task. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, this embodiment of the invention discloses a method for removing texture features from sonar images based on deep neural networks, including:

[0032] S1. Construct a multi-layer feature extraction convolutional neural network, which includes a convolutional layer, a pooling layer and a first activation function layer, and use the multi-layer feature extraction convolutional neural network to extract the depth features of the input image;

[0033] In step S1, the multi-layer feature extraction convolutional neural network is composed of multiple stacked feature extraction units. Each feature extraction unit is formed by sequentially connecting a convolutional layer, a pooling layer, and a corresponding first activation function layer.

[0034] S2. Use the feature transformation formula to perform feature transformation on the extracted depth features to obtain the feature-transformed depth features;

[0035] Specifically, the feature transformation formula includes In the formula f c For deep features extracted by a multi-layer feature extraction convolutional neural network, f t For the transformed depth features, D c Let f be a diagonal matrix, and let f be the eigenvalue of the diagonal matrix. t covariance matrix E c f t The covariance matrix is ​​an orthogonal matrix, and the covariance matrix satisfies the formula:

[0036]

[0037] S3. Add noise contamination to the transformed depth features;

[0038] Specifically, this includes using the following formula: f n =f t +λf noise In the formula, f n To add depth features after noise contamination, f noise For feature dimension and f t The same noise matrix, λ is the noise matrix f noise The weights, in this embodiment of the invention, the noise matrix f noise The noise is randomly generated in the range of 0 to 1 and follows a Gaussian distribution.

[0039] S4. Construct a multi-layer decoding deep neural network, which includes a deconvolution layer, an interpolation upsampling layer, an unpooling layer, and a second activation function layer. Input the depth features after adding noise contamination into the multi-layer decoding deep neural network for decoding to obtain image features after removing texture features.

[0040] The network structure of the multi-layer decoding deep neural network in step S4 is symmetrically set with the network structure of the multi-layer feature extraction convolutional neural network in step S1. It includes a first type of inverse transform group consisting of deconvolution layers, interpolation upsampling layers and corresponding second activation function layers, and a second type of inverse transform group consisting of deconvolution layers, unpooling layers and corresponding second activation functions. The first type of inverse transform group is used to remove texture features that are different between the source and target domains, and the second type of inverse transform group is used to retain texture features that are the same between the source and target domains. The first type of inverse transform group and the second type of inverse transform group are alternately linked.

[0041] In this invention, symmetrical setup means that, for example, if the feature extraction part has 8 layers, then the decoding convolutional neural network also has 8 layers. The parameter settings such as the number and size of the deconvolution kernels in the first layer of the decoding convolutional neural network are the same as the parameter settings such as the number and size of the convolution kernels in the 8th layer of the feature extraction part, that is, a mirror-like setup.

[0042] The inventive steps of this invention will be further explained below.

[0043] Texture feature removal works on both the source and target domains to preserve features present in both domains, thereby reducing the domain gap between the source and target domains. In this embodiment of the invention, the domain refers to the sample set composed of image features extracted by a multi-layer feature extraction convolutional neural network.

[0044] Texture feature removal works on both the source and target domains, meaning that samples from both the source and target domains are processed through steps S1-S4.

[0045] The core of this invention is that an image is fed into a feature extraction network to extract depth features, and then the transformation formula in step S2 is used to obtain the transformed features. Subsequently, the transformed features are added to noise, and finally, the image is fed into a decoding network to restore the features to the image. The resulting image has had certain texture features removed. The transformations in steps S1 to S4 can make images of different styles look more similar after being transformed by this invention, thus reducing the domain gap. Reducing the domain gap can improve the effectiveness of transfer learning methods.

[0046] When performing texture feature removal, the neuron connection weights in the feature extraction convolutional neural network and the decoding deep neural network are fixed. In this invention, the neuron connection weights refer to the parameters in the deep neural network.

[0047] The weights of the feature extraction convolutional neural network and the decoding deep neural network are first trained by reconstructing the input image. That is, the feature extraction convolutional neural network and the decoding deep neural network are directly connected. The deep features obtained by the forward propagation of the input image through the feature extraction convolutional neural network are directly fed into the decoding convolutional neural network to obtain the reconstructed output image. The pixel difference between the output image and the input image is calculated, and this is used as the basis for adjusting the neuron connection weights of the feature extraction convolutional neural network and the decoding convolutional neural network, so that the network is adjusted in the direction of minimizing the pixel difference.

[0048] This invention completes the training of the feature extraction convolutional neural network and the decoding convolutional neural network, which can be specifically achieved using the following methods.

[0049] During the training phase, the feature extraction convolutional neural network and the decoding convolutional neural network are directly connected, with the input image x... i The input is fed into a feature extraction convolutional neural network for feature extraction. The network is then processed by convolution e(·) to obtain the deep features f = e(x). i Then, the depth features f are directly fed into the decoding convolutional neural network d(·) to obtain the reconstructed image x. r =d(f), calculate the reconstructed image x rWith input image x i The pixel differences are used as the loss function. Optimize the connection weights of network neurons to make the loss function Minimum. The input image can be any image.

[0050] After the feature extraction convolutional neural network and the decoding convolutional neural network are trained, all neuron connection weights are frozen, and a noise feature generation unit is added. At this point, feeding in an image yields an image with texture features removed. The network structure is as follows: Figure 2 As shown.

[0051] The overall transformation formula can be expressed as: It is important to note that d() is a multi-layer deep neural network for decoding, and e() is a multi-layer convolutional neural network for feature extraction.

[0052] The purpose of adding a noise feature generation unit is to use noise information to contaminate the feature extraction of the deep features output by the convolutional neural network. This allows the texture features to be removed to the greatest extent after the contaminated deep features are decoded by the decoding convolutional neural network, thus maximizing the removal of texture features and improving the processed neighborhood distance between the source and target neighborhood images.

[0053] In decoding convolutional neural networks, two different upsampling methods are used: an inverse transform group (consisting of deconvolutional layers, interpolation upsampling layers, and activation function layers) using interpolation upsampling layers, and an inverse transform group (consisting of deconvolutional layers, unpooling layers, and activation function layers) using unpooling layers. These two inverse transform groups are used alternately. The inverse transform group using interpolation upsampling layers can greatly remove texture features; however, too many upsampling layers can destroy the shape and contour features that need to be preserved. Unpooling layers, on the other hand, can greatly preserve shape and contour features. Alternating between these two upsampling layers achieves an excellent balance between removing texture features and preserving shape and contour features.

[0054] The following specific embodiments verify the image texture feature removal effect of the present invention.

[0055] Since the method of this invention shortens the domain distance between the source and target domains by removing texture features, thereby improving the effectiveness of transfer learning methods, it needs to be compared with conventional transfer learning methods during experimental verification to prove the effectiveness of the method of this invention. In the comparative experiment, the most widely used and simplest transfer learning method, fine-tuning, is used as the basic transfer learning method. By comparing the data processed by the method of this invention with the data processed directly using the original image without the method of this invention, the effectiveness of the method is demonstrated.

[0056] The invention validated its method in two self-constructed tasks: side-scan sonar image target classification and forward-looking sonar image target detection. For both tasks, source domain datasets composed of conventional optical images were constructed, specifically including side-scan sonar image classification and forward-looking sonar image target detection tasks.

[0057] 1.1 Side-scan sonar image classification task

[0058] The source domain uses remote sensing images, divided into three categories: aircraft, ships, and others, totaling 2000 samples. The images are derived from a remote sensing image dataset, cropped and filtered. Examples of samples from the dataset are shown below. Figure 3 As shown.

[0059] The target domain consists of side-scan sonar images, categorized into three types: aircraft, shipwrecks, and others. There are 34 aircraft samples, 197 shipwreck samples, and 256 other samples. 30% of the samples are randomly selected as the validation set. Example samples from the target domain are shown below. Figure 4 As shown.

[0060] In the experiment, the method proposed in this invention was first applied to the source and target domains to remove texture features. The effect after removing texture features is as follows: Figure 5 As shown, finally, the fine-tune method is applied to the dataset with texture features removed, and the comparison experiment is fine-tune without using this method.

[0061] The basic steps of the fine-tuning experiment are as follows: First, train the classification network using the source domain dataset. After training, use the target domain dataset to fine-tune the weights of the source domain classification network. Finally, evaluate the number of correctly classified samples, global accuracy, and mean accuracy of the fine-tuned target domain classification network on the validation set. The accuracy comparison table is shown in Table 1.

[0062]

[0063] Table 1. Accuracy of Side-Scan Sonar Image Classification Experiment

[0064] As can be seen from the table above, after using this method to remove texture features from the source and target domains, the mean accuracy of the target domain classification network on the validation set is effectively improved.

[0065] 1.2. Forward-looking sonar image target detection task

[0066] The source domain is a self-constructed pedestrian image dataset, consisting of 177 samples collected by surveillance cameras. Some sample examples are shown below. Figure 6As shown. The target domain is a forward-looking sonar image dataset, the target is a diver, and there are a total of 42 samples. 10 images are randomly selected as the test set, and the remaining 32 are used as the training set. Examples of image samples from the target domain are shown below. Figure 7 As shown.

[0067] The overall experimental and comparative experimental process is the same as that in section 1.1. The effect after removing texture features from the source region is as follows: Figure 8 As shown.

[0068] The basic experimental steps are as follows: First, train the detection network using the source domain dataset. After training, fine-tune the weights of the source domain detection network using the target domain dataset. Finally, evaluate the average recall (AP) of the fine-tuned target domain detection network when the IOU on the validation set is 0.5. 50 And the average recall (AP) when IOU is 0.5-0.95. all The results are shown in Table 2.

[0069] Fine-tuning without incorporating the method of this invention 0.632 0.246 fine-tuning in conjunction with the method of this invention 0.711 0.356

[0070] Table 2 shows the average recall rate of forward-looking sonar image detection experiments. Based on the results in the table, it can be seen that the accuracy of target detection is greatly improved after applying the method of this invention, resulting in more precise target detection. Examples of some detection results are shown below. Figure 9 As shown.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0072] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for removing texture features from sonar images based on deep neural networks, characterized in that, The method includes the following steps: S1. Construct a multi-layer feature extraction convolutional neural network, which includes a convolutional layer, a pooling layer and a first activation function layer, and use the multi-layer feature extraction convolutional neural network to extract the depth features of the input image; S2. Use the feature transformation formula to perform feature transformation on the extracted depth features to obtain the feature-transformed depth features; S3. Add noise contamination to the transformed depth features; S4. Construct a multi-layer decoding deep neural network. Input the depth features after adding noise contamination into the multi-layer decoding deep neural network for decoding to obtain image features after removing texture features. Specifically, the network structure of the multi-layer decoding deep neural network is symmetrically set with the network structure of the multi-layer feature extraction convolutional neural network in step S1. It includes a first type of inverse transform group consisting of deconvolution layers, interpolation upsampling layers and corresponding second activation function layers, and a second type of inverse transform group consisting of deconvolution layers, unpooling layers and corresponding second activation functions. The first type of inverse transform group is used to remove texture features that are different in the source and target domains, and the second type of inverse transform group is used to retain texture features that are the same in the source and target domains. The first type of inverse transform group and the second type of inverse transform group are alternately linked.

2. The sonar image texture feature removal method based on deep neural networks according to claim 1, characterized in that, In step S1, the multi-layer feature extraction convolutional neural network is composed of multiple stacked feature extraction units, and each feature extraction unit is formed by sequentially connecting a convolutional layer, a pooling layer and a corresponding first activation function layer.

3. The sonar image texture feature removal method based on deep neural networks according to claim 1, characterized in that, The feature transformation formula mentioned in step S2 includes In the formula For multi-layer feature extraction, deep features are extracted by convolutional neural networks. The transformed depth features Let be a diagonal matrix, and the eigenvalues ​​of the diagonal matrix are equal to covariance matrix , for The covariance matrix is ​​an orthogonal matrix, and the covariance matrix satisfies the formula: 。 4. The sonar image texture feature removal method based on deep neural networks according to claim 1, characterized in that, In step S3, noise contamination is added to the transformed depth features, specifically including... In the formula, To add depth features after noise contamination, For feature dimensions and The same noise matrix, Noise matrix The weight.

Images