Seabed sediment image feature enhancement method based on image style migration

Abstract

The invention discloses a seabed sediment image feature enhancement method based on image style migration, and the method introduces a Transform architecture, overcomes the problem that deviation representation exists when image style migration is realized based on a traditional convolutional neural network, and realizes unbiased style migration of an image. In the training optimization process of the network, the Huber loss is used for guiding and constraining the optimization of an image style migration Transform model; an encoder and a decoder which are simpler in structure are used for extracting and fusing content features and style features of an image to generate a stylized image, so that an over-fitting phenomenon of a model caused by smaller sample capacity of a data set can be prevented, and the calculation complexity of the model can be reduced. According to the method, the content features of the high-physical-resolution optical image and the style features of the low-physical-resolution seabed sediment sonar image are organically combined, and effective seabed sediment image feature enhancement is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of underwater acoustic image processing and relates to a method for processing seabed sediment images, and in particular to a method for enhancing seabed sediment image features based on image style migration. Background Art

[0002] The analysis and recognition of seafloor sediment images is a core task in fields such as marine resource exploration and marine engineering. Seafloor sediment type (e.g., sand, mud, rock, coral reef, etc.) is directly related to the distribution of mineral resources (such as polymetallic nodules and rare earth elements), oil and gas reservoir structure, and submarine cable / pipeline routing. High-precision sediment images reveal sediment physical properties (such as grain size and hardness), providing critical information for resource assessment and mining plans. The safe operation of submarine pipelines, drilling platforms, and underwater robots (AUVs) is highly dependent on the stability of the sediment. Subsurface image recognition can assess the risk of submarine landslides and liquefaction, as well as the bearing capacity of anchorage areas, to prevent damage to engineering facilities caused by subsurface collapse or erosion. Furthermore, AUVs (autonomous underwater vehicles) must adjust their navigation strategies in real time based on the sediment type to avoid getting stuck in soft mud or colliding with reefs.

[0003] The detection capability of underwater optical equipment will be greatly weakened as the propagation distance increases, and it is easily affected by the clarity of the sea water. Sonar equipment has the advantages of long underwater detection distance and is not easily affected by the clarity of the water body. Therefore, the current recognition of seabed sediments mainly relies on the acoustic image signals collected by sonar equipment. However, the physical resolution of seabed sediment acoustic images is usually at the meter level or sub-meter level, while the physical resolution of optical images such as faces taken by cameras is at the millimeter level or sub-millimeter level. In comparison, images with high physical resolution can provide more detailed information such as subtle texture changes, which can help the classification model more accurately construct the mapping relationship between images and types. Therefore, the use of acoustic images to realize seabed sediment recognition has the following problems: (1) Sonar images with low physical resolution will lose many image details, and the features in the image are more sparse; (2) It is difficult for the classification model to learn the mapping relationship between seabed sediment images and their types from data with sparse features; (3) The current image feature enhancement method has the problem of biased representation. Summary of the Invention

[0004] The present invention addresses the problems of sparse sonar image features, poor model representation ability of seabed substrate images with sparse features, and biased representation in current image feature enhancement methods. By combining the content features of high-physical-resolution optical images and the style features of low-physical-resolution seabed substrate sonar images, a seabed substrate image feature enhancement method based on image style transfer is provided.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for enhancing seabed image features based on image style transfer includes the following steps:

[0007] Step 1: Acquire a low-resolution sonar image of the seabed;

[0008] Step 2: Obtain a high-resolution optical image that is similar in visual effect to the seabed sonar image as content image data;

[0009] Step 3: Construct a content-aware positional encoding that adds the relative position of each image patch to the input sequence;

[0010] Step 4: Construct Transformer encoders to represent content features and style features respectively. By capturing the dependencies in the input sequence, the image features are captured, the overall content of the image is understood, and the abstract features of the image are extracted.

[0011] Step 5: Build a Transformer-based decoder to convert the image content feature sequence and image style feature sequence output by the encoder into a specific image;

[0012] Step 6: Construct a convolutional neural network decoder that considers the uniformity of feature scale to restore the size information of the image;

[0013] Step 7: Construct an objective function network to evaluate the difference between the feature-enhanced image and the original style image;

[0014] Step 8: Select RMSE, PSNR, SSIM, UQI, and image content loss , image style loss , id1 loss , id2 loss As an objective evaluation indicator;

[0015] Step 9: Conduct seabed sediment image feature enhancement model parameter and hyperparameter optimization experiments to obtain the best experimental results of the model;

[0016] Step 10: Use the model selected in step 9 to enhance the seabed sediment image features.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. This paper introduces the Transformer architecture, which overcomes the problem of biased representation in traditional convolutional neural networks when implementing image style transfer, and realizes unbiased style transfer of images.

[0019] 2. Use Huber loss in the training optimization process of the network to guide and constrain the optimization of the image style transfer Transformer model. Huber loss combines the advantages of mean square error and absolute value error, which can obtain more stable gradient in the optimization process of the network, and improve the image style transfer performance of the image style transfer Transformer model.

[0020] 3. A simpler encoder and decoder are used to extract and fuse the content features and style features of the image to generate a stylized image, which can prevent overfitting of the model due to small sample size of the data set, and reduce the computational complexity of the model. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of the seabed bottom image feature enhancement method based on image style transfer.

[0022] Figure 2 The overall framework of the seabed bottom image feature enhancement method based on image style transfer.

[0023] Figure 3 A schematic diagram of content-aware position encoding.

[0024] Figure 4 The structure of the Transformer encoder.

[0025] Figure 5 The structure of the Transformer decoder.

[0026] Figure 6 The structure of the convolutional neural network decoder considering feature scale unification. DETAILED DESCRIPTION

[0027] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto. Any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0028] In existing seabed bottom identification research, there are problems of sparse sonar image features, difficulty for classification model to represent sparse sonar image features, and biased representation in current image feature enhancement methods. In view of this, the present application provides a seabed bottom image feature enhancement method based on image style transfer, which combines the content features of high physical resolution optical images and the style features of low physical resolution seabed bottom sonar images, and realizes effective seabed bottom image feature enhancement. As shown in Figure 1 The method comprises the following steps:

[0029] Step 1: Obtain a low-physical-resolution seafloor bottom type sonar image, and divide the part of the data into two parts, which are a training set and a test set, wherein the training set is used as both a style dataset for image style transfer and a training set for a classification model.

[0030] In this step, the low-physical-resolution seafloor bottom type sonar image can be obtained by various devices such as side scan sonar and multi-beam sonar, and the source can not be limited. The collected sonar image is segmented and screened to obtain a high-quality data set. The size of the image can not be limited, but in the research field, the length and width of the image are usually the same. In addition, in order to prove the effectiveness of the present application, the number of samples in the training set is 0.1 times the number of samples in the test set.

[0031] Step 2: Obtain a high-physical-resolution optical image with similar visual effects to the seafloor bottom type sonar image as content image data.

[0032] In this step, images with the same data type as the seafloor bottom type or similar visual effects in the Kylberg texture image dataset and publicly available soil texture images are selected as content images. The size of the image is also not limited, but in the research field, the length and width of the image are usually the same.

[0033] Step 3: Construct a content-aware position encoding, and add the relative position of each image block to the input sequence to ensure that the model can learn the relative position information of the image block.

[0034] In this step, image blocks with similar content should have similar styles, and Figure 3 In the left image, the distance between the red image block and the green image block and the distance between the red image block and the blue image block should be small. Therefore, the semantic information of the image needs to be considered when calculating the position encoding. In addition, as Figure 3 As shown in the right image, when the size of the image changes, the relative position of the two image blocks will change greatly, which will also affect the effect of image style transfer. Therefore, the traditional cosine-based position encoding may not be suitable for multi-scale image style transfer tasks. Therefore, the present application introduces a content-aware position encoding that can adapt to multi-scale and is more suitable for image style transfer tasks. For any image, the content-aware position encoding will scale the original position encoding to , so the size of the image will not affect the relative position of the two image blocks. The content-aware position encoding is as follows:

[0035]

[0036] wherein, is the content-aware position encoding of the image block , is the weight, It follows the sequence Variational learnable positional encoding, It is a learnable The positional encoding function of the convolution, is the number of adjacent image blocks.

[0037] Finally, the content information of the image is encoded as As the input of the Transformer encoder, the encoding The mathematical expression is as follows:

[0038]

[0039] in, and denote the input sequence and the corresponding content-aware position encoding, respectively.

[0040] Step 4: Construct Transformer encoders to represent content features and style features respectively. The main function of the encoder is to capture image features by capturing the dependencies in the input sequence, understand the overall content of the image, and thus extract the abstract features of the image and convert the input sequence into a meaningful internal representation vector.

[0041] In this step, the constructed Transformer encoder mainly consists of three encoder layers, each of which includes a multi-head attention module, a residual module, a regularization operation, and a feedforward neural network. Figure 4 The overall architecture and parameter settings of the Transformer encoder are shown.

[0042] For any image content input sequence , the input sequence is first encoded as a query ( ),key( ) and value( ), , , Calculate according to the following formula:

[0043]

[0044] in, , , The input sequence is converted into , , The encoding matrix.

[0045] Then, the multi-head attention mechanism is used to calculate , , Score, multi-head attention It is composed of multiple self-attentions, and its mathematical expression is shown as follows:

[0046]

[0047] in, is the input, is the learnable parameter matrix, The number of self-attentions.

[0048] Then, a feedforward neural network with residual connections is used to calculate and encode the multi-head attention scores to obtain the encoding sequence of content features. :

[0049]

[0050] in, , , , are the learnable parameters of the 2-layer feedforward neural network in the Transformer encoder layer. In addition, layer regularization is applied after each module mentioned above. Then, the input image content sequence is encoded into .

[0051] Similarly, for any image style input sequence , the Transformer encoder encodes the style input sequence according to the same calculation rules as ,Different from the input image content sequence, the style sequence input does not include position encoding information,,because the structure of the style features does not need to be preserved in the final output.

[0052] Step 5: Build a Transformer-based decoder. The decoder's task is to convert the image content feature sequence and image style feature sequence output by the encoder into a specific image.

[0053] In this step, the decoder structure based on Transformer is constructed as follows Figure 5 As shown in Figure 2, the Transformer decoder consists of a multi-head attention mechanism, summation & regularization, and a feedforward neural network. The decoder uses the image’s content feature sequence to Building a query , using the style feature sequence of the image Build Key Sum , the specific calculation process is shown in the following formula:

[0054]

[0055] in, , , The input sequence is converted into , , Then, a feedforward neural network with residual connections is used to calculate and encode the multi-layer multi-head attention scores to obtain the output image sequence , the specific calculation process is shown in the following formula:

[0056]

[0057] Layer regularization operations are applied after each of the above modules.

[0058] Step 6: Construct a convolutional neural network decoder that considers the uniform feature scale to restore the size information of the image.

[0059] In this step, the convolutional neural network decoder constructed considering the uniform feature scale is composed of a convolution layer, an activation function, a padding function, and an upsampling function. All padding functions fill one row on the top, bottom, left, and right of the feature map. The structure of the decoder is as follows: Figure 6 As shown in Table 1, the specific parameters of the decoder are shown.

[0060] Table 1 Specific parameters of the convolutional neural network decoder

[0061]

[0062] Step 7: Construct an objective function network to evaluate the difference between the feature-enhanced image and the original style image.

[0063] In this step, Huber loss is more robust to outliers than mean square error in regression tasks. It combines the advantages of mean square error and absolute value error and can obtain more stable gradients in the network optimization process. Therefore, the present invention introduces Huber loss. Since the original VGG network model is relatively complex, the present invention constructs an objective function network with lower computational complexity based on the VGG network, and uses the image features extracted from different layers in the network to construct image content loss. and image style loss Huber loss, and The mathematical expression is as follows:

[0064]

[0065] in, is the input of the Huber loss function, The first one in the simplified version of the pre-trained VGG network The features extracted by the layer, is the number of layers in the network, and are the output image and the content image respectively. and They represent the mean and variance of the extracted features respectively.

[0066] Table 2 shows the specific parameters of the objective function network constructed by the present invention.

[0067] Table 2 Specific parameters of the objective function network

[0068]

[0069] In addition, the present invention also introduces identity loss to ensure that the image style transfer transformer can obtain a more comprehensive and accurate representation of image content and image style. Specifically, when two images with the same content or style are placed in the image style transfer transformer, the content features or style features of the generated image should be the same as those of the input image. The id1 loss defined in the present invention The mathematical expression is as follows:

[0070]

[0071] in, , , , They represent the generated content image, the input content image, the generated style image, and the input style image respectively.

[0072] ID2 loss The mathematical expression is as follows:

[0073]

[0074] Finally, by minimizing the optimization function of the image style transfer Transformer To implement network updates, image style migration Transformer optimization function As shown in the following formula:

[0075]

[0076] in, , , , is the loss function weight.

[0077] Step 8: Determine the evaluation indicators of the effectiveness of the present invention and select some mainstream methods as baseline methods to prove the effectiveness of the present invention.

[0078] In this step, in order to prove the effectiveness of the present invention, RMSE, PSNR, SSIM, UQI, image content loss are introduced. , image style loss , id1 loss , id2 loss As an objective evaluation indicator. The subjective evaluation indicator is to select 10 researchers engaged in image processing or seabed identification to subjectively score the generated images, with scores ranging from 1 to 5. When the average score is less than 4, the method is directly judged as unqualified; when the average score is greater than or equal to 4, the effectiveness of each method is evaluated according to the objective evaluation indicators. Among the objective evaluation indicators, SSIM and UQI have the same and largest weights, RMSE and PSNR have the same and smaller weights than SSIM and UQI. Image content loss is only considered when the image effect generated by the image style transfer Transformer cannot be evaluated based on the four indicators of RMSE, PSNR, SSIM, and UQI. , image style loss , id1 loss , id2 loss Four factors.

[0079] In addition, in order to prove the effectiveness of the present invention, AdaIN, AdaAttN, ArtFlow, MASTN, IESTN, and StyTr2 methods are selected as baseline methods.

[0080] Step 9: Conduct seabed sediment image feature enhancement model parameter and hyperparameter optimization experiments to obtain the best experimental results of the model.

[0081] In this step, the model iteration number epoch, initial learning rate , learning rate adjustment coefficient , loss function coefficient , , , We experimented with batch size batch_size to achieve optimal results. Multiple experiments have shown that when the hyperparameters are set to the values ​​shown in Table 3, the model performance is optimal.

[0082] Table 3 Overall hyperparameter settings of the model

[0083]

[0084] Step 10: Compare the model selected in step 9 with the baseline method to demonstrate the effectiveness of the present invention. Table 4 shows the comparison of evaluation indicators of different methods. According to the evaluation criteria in step 8, the present invention has the best experimental results.

[0085] Table 4 Comparison of evaluation indicators of different image style transfer methods

[0086]

Claims

1. A method for enhancing seabed image features based on image style transfer, characterized in that The method comprises the following steps: Step 1: Acquire a low-resolution sonar image of the seabed; Step 2: Obtain a high-resolution optical image that is similar in visual effect to the seabed sonar image as content image data; Step 3: Construct a content-aware positional encoding that adds the relative position of each image patch to the input sequence; Step 4: Construct Transformer encoders to represent content features and style features respectively. By capturing the dependencies in the input sequence, the image features are captured, the overall content of the image is understood, and the abstract features of the image are extracted. Step 5: Build a Transformer-based decoder to convert the image content feature sequence and image style feature sequence output by the encoder into a specific image; Step 6: Construct a convolutional neural network decoder that considers the uniformity of feature scale to restore the size information of the image; Step 7: Construct an objective function network to evaluate the difference between the feature-enhanced image and the original style image; Step 8: Select RMSE, PSNR, SSIM, UQI, and image content loss , image style loss , id1 loss , id2 loss As an objective evaluation indicator; Step 9: Conduct seabed sediment image feature enhancement model parameter and hyperparameter optimization experiments to obtain the best experimental results of the model; Step 10: Use the model selected in step 9 to enhance the seabed sediment image features.

2. The method for enhancing seabed image features based on image style transfer according to claim 1 is characterized in that In step 3, for any image, content-aware position coding will convert The original position encoding is scaled to The content-aware position encoding is as follows: in, is an image block Content-aware position encoding, is the weight, It follows the sequence Variational learnable positional encoding, It is a learnable The positional encoding function of the convolution, is the number of adjacent image blocks; Finally, the content information of the image is encoded as As the input of Transformer encoder, the encoding The mathematical expression is as follows: in, and denote the input sequence and the corresponding content-aware position encoding, respectively.

3. The method for enhancing seabed image features based on image style transfer according to claim 1 is characterized in that In step 4, for any image content input sequence , the input sequence is first encoded as a query ,key Sum , , , Calculate according to the following formula: in, , , The input sequence is converted into , , The encoding matrix of Then, the multi-head attention mechanism is used to calculate , , Score, multi-head attention The mathematical expression is as follows: in, is the input, is the learnable parameter matrix, The number of self-attention points; Then, a feedforward neural network with residual connections is used to calculate and encode the multi-head attention scores to obtain the encoding sequence of content features. : in, , , , are the learnable parameters of the 2-layer feed-forward neural network in the Transformer encoder layer.

4. The method for enhancing seabed image features based on image style transfer according to claim 1 is characterized in that In step 6, the convolutional neural network decoder constructed with consideration of feature scale unification is composed of a convolutional layer, an activation function, a padding function, and an upsampling function, wherein all padding functions fill one row on the top, bottom, left, and right sides of the feature map.

5. The method for enhancing seabed image features based on image style transfer according to claim 1 is characterized in that The specific steps of step 7 are as follows: Refer to the VGG network to build the objective function network, introduce Huber loss, and use the image features extracted from different layers in the network to build image content loss and image style loss , Huber loss, and The mathematical expression is as follows: in, is the input of the Huber loss function, For the first The features extracted by the layer, is the number of layers in the network, and are the output image and the content image respectively, and Represent the mean and variance of the extracted features respectively; The introduction of identity loss ensures that the image style transfer Transformer can obtain a more comprehensive and accurate representation of image content and image style. The mathematical expression is as follows: in, , , , Represent the generated content image, input content image, generated style image, and input style image respectively; ID2 loss The mathematical expression is as follows: By minimizing the optimization function of the image style transfer Transformer Implement network updates and image style migration Transformer optimization functions As shown in the following formula: in, , , , is the loss function weight.

Images