Sonar image generation method fusing stable diffusion and style migration

Abstract

The invention discloses a sonar image generation method fusing stable diffusion and style migration, and belongs to the field of underwater sonar image generation and image processing. The method comprises the following steps: firstly, making a data set for training a composite image model; thirdly, finely adjusting a large-scale stable diffusion model by using the manufactured data set, so that the large-scale stable diffusion model has the capability of synthesizing sonar images in a specific field; and finally, synthesizing a sonar image with specified content by using a style injection method in a reasoning stage by utilizing the generation capability of the fine tuning weight model. And quantitative and qualitative evaluation analysis is carried out on the synthesized image. According to the image generation method established by the invention, the generated image is closer to the performance of real sonar data, the diversity of the sonar data is remarkably improved, and richer and more reliable data support is provided for subsequent sonar image analysis and application.

Description

Technical Field

[0001] The invention belongs to the field of underwater sonar image generation and image processing, and in particular relates to a sonar image generation method integrating stable diffusion and style migration. Background Art

[0002] Sonar images are an important data source for underwater target detection and environmental monitoring, and are mainly collected through sonar sensors (such as side-scan sonar, forward-looking sonar, etc.). Since sonar data acquisition usually relies on high-cost, complex hardware facilities and specific operating environments, the process of acquiring sonar images is both expensive and challenging. In addition, there is the problem that sonar image data cannot be made public, making the existing sonar image data scarce and single-target. This data scarcity has, to a certain extent, restricted downstream tasks based on sonar images, such as underwater target classification, target detection, and research and development of sonar image semantic segmentation methods. At the same time, high-quality and diverse sonar image data is of great significance in promoting the advancement of underwater exploration technology, improving the accuracy of environmental monitoring, and enhancing the autonomous perception capabilities of underwater robots.

[0003] For sonar image generation, on the one hand, by generating sonar data, the number of samples available for model training can be increased, thereby alleviating the problem of data scarcity. On the other hand, by creating synthetic sonar data with known features and labels, the model can learn and understand the sonar image features of different targets more quickly.

[0004] The most direct and effective sonar image amplification methods are sonar image simulation and sonar image generation. Sonar image simulation simulates real sonar images by establishing underwater sound propagation models, seabed terrain simulation, emission directivity and platform motion characteristics. Bell et al. systematically explained the modeling and simulation methods of side-view acoustic images, and made improvements in seabed texture and sonar modeling in subsequent studies. When the Generative Adversarial Network (GAN) appeared, adversarial networks were used to generate more realistic sonar images. Li Xiang et al. used optical images as guide images and successfully generated high-quality sonar images using the improved CycleGAN network.

[0005] The challenge of sonar image synthesis is how to generate high-quality and diverse data to ensure consistency with real data while improving data diversity. This includes considering the noise characteristics of sonar sensors, the shadow characteristics of sonar images, and seabed structure factors. However, sonar image simulation technology has the problem of accurately simulating the reflection, noise and complex physical behavior of target shapes of sonar signals, resulting in the lack of detailed features of the generated images, such as low contrast between image targets and backgrounds, and inconsistent image blur, which directly affects the authenticity of the synthesized images. With the development of deep learning technology, existing studies have mostly used GAN methods to generate more realistic sonar images. However, this method still faces high data requirements and is prone to mode collapse during model training. In addition, the images generated by the model rely heavily on the diversity of training data and perform poorly when generating targets of other categories or different features.

[0006] The generation of sonar images not only needs to consider their unique details and imaging features, but also must ensure the stability of the model and the ability to generate diverse images. Therefore, the present invention provides a sonar image generation method that integrates stable diffusion and style transfer. Summary of the invention

[0007] Technical issues to be solved:

[0008] In order to avoid the shortcomings of the prior art, the present invention provides a sonar image generation method that integrates stable diffusion and style migration, utilizes the generation capability of the fine-tuned diffusion model, introduces a style injection method in the inference stage, and quickly synthesizes sonar image data of specified content without updating the model gradient. The present invention provides a solution for sonar image data synthesis, and greatly promotes the development of underwater detection technology by performing effective image synthesis on limited sonar image data.

[0009] The technical solution of the present invention is: a sonar image generation method integrating stable diffusion and style migration, and the specific steps are as follows:

[0010] Construct a data set for training a stable diffusion model; obtain original sonar sample images, preliminarily classify the image data according to image quality evaluation indicators, and then perform target grid classification on the preliminarily classified image data according to target categories, and then add corresponding content labels to the image data that has completed target grid classification to obtain a data set;

[0011] Based on the constructed dataset, the low-rank adaptation method is used to fine-tune the stable diffusion model; the conditional denoising network in the stable diffusion model is adjusted, and cross attention is used in the denoising process to capture the correlation between the input text features and image features, so that the model can adaptively adjust the focused image features according to the content of the text input;

[0012] Combining fine-tuned stable diffusion model and style injection technique to synthesize diverse sonar images.

[0013] A further technical solution of the present invention is: the preliminary classification is based on image edges, textures, color contrast, and image blur; the target grid classification is based on shipwrecks, sunken planes, drowned people, raised rocks, and seabed terrain; the content labels include target structure, shadow features, and background features, wherein the target structure describes the shape and outline of the target, the shadow features describe the height and shape of the target relative to the seabed protrusion, and the background features describe the type of terrain and the contrast with the target.

[0014] A further technical solution of the present invention is: the conditional denoising network is a Unet model with cross attention in a stable diffusion model, and the attention layer in the Unet model is calculated as follows:

[0015] The Unet model maps the input image features to the query feature vector Q, which is expressed as follows:

[0016] Q=W Q ·φ img (1)

[0017] Where W Q is the weight matrix parameter to be learned, φ img Represents the input image features;

[0018] The Unet model maps the input text features into key feature vectors K and value feature vectors V, as shown below:

[0019] K=W K ·φ text ,V=W V ·φ text (2)

[0020] Where W K and W V is the weight matrix parameter that the model needs to learn, φ text Represents text features;

[0021] The attention mechanism dynamically adjusts the focus of image features to obtain the attention output, which is expressed as follows:

[0022]

[0023] Use the Softmax function to convert the score into a probability distribution and get the matching weight, d k represents the dimension of the key feature vector, i.e. the scaling factor;

[0024] The image features extracted by the model are updated using the attention output. The expression is as follows:

[0025] φ′ img =φ img +Attention(Q,K,V) (4).

[0026] A further technical solution of the present invention is: the specific process of fine-tuning the stable diffusion model using the low-rank adaptation method is:

[0027] Calculate the learning weight matrix W of the attention layer of the Unet model through the low-rank adaptation method Q , by introducing two low-rank matrices A and B, Q Perform low-rank adaptation, the expression is as follows:

[0028] W Q =W Q0 +ΔW Q =W Q0 +A Q B Q (5)

[0029] Where W Q0 Represents the original weight matrix of the pre-trained model, ΔW Q Indicates that the introduced low-rank matrix needs to be trained to adapt to the sonar image domain, is the projection matrix, d and k represent the projection matrix W Q The dimension of d determines the dimension of the input feature, and k determines the dimension of the query feature vector, that is, the dimension of the query vector output;

[0030] During fine-tuning, the original weight matrix W of the pre-trained model is Q0 Keep frozen; the number of fine-tuned parameters is d×r+r×k, where r is the dimension of the low-rank matrix. During model training, the set r hyperparameter determines the number of parameters updated and stored during model fine-tuning.

[0031] A further technical solution of the present invention is: the pre-training process of the stable diffusion model is:

[0032] Based on the fine-tuned stable diffusion model, the forward propagation algorithm is used to calculate the predicted value of the model;

[0033] By comparing the model's predicted output with the true label, we get a value that measures the prediction error.

[0034] Through the automatic differentiation mechanism, the partial derivative of the loss function for each trainable parameter, i.e., the parameter gradient, is calculated to guide the parameter update of the model;

[0035] According to the calculated gradient and optimization algorithm, the value of each parameter is updated so that the model parameters are gradually adjusted in the direction that can reduce the loss value, thereby reducing the loss value;

[0036] At the end of each training step, the current learning rate is updated according to the given cosine annealing scheduling strategy. By continuously optimizing the parameters, the model gradually improves its performance on sonar images.

[0037] A further technical solution of the present invention is: the pre-training parameters of the stable diffusion model are set as follows: the dimension r of the low-rank matrix is ​​set to 5, the maximum number of training steps is 15000, the initial learning rate is 1e-04, and cosine annealing is used as the learning rate scheduler; the number of training steps is used as the termination condition, and in each step of training, each batch of data undergoes forward propagation, loss calculation, back propagation, optimizer update, and learning rate scheduler stepping.

[0038] A further technical solution of the present invention is that the style injection includes style features of the real sonar image and target features of the content image.

[0039] A further technical solution of the present invention is that the diversified sonar images incorporate the style of real sonar images on the basis of retaining the basic structure of the content images, thereby achieving the generation of diversified and realistic sonar target images.

[0040] A sonar image generation transposition integrating stable diffusion and style migration, including a fine-tuning module, a training module and a generation module;

[0041] Fine-tune the stable diffusion model through the fine-tuning module;

[0042] Pre-training the stable diffusion model through the training module;

[0043] Diversified sonar images incorporating the style of real sonar images are reviewed by the generation module.

[0044] A stable diffusion model includes a text encoding unit, an image information creation unit and an image decoding unit. The image information creation unit also includes a sonar image generation transpose that integrates stable diffusion and style migration.

[0045] Beneficial Effects

[0046] The beneficial effect of the present invention is that the present invention establishes a method framework for generating high-quality and diversified sonar image data. The synthesized diversified sonar image data can provide strong support for subsequent research on underwater sonar image target recognition, environmental monitoring and other tasks. Compared with the existing image generation methods, the present invention uses a fine-tuned stable diffusion model to avoid the common mode collapse problem in the generation process, and the fine-tuned model has the ability to better understand and capture the characteristics of sonar images. In addition, the style injection technology is used in the model reasoning stage to further improve the model's generation of sonar images of different styles and target contents. This method can not only quickly generate sonar images of specified content, but also generate diversified sonar images under different styles and different target contents, providing rich and reliable data resources for subsequent research in the field of sonar image target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 : The framework diagram of the established method;

[0048] Figure 2 : Sonar image samples for making datasets;

[0049] Figure 3 : Changes in loss during model training;

[0050] Figure 4 : Sonar images generated after fine-tuning the model; (a) generated sunken aircraft, (b) generated drowning person, (c) generated rough seabed terrain, (d) generated shipwreck target, (e) generated small circular target with target shadow feature;

[0051] Figure 5 : Sonar images generated by the established method; (a) content image, (b) real sonar image, (c) generated sonar image. DETAILED DESCRIPTION

[0052] The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0053] With the rapid development of deep learning and image processing technology, research based on sonar images has gradually developed towards super-resolution, target detection, semantic segmentation and other directions, but these studies require a large amount of sonar image data. Therefore, it is of great significance to carry out research on sonar image data enhancement. Diffusion model is a type of deep learning method based on probabilistic generative model. By gradually introducing noise and restoring data in the reverse process, it can efficiently generate high-quality images. In recent years, diffusion model has received extensive attention in the field of computer vision, especially in generation tasks, showing more stable and high-quality image generation capabilities than traditional generation models (such as generative adversarial networks GAN). Unlike models such as GAN, diffusion model does not rely on adversarial training, thus avoiding problems such as mode collapse. Based on this, the present invention provides a sonar image generation method integrating stable diffusion and style transfer. First, a data set for training a synthetic image model is prepared; then, a large-scale stable diffusion model is fine-tuned using the prepared data set so that it has the ability to synthesize sonar images in a specific field; finally, the generation ability of the fine-tuned weight model is used to synthesize sonar images of specified content using a style injection method in the inference stage. And the synthesized images are quantitatively and qualitatively evaluated and analyzed respectively.

[0054] The above technical solution is further described below in conjunction with the accompanying drawings:

[0055] In order to obtain high-quality and diverse sonar image data, this embodiment establishes a sonar image synthesis method based on a stable diffusion model and style injection. By utilizing the generation capability of the fine-tuned diffusion model, a style injection method is introduced in the inference stage to quickly synthesize sonar image data of specified content without updating the model gradient. This method is mainly divided into three stages. First, a sonar image dataset of a fine-tuned stable diffusion model is produced. Second, the LoRA (low rank adaptation) method is used to fine-tune the stable diffusion model on its own sonar image dataset so that the model has the ability to learn specific features of sonar images (such as noise, texture, shadows, etc. of sonar images). Finally, in the generation stage, the fine-tuned weight model is used to modify the features of the model's self-attention layer to apply the style of the real sonar image to the content image, thereby generating a sonar image of specified content. The image generation method architecture established by the present invention is as follows: Figure 1 This method can synthesize sonar images with specific requirements based on different content images and style images by using a fine-tuned weight model, thus solving the problems of scarce sonar image data and single samples.

[0056] In one embodiment, the specific steps of the sonar image synthesis method based on the stable diffusion model and style injection are as follows:

[0057] Step 1: Dataset preparation for fine-tuning the stable diffusion model

[0058] For the model training task, we first need to create a dataset for training the model. In this example, we referenced SeabedObjects KLSG and SCTD open source sonar images and fused them with our own sonar images to build a comprehensive dataset containing 1,290 images. This dataset is classified according to the targets in the image, including five categories: shipwrecks, sunken aircraft, drowned people, raised rocks, and seabed terrain. The data samples are as follows: Figure 2 Through this classification, targeted data input is provided for training the stable diffusion model, optimizing the model's image generation and recognition capabilities in complex ocean environments.

[0059] Fine-tuning the large-scale stable diffusion model requires making text labels for each sonar image. Since the contrast, target structure, shadow features, and terrain features of each sonar image are different, in order to facilitate the production of text labels, the sonar image data is preprocessed in this embodiment: first, the image edge, texture, color contrast, and image blur preprocessing features of the sonar image are extracted using the ResNet50 pre-trained network with the last classification layer removed. Then, the image features are reduced in dimension using a nonlinear dimensionality reduction method for subsequent data clustering, and the data is classified using a Gaussian distribution clustering method. The Gaussian distribution clustering method represents each cluster of data as a Gaussian distribution, and by maximizing the likelihood function of the data, the mean and variance of each cluster are estimated, and a Gaussian model is established for different data clusters.

[0060] Based on this data preprocessing method, the preprocessed data is further classified in terms of target categories through manual inspection. Finally, a text description is established for the preprocessed sonar images in terms of target structure, shadow features, and background feature contrast. The target structure mainly describes the shape and outline of the target, the shadow feature describes the height and shape of the target relative to the seabed protrusion, and the background feature describes the type of terrain and the contrast with the target. Using this method, a corresponding text label is created for each sonar image, and further, these text labels are used to generate metadata labels for training the stable diffusion model.

[0061] Step 2: Fine-tune the stable diffusion model based on low-rank adaptation method

[0062] The stable diffusion model is a powerful generative model that has the ability to generate diverse, high-quality images through pre-training on large-scale general datasets. However, the pre-trained stable diffusion model is mainly suitable for general tasks within the distribution of its training data, and its performance in specific fields or specific tasks is not ideal. In particular, there are significant differences between sonar image data and general image data, which makes the pre-trained model lack the ability to generate sonar images.

[0063] Therefore, in this embodiment, the pre-trained large-scale model is fine-tuned to enable the model to generate sonar images in a specific field. In the stable diffusion model, the denoising process is the core step in generating high-quality images. Specifically, the generation process starts with a random noise tensor in a latent space, and the model gradually recovers a clear image from the noise through a series of denoising steps. Cross-attention is used in the model denoising process to capture the correlation between the input text features and the image features, so that the model can adaptively adjust the image features of interest according to the content of the text input.

[0064] Specifically, the model maps the input image features to the query Q,

[0065] Q=W Q ·φ img (1)

[0066] Where W Q is the weight matrix parameter to be learned, φ img Represents image features.

[0067] The model maps the input text features into keys K and values ​​Q, expressed as,

[0068] K=W K ·φ text ,V=W V ·φ text (2)

[0069] Where W K and W V is the weight matrix parameter that the model needs to learn, φ text Represents text features.

[0070] The attention mechanism dynamically adjusts the focus of image features and outputs the attention weight score of the image.

[0071]

[0072] Use the Softmax function to convert the score into a probability distribution and get the matching weight, d k Represents the dimension of the key feature vector, that is, the scaling factor, to prevent the dot product value of Q,K from being too large, resulting in numerical instability, making the model gradient smaller and the training unstable.

[0073] Use the output attention weights to update the image features extracted by the model,

[0074] φ′ img =φ img +Attention(Q,K,V) (4)

[0075] The Unet model with cross attention is the denoising network of the diffusion model, which is used to predict the denoising results of the latent features in each diffusion step and is the core part of feature expression and image generation in the entire diffusion model.

[0076] For the sonar image generation task, which belongs to the data generation task in a specific field, it is necessary to retrain the pre-trained model or use an adapter to fine-tune the model so that the model can generate specified and reasonable sonar image data. However, retraining the model requires not only a large amount of data, but also sufficient computing resources, which is difficult for sonar images with scarce samples. Therefore, the low-rank adapter method is used to perform lightweight fine-tuning on the model. The low-rank adaptation method LoRA method introduces a low-rank matrix to adapt to the update of model parameters, reducing the parameters that need to be stored during the training process, and achieving efficient model fine-tuning.

[0077] Specifically, the attention layer calculation in the Unet model is shown in formulas (1) and (2). The learnable weight matrix W in the attention layer of the Unet model is Q Apply LoRA, introduce two low-rank matrices A and B respectively, and Q Perform low-rank adaptation, expressed as,

[0078] W Q =W Q0 +ΔW Q =W Q0 +A Q B Q (5)

[0079] During fine-tuning, the original weight matrix W of the pre-trained model Q0 Keep frozen, ΔW Q The low-rank matrix introduced needs to be trained to adapt to the sonar image domain. is the projection matrix. Similarly, in the weight matrix W K and W V The number of parameters for LoRA fine-tuning is d×r+r×k, where r is the dimension of the low-rank matrix. During model training, the value of parameter r is given in advance. In this embodiment, the value of r is 5. The r hyperparameter determines the number of parameters updated and stored during model fine-tuning.

[0080] In this embodiment, LoRA is applied to the Q, K, V of the attention layer of the Unet denoising model and the linear change layer of the attention output, as shown in formula (5), respectively. Q , W K , W VUsing low-rank adaptation, in the downstream change layer of the attention output, the output learnable weight matrix is ​​adapted as shown in formula (5). By specifying these modules, only the parameters of these modules are updated during training, and other model parameters remain unchanged.

[0081] In this example, the value of r is set to 5, the maximum number of training steps is set to 15000, the initial learning rate is 1e-04, and cosine annealing is used as the learning rate scheduler to adjust the learning rate during training. The number of training steps is used as the termination condition. In each step of training, each batch of data undergoes forward propagation, loss calculation, back propagation, optimizer update, and learning rate scheduler stepping.

[0082] Forward propagation refers to passing the input data through each layer of the model to calculate the predicted value of the model; then, by comparing the predicted output of the model with the true label, a value measuring the prediction error is obtained, namely, loss calculation, and the loss value is used to guide the update direction and amplitude of the model parameters, with the purpose of minimizing the loss value, thereby improving the prediction accuracy of the model; then, through the automatic differentiation (Autograd) mechanism, the partial derivative of the loss function for each trainable parameter, namely, the parameter gradient, is calculated to guide the parameter update of the model; further, according to the calculated gradient and optimization algorithm, the present invention adopts the AdamW optimization algorithm to update the value of each parameter, so that the model parameters are gradually adjusted in the direction that can reduce the loss value, thereby reducing the loss value; finally, at the end of each training step, the current learning rate is updated according to the given cosine annealing scheduling strategy. By continuously optimizing the parameters, the model gradually improves its performance on sonar images.

[0083] The loss changes during model training are as follows: Figure 3 As shown in the figure, the loss curve shows that the model has obvious fluctuations when the number of training steps is 10k. This stage is usually the period when the model quickly learns and adjusts weights, and the model tries to adapt to the new data distribution and task requirements. After 10k steps, as the number of training steps increases, the loss curve tends to be stable, indicating that the learning rate of the model slows down and enters the stable optimization stage. In this stage, the model's fit to the data tends to be stable, the parameter adjustment range decreases, and the loss value gradually approaches the minimum value, showing that the model has good adaptability to the current task. Overall, the change in the loss curve reflects the smooth progress of the fine-tuning process.

[0084] After the model training is completed, it is subjected to inference evaluation, and the generated sonar images are as follows Figure 4 As shown in the figure, the model is able to synthesize sonar images of the specified target from the generated results, but from the inference results, the model's generalization ability is weak and it can only generate the data sample type given during fine-tuning, while its ability to synthesize other types of targets is poor. Figure 4As shown in (e), when the input text condition is to generate a circular sphere target with shadow features, the target features in the generated image are not obvious enough and fail to accurately reflect the expected geometric shape and features.

[0085] In order to improve the model's ability to generate diverse sonar images, this embodiment proposes a method of introducing style features into the attention layer of the fine-tuned model. This method aims to enhance the model's adaptability to different scenes and targets and improve its generalization ability, thereby achieving accurate generation of multiple types of sonar images. Through this improvement, the proposed method shows higher practicality and reliability in generating sonar images of different targets and complex environments.

[0086] Step 3: Combine fine-tuning model and style injection techniques to synthesize diverse sonar images

[0087] In order to enhance the model's ability to generate diversified sonar images, this embodiment uses a fine-tuned stable diffusion model and a method of introducing style features in the attention layer to improve the model's generalization ability to generate sonar images of different scenes and targets, making the proposed method practical and reliable.

[0088] Style injection technology is a technology used in computer vision for image-to-image conversion. It generates new images by fusing content features with style features, aiming to convert and enhance the performance of content images. In this embodiment, the style features of real sonar images and the target features of content images are used to generate realistic and diverse sonar images.

[0089] In this embodiment, small circular targets are selected as content images to simulate small target objects commonly seen in sonar images. These circular target images are first converted into grayscale images to match the format of real sonar images. This processing step ensures that the generated images have a high similarity in visual characteristics with the actual sonar images, thereby improving the practicality and authenticity of the generated images.

[0090] Next, in order to introduce the style features of real sonar images into the content images, real sonar images are used as style feature sources in this embodiment. These real sonar images contain rich textures, noises, and features related to sonar imaging. By extracting and applying these features, the authenticity and diversity of the generated images can be enhanced.

[0091] Finally, the image generation capability of the fine-tuned diffusion model is utilized, combined with the style features of the real sonar images and the target features of the content images. The generated images not only retain the basic structure of the content images (such as small circular targets), but also incorporate the style features of the real sonar images, thus achieving diversified and realistic sonar target image generation.

[0092] Compared with the existing sonar image generation method, the method of this embodiment improves the ability of the model to generate diversified sonar target images, and the generated images have reasonable structural features of real sonar images, thereby improving the quality and diversity of the generated images.

[0093] In one embodiment, in the attention layer of the denoising network Unet model, the style features of the real sonar image are introduced through the style injection technology, as shown in formula (2), and the style features of the real sonar image are mapped to the key K and value V in the attention mechanism. The goal of style injection is to merge the style features with the content features of the original image, and to transfer the style while maintaining the content structure. Using the feature extraction capability of the fine-tuning model, the potential noise representation z of the content image and the style image is obtained. c , z s , and in the process, the query features of the content image are obtained and style image key features Value Features Represent the content image as:

[0094] Q c =W Q ·φ c (7)

[0095] Among them, φ c Represents the content features obtained from the small circular object content image.

[0096] The style features extracted from real sonar images are expressed as:

[0097] K s =W K ·φ sty ,V s =W V ·φ sty (8)

[0098] Then, we use adaptive style injection technology to inject z c and z s Fusion, get the fused stylized potential noise z cs ,Furthermore, in the image generation process, in order to guide the model to generate realistic sonar-style images with content targets, the content query feature is converted into With styled query features Mixed, where content query features The input is the potential noise z of the content image c Features extracted by the attention module of the denoising network Unet model. Stylized query features The input is the stylized potential noise z csFeatures extracted by the attention module of the denoising network Unet model.

[0099]

[0100] During model inference, the value of the parameter γ is given to adjust the degree of style injection into the generated image. In this embodiment, the value of γ is given as 0.5, so that the model pays equal attention to the features of content and style.

[0101] Finally, the mixed Key and value features with style The attention output is calculated through the attention mechanism, that is, the key features of the image that need to be paid attention to.

[0102]

[0103] Represents the attention output obtained through the attention mechanism, which is the key representation passed from the attention layer to the subsequent layers. Represents the key features obtained by the subsequent layers based on the key representation of the attention output.

[0104] This output is used to update the stylized latent noise z cs , and finally generate images with the style of real sonar images.

[0105] In one embodiment, the calculation steps of the sonar image synthesis method based on the stable diffusion model and style injection are as follows:

[0106]

[0107] By fusing content and style features in the attention layer, precise control over the stylized results is achieved, ensuring a balance and smooth transition between content preservation and style injection.

[0108] The present invention adopts small sphere and small cylinder target images similar to remote sensing images as content images, uses real sonar images as style images, and generates sonar images with real sonar image style and content targets based on the method combining fine-tuning weight model and style injection established by the present invention.

[0109] In this embodiment, the mixing ratio γ is set to 0.5, so that the model pays equal attention to the features of content and style, and the sampling time step is set to 50 steps. The sonar image generated by the established method is as follows Figure 5 shown.

[0110] The results of the generated images show that the images not only accurately depict the boundary contours of the target, but also effectively reproduce the noise patterns and imaging characteristics unique to underwater acoustic imaging, such as reflected echoes. Figure 5As shown in the third row of , when the real sonar image is low-contrast and high-noise, the generated sonar image still retains the integrity of the content target. At the same time, the model can accurately distinguish the foreground target from the background noise through the attention mechanism, which verifies the effectiveness of the style injection method based on the attention mechanism in enhancing the model's recognition ability.

[0111] The image generation method established by the present invention not only makes the generated images closer to the performance of real sonar data, but also significantly improves the diversity of sonar data, providing richer and more reliable data support for subsequent sonar image analysis and application.

[0112] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A sonar image generation method integrating stable diffusion and style migration, characterized in that The specific steps are as follows: Construct a data set for training a stable diffusion model; obtain original sonar sample images, preliminarily classify the image data according to image quality evaluation indicators, and then perform target grid classification on the preliminarily classified image data according to target categories, and then add corresponding content labels to the image data that has completed target grid classification to obtain a data set; Based on the constructed dataset, a low-rank adaptation method is used to fine-tune the stable diffusion model; Adjust the conditional denoising network in the stable diffusion model. Use cross attention in the denoising process to capture the correlation between input text features and image features, so that the model can adaptively adjust the focused image features according to the content of the text input. Combining fine-tuned stable diffusion model and style injection technique to synthesize diverse sonar images.

2. The sonar image generation method integrating stable diffusion and style migration according to claim 1, characterized in that: The preliminary classification is based on image edge, texture, color contrast, and image blur; the target grid classification is based on shipwrecks, sunken planes, drowned people, raised rocks, and seabed terrain; the content label includes target structure, shadow features, and background features, where the target structure describes the shape and outline of the target, and the shadow feature describes the height and shape of the target relative to the seabed protrusion. Background features describe the type of terrain and the contrast between it and the target.

3. The sonar image generation method integrating stable diffusion and style migration according to claim 1, characterized in that: The conditional denoising network is a Unet model with cross attention in a stable diffusion model. The attention layer in the Unet model is calculated as follows: The Unet model maps the input image features to the query feature vector Q, which is expressed as follows: Q=W Q ·φ img (1) Where W Q is the weight matrix parameter to be learned, φ img Represents the input image features; The Unet model maps the input text features into key feature vectors K and value feature vectors V, as shown below: K=W K ·f text ,V=W V ·f text (2) Where W K and W V is the weight matrix parameter that the model needs to learn, φ text Represents text features; The attention mechanism dynamically adjusts the focus of image features to obtain the attention output, which is expressed as follows: Use the Softmax function to convert the score into a probability distribution and get the matching weight, d k represents the dimension of the key feature vector, i.e. the scaling factor; The image features extracted by the model are updated using the attention output. The expression is as follows: f′ img =φ img +Attention(Q,K,V) (4).

4. The sonar image generation method integrating stable diffusion and style migration according to claim 3, characterized in that: The specific process of fine-tuning the stable diffusion model using the low-rank adaptation method is as follows: Calculate the learning weight matrix W of the attention layer of the Unet model through the low-rank adaptation method Q , by introducing two low-rank matrices A and B, Q Perform low-rank adaptation, the expression is as follows: IN Q =In Q0 +ΔW Q =In Q0 +A Q B Q (5) Where W Q0 Represents the original weight matrix of the pre-trained model, ΔW Q Indicates that the introduced low-rank matrix needs to be trained to adapt to the sonar image domain, is the projection matrix, d and k represent the projection matrix W Q The dimension of d determines the dimension of the input feature, and k determines the dimension of the query feature vector, that is, the dimension of the query vector output; During fine-tuning, the original weight matrix W of the pre-trained model is Q0 Keep frozen; the number of fine-tuned parameters is d×r+r×k, where r is the dimension of the low-rank matrix. During model training, the set r hyperparameter determines the number of parameters updated and stored during model fine-tuning.

5. The sonar image generation method integrating stable diffusion and style migration according to claim 4, characterized in that: The pre-training process of the stable diffusion model is: Based on the fine-tuned stable diffusion model, the forward propagation algorithm is used to calculate the predicted value of the model; By comparing the model's predicted output with the true label, we get a value that measures the prediction error. Through the automatic differentiation mechanism, the partial derivative of the loss function for each trainable parameter, i.e., the parameter gradient, is calculated to guide the parameter update of the model; According to the calculated gradient and optimization algorithm, the value of each parameter is updated so that the model parameters are gradually adjusted in the direction that can reduce the loss value, thereby reducing the loss value; At the end of each training step, the current learning rate is updated according to the given cosine annealing scheduling strategy. By continuously optimizing the parameters, the model gradually improves its performance on sonar images.

6. The sonar image generation method integrating stable diffusion and style migration according to claim 5, characterized in that: The pre-training parameters of the stable diffusion model are set as follows: the dimension r of the low-rank matrix is ​​set to 5, the maximum number of training steps is set to 15000, the initial learning rate is set to 1e-04, and cosine annealing is used as the learning rate scheduler; The number of training steps is used as the termination condition. In each step of training, each batch of data undergoes forward propagation, loss calculation, backpropagation, optimizer update, and learning rate scheduler stepping.

7. The sonar image generation method integrating stable diffusion and style migration according to claim 6, characterized in that: The style injection includes style features of the real sonar image and target features of the content image.

8. The sonar image generation method integrating stable diffusion and style migration according to claim 7, characterized in that: The diversified sonar images incorporate the style of real sonar images while retaining the basic structure of the content images, thereby achieving the generation of diversified and realistic sonar target images.

9. A sonar image generation transpose integrating stable diffusion and style transfer, used to implement the sonar image generation method integrating stable diffusion and style transfer as claimed in any one of claims 1 to 8; characterized in that: Includes fine-tuning module, training module and generation module; Fine-tune the stable diffusion model through the fine-tuning module; Pre-training the stable diffusion model through the training module; Diversified sonar images incorporating the style of real sonar images are reviewed by the generation module.

10. A stable diffusion model, characterized in that: It includes a text encoding unit, an image information creation unit and an image decoding unit, wherein the image information creation unit also includes a sonar image generation transpose that integrates stable diffusion and style migration as described in claim 9.

Images