Virtual sonar image generation method and system

Abstract

The application belongs to the technical field of underwater sonar imaging and intelligent identification, and relates to a virtual sonar image generation method and system. The method comprises: mapping a real optical target image to a virtual sonar target image by using a vision-text fusion network; embedding the virtual sonar target image into a real sonar background image to generate a combined image; and performing background compensation and texture consistency adjustment on the combined image by using a virtual sonar image background reconstruction network to obtain a virtual sonar image with a background. The application breaks through the bottleneck of real sonar data collection being limited by environment, cost and equipment conditions; at the same time, the target type and quantity can be flexibly expanded, effectively solving the problems of insufficient sonar sample categories and unbalanced distribution.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of underwater sonar imaging and intelligent recognition, and particularly relates to a virtual sonar image generation method and system. BACKGROUND

[0002] As an important means of underwater detection and recognition, sonar imaging technology is widely used in underwater equipment detection, target search, ocean resource investigation, security monitoring and other fields. At present, intelligent recognition methods represented by deep learning have become the main direction to improve the accuracy of sonar image target analysis, but model training is heavily dependent on large-scale labeled samples. The actual underwater sonar data acquisition is limited by water conditions, equipment types, operation costs and sea conditions, and it is difficult, costly and limited in scene coverage. Moreover, the target categories are generally insufficient and unevenly distributed. On the other hand, the manual labeling of sonar targets is time-consuming, especially in low signal-to-noise ratio scenarios where target boundaries are blurred, making it difficult to ensure the stability and consistency of the labeling results, making high-precision labeled samples even more scarce.

[0003] In order to make up for the lack of data, many studies have tried to use data augmentation, transfer learning, simulated sonar imaging models and other methods to expand the sample size. However, traditional simulation-generated images often differ significantly from real sonar images in terms of texture details, scattering patterns and noise statistical characteristics, making it difficult to support model training in terms of image authenticity and diversity. Existing augmentation methods also mainly focus on geometric or lighting changes, and cannot solve the problem of target category scarcity and high labeling cost. The construction of complex virtual samples in natural scenes with multiple backgrounds, multiple poses and multiple scales is still insufficient.

[0004] In summary, how to efficiently generate high-quality virtual sonar images that are highly consistent with real image characteristics under the condition of lacking a large amount of real sonar data, to significantly increase the sample size, enrich the target categories and reduce the labeling cost, has become a technical problem that needs to be solved. SUMMARY

[0005] The present application provides a virtual sonar image generation method and system to address the problems of sonar image data scarcity, high acquisition cost and labeling difficulty. By constructing a virtual sonar target generation module, real optical target images are effectively converted into virtual sonar target images. And using a virtual sonar image background reconstruction module, the virtual sonar target image is naturally fused with the real sonar background image to obtain high-quality virtual sonar images with background. The present application uses adversarial training and unsupervised constraints to improve the authenticity and diversity of the generated results, which can significantly expand the number and categories of sonar samples, reduce the labeling cost, and provide reliable data support for underwater target detection and recognition tasks.

[0006] To achieve the above purpose, the first aspect of the present application provides a virtual sonar image generation method, comprising:

[0007] mapping real optical target images to virtual sonar target images using a vision-texture fusion network;

[0008] embedding the virtual sonar target images into real sonar background images to generate combined images;

[0009] performing background compensation and texture consistency adjustment on the combined images using a virtual sonar image background reconstruction network to obtain virtual sonar images with background;

[0010] The embedding of the virtual sonar target images into real sonar background images to generate combined images comprises the following steps:

[0011] performing a scaling operation on the virtual sonar target images and obtaining scaled virtual sonar target images using bilinear interpolation;

[0012] selecting an insertion position in the real sonar background images and embedding the scaled virtual sonar target images into the position to generate combined images;

[0013] constructing a binary mask for the combined images wherein the region with a mask value of 1 corresponds to the region of the scaled virtual sonar target images, and the region with a mask value of 0 corresponds to the background region;

[0014] combining the combined images with the binary mask pasting in the channel dimension to form the input of the virtual sonar image background reconstruction network.

[0015] In some embodiments, the virtual sonar image background reconstruction network comprises:

[0016] a background reconstruction encoder for performing layer-by-layer down-sampling on the input image after channel pasting and extracting multi-scale features covering background structure information and noise distribution information;

[0017] a background reconstruction decoder for recovering the spatial resolution and texture details of the image layer by layer based on the multi-scale features;

[0018] a multi-scale skip connection layer for establishing a connection between the corresponding feature layers of the background reconstruction encoder and the background reconstruction decoder, directly transmitting the multi-scale features to the corresponding layers of the background reconstruction decoder, and realizing the fusion of global background structure and local texture features.

[0019] In some embodiments, the background reconstruction encoder comprises a plurality of two-dimensional convolution layers connected in sequence, a batch normalization layer and an activation function layer, the plurality of two-dimensional convolution layers being used to extract spatial features and reduce feature map resolution, the batch normalization layer being used for normalization processing to stabilize training, and the activation function layer being used to introduce a nonlinear transformation; the background reconstruction decoder comprises a deconvolution layer connected with an activation function layer through a multi-scale skip connection layer, and is used for layer-by-layer up-sampling to restore the spatial resolution of the image.

[0020] In some embodiments, the total loss function of the virtual sonar image background reconstruction network is represented as:

[0021]

[0022] wherein, Ltotal is the total loss function of the virtual sonar image background reconstruction network, is a weight coefficient balancing the adversarial loss, is the adversarial loss function, is a weight coefficient balancing the spectral consistency loss, is the spectral consistency loss function, is a weight coefficient balancing the speckle noise statistics loss, is the speckle noise statistics loss function, is a weight coefficient balancing the total variation smoothing loss, is the total variation smoothing loss function, is a weight coefficient balancing the target reconstruction loss, is the target reconstruction loss function;

[0023] The adversarial loss function is represented as:

[0024]

[0025] wherein, E[·] is an expectation value operator, is a local region authenticity score map of the virtual sonar image with background ;

[0026] The spectral consistency loss function is represented as:

[0027]

[0028] wherein, is a radial spectrum energy curve of the background region, and is used to constrain the frequency distribution characteristics of the generated background; and represents a pixel-by-pixel multiplication operation; is a real sonar background image;

[0029] The speckle noise statistics loss function is represented as:

[0030]

[0031] The total variation smoothing loss function is represented as:

[0032]

[0033] In the formula, N is a virtual sonar image with background total number of pixels of the background; is a coordinate value of the right adjacent pixel; is a coordinate value of the upper adjacent pixel; is a coordinate current pixel value at

[0034] The target reconstruction loss function is represented as:

[0035]

[0036] In the formula, is a balanced difference loss L weight coefficient of 1, is a weight coefficient for balancing the learning of the perceptual image block similarity loss, is a weight coefficient for balancing the structural similarity index loss.

[0037] In some embodiments, the training method of the visual-text fusion network is:

[0038] Given a real sonar target image and a real optical target image, input them into the same structure generator with different text prompts as conditions respectively, to generate a virtual optical target image and a virtual sonar target image respectively;

[0039] Construct a discriminator to distinguish between virtual optical target images and real optical target images, construct a discriminator to distinguish between virtual sonar target images and real sonar target images;

[0040] Define the adversarial loss function so that the discriminator cannot distinguish between virtual optical target images and real optical target images, and the discriminator cannot distinguish between virtual sonar target images and real sonar target images;

[0041] Define the cycle consistency loss function so that the image maps back to its original appearance after continuous conversion;

[0042] An identity loss function is introduced to further preserve the image content that already belongs to the target domain.

[0043] After training, only the generator The generation of the virtual sonar target image is completed.

[0044] In some embodiments, the vision-text fusion network adopts a pre-trained Stable Diffusion Turbo model as the generator, the generator includes a diffusion model encoder, a U-Net, and a diffusion model decoder, and the vision-text fusion network is configured to:

[0045] Input the real optical target image into the noise encoding path of the U-Net;

[0046] Generate a text embedding through a CLIP-based text encoder, the text embedding is injected into the U-Net through a cross-attention mechanism;

[0047] Establish a skip connection between the diffusion model encoder and the diffusion model decoder through a zero convolution layer to preserve the detail information of the real optical target image in the virtual sonar target image.

[0048] In some embodiments, low-rank adaptation modules are embedded in the diffusion model encoder, the U-Net, and the diffusion model decoder, during the generator fine-tuning process, the original weights of the Stable Diffusion Turbo are kept frozen, only the parameters of the low-rank adaptation modules, the initial convolution layers of the U-Net, and the zero convolution layer are updated; at the same time, the text encoder is kept in a completely frozen state during the fine-tuning process.

[0049] The second aspect of the present application provides a virtual sonar image generation system for implementing the virtual sonar image generation method of the first aspect of the present application, comprising:

[0050] A virtual sonar target generation module for mapping a real optical target image into a virtual sonar target image;

[0051] A virtual sonar image background reconstruction module for embedding the virtual sonar target image into a real sonar background image to output a virtual sonar image with background.

[0052] In some embodiments, the virtual sonar target generation module includes an image conversion discriminator, the image conversion discriminator adopts a pre-trained CLIP model as a frozen backbone network to extract a high-dimensional feature representation from an input target image, the feature representation is first processed through two consecutive convolution downsampling blocks, then sent to a feedforward module, and finally mapped to a discriminator score through a fully connected layer.

[0053] Compared with the prior art, the application has the advantages and positive effects that:

[0054] (1) High-quality virtual sonar images can be generated using real optical images, breaking through the bottleneck of real sonar data collection limited by environment, cost and equipment conditions; at the same time, the target type and quantity can be flexibly expanded, effectively solving the problems of insufficient sonar sample categories and unbalanced distribution;

[0055] (2) The generated virtual sonar images automatically have accurate target position and category information, without tedious manual annotation, avoiding the problems of difficult determination of target boundary, long time consumption and large error in traditional sonar image annotation;

[0056] (3) Through visual-textual condition fusion and spectral, texture and noise statistical constraints, the virtual sonar target approximates the real imaging effect in terms of photoacoustic characteristics, energy attenuation law and noise distribution, enhancing the training contribution and generalization ability of virtual images to detection, classification and other tasks;

[0057] (4) A virtual sonar image background reconstruction network is used to dynamically match virtual targets according to scene brightness and scattering characteristics, so that the generated images are highly consistent with real sonar images in overall vision, and can be used to simulate underwater operation scenes under different sea conditions and different background conditions;

[0058] (5) Multi-source sonar data for algorithm development, performance testing, data enhancement and simulation deduction can be quickly generated, providing stable information support for underwater security, marine exploration, equipment detection and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The flowchart of the virtual sonar image generation method in the embodiments of the application.

[0060] Figure 2 The generator network structure diagram in the embodiments of the application.

[0061] Figure 3 The image conversion discriminator structure diagram in the embodiments of the application.

[0062] Figure 4 The visual-textual fusion network training diagram in the embodiments of the application.

[0063] Figure 5 The virtual sonar image background reconstruction network training diagram in the embodiments of the application.

[0064] Figure 6 The result diagram of generating virtual sonar target images in the embodiments of the application.

[0065] Figure 7 The result diagram of generating virtual sonar images in the embodiments of the application. Detailed Implementation

[0066] The present application will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0067] like Figure 1 As shown, in one embodiment of the virtual sonar image generation method and system of this application, the virtual sonar image generation method includes: first, using a visual-text fusion network to map a real optical target image into a virtual sonar target image; second, embedding the virtual sonar target image into a real sonar background image to generate a combined image; and using a virtual sonar image background reconstruction network to perform background compensation and texture consistency adjustment on the combined image to obtain a virtual sonar image with a background.

[0068] In the above illustrative embodiments, the virtual sonar image generation method can directly output high-quality virtual sonar images with backgrounds, significantly enriching the number and categories of samples, reducing annotation costs, and providing reliable data support for subsequent target detection, classification, and simulation testing.

[0069] This application also provides a virtual sonar image generation system, which is used to implement the above-mentioned virtual sonar image generation method, including a virtual sonar target generation module and a virtual sonar image background reconstruction module. The virtual sonar target generation module is used to map a real optical target image into a virtual sonar target image; the virtual sonar image background reconstruction module is used to embed the virtual sonar target image into a real sonar background image and output a virtual sonar image with background.

[0070] Specifically, the virtual sonar target generation module leverages the single-step reasoning capability and strong prior knowledge of the pre-trained Stable Diffusion Turbo model to achieve efficient visual-text feature fusion and robust zero-shot generalization. By employing a non-paired training strategy—requiring only textual prompts and arbitrary real optical target images as references—this module can generate virtual sonar target images without relying on paired training data.

[0071] Specifically, the visual-text fusion network uses a pre-trained Stable Diffusion Turbo model as the generator. Unlike traditional diffusion models that typically require hundreds of iterations for denoising, Stable Diffusion Turbo supports single-step inference, significantly improving processing efficiency. Furthermore, leveraging the strong prior knowledge introduced by the pre-trained model, the generator possesses robust zero-shot generalization capabilities. Therefore, high-quality performance can be achieved with only minor fine-tuning.

[0072] Specifically, such asFigure 2 As shown, the generator includes a diffusion model encoder, a U-Net, and a diffusion model decoder, and the generator integrates the diffusion model encoder, the U-Net, and the diffusion model decoder in the pre-trained Stable Diffusion Turbo model into a unified end-to-end framework. A text encoder based on a contrastive language-image pre-training model (CLIP) is also used to generate a text embedding for conditional conditioning of the U-Net, and the text encoder remains completely frozen during the fine-tuning process. To input the image and text conditions without conflict, the real optical target image is input into the noise encoding path of the U-Net; and the CLIP text embedding is injected into each U-Net module through a cross-attention mechanism, so that the model can jointly focus on visual and language clues, effectively alleviating the structural inconsistency problem between the input content and the generated noise map. To preserve fine-grained image details, a 1x1 convolution layer (i.e., a zero convolution layer) with zero initialization is established between the diffusion model encoder and the diffusion model decoder. These skip paths enable the key spatial features of the real optical target image to be preserved in the output, thereby effectively preserving the detailed information of the real optical target image.

[0073] Specifically, to achieve efficient adaptation with minimal computational overhead, a low-rank adaptation (LoRA) module is embedded in the diffusion model encoder, the U-Net, and the diffusion model decoder. During the generator fine-tuning process, all original weights of the Stable Diffusion Turbo remain frozen, and only the low-rank adaptation (LoRA) module parameters, the initial convolution layer of the U-Net, and the zero convolution layer are updated. This selective training strategy effectively preserves the prior knowledge embedded in the pre-trained model while only requiring a small number of trainable parameters.

[0074] Specifically, the virtual sonar target generation module includes an image conversion discriminator. For the image conversion discriminator, as shown in Figure 3 Inspired by the design of a vision-aided generative adversarial network (Vision-Aided GAN), a pre-trained CLIP model is used as a frozen backbone network to extract a high-dimensional feature representation from each input target image I t The feature representation is then processed by two consecutive convolution downsampling blocks: each downsampling block includes a small-scale spatial convolution layer (with a 3x3 convolution kernel) for channel projection, a LeakyReLU activation function layer, a blur pooling layer for anti-aliasing downsampling, and a point-wise convolution layer (1x1 convolution with a stride of 2) for further downsampling. The downsampled features are fed into a feedforward module consisting of a fully connected layer and a subsequent LeakyReLU activation function layer. Finally, the fully connected layer maps a 256-dimensional vector to a discriminator score D ( It ).

[0075] Specifically, such as Figure 4 As shown, the training and enhancement process of the visual-text fusion network includes: given a real sonar target image and real optical target images Input them into the generator respectively and Generator and Using the same network structure, but with different text prompts. and As a condition. Generator and Generating virtual optical target images respectively and virtual sonar target images Specifically, it is expressed as: , ,in, and These respectively represent the text prompts "This is an optical image" and "This is a sonar image".

[0076] An adversarial loss function is employed to ensure that the discriminator cannot distinguish between virtual optical target images and real optical target images, or between virtual sonar target images and real sonar target images. Two discriminators are constructed. and : Among them, the discriminator Used to distinguish virtual optical target images and real optical target images Among them, the discriminator Used to distinguish virtual sonar target images and real sonar target images To enable the discriminator Unable to distinguish virtual optical target images and real optical target images And the discriminator Unable to distinguish virtual sonar target images and real sonar target images Adversarial loss function The definition is as follows: , where E[·] is the expectation value operator.

[0077] In addition, a cycle consistency loss function is used to ensure that the image can be mapped back to its original appearance after successive transformations. This function enforces consistency between the forward and reverse mappings, thereby improving the stability of the training process and the reliability of the generated results. Cycle consistency loss function is defined as follows:

[0078]

[0079] wherein, denotes the reconstruction loss function, which consists of three parts, including L 1 the difference, the learning perceptual image block similarity (LPIPS), and the structural similarity index (SSIM). L 1 The loss emphasizes point-by-point accuracy at the pixel level, which helps to preserve overall fine details; the LPIPS loss targets high-level semantics and perceptual quality, effectively addressing the problem of similar pixel values but different visual perception; the SSIM loss constrains local structure and texture consistency, ensuring that the generated image maintains coherent structure in local regions. These components collectively form balanced constraints on image quality.

[0080] The reconstruction loss function is defined as follows:

[0081]

[0082] wherein, X is the processed image, Y is the label image, denotes X the comprehensive reconstruction error between Y in three dimensions of pixel accuracy, perceptual quality, and structural consistency; denotes X the absolute value sum of the difference between Y corresponding pixel values; denotes X the difference between Y in high-level semantics and visual perception; denotes X the similarity between Y in local structure and texture; is the weight coefficient for balancing the difference loss L 1, is the weight coefficient for balancing the learning perceptual image block similarity loss LPIPS, is the weight coefficient for balancing the structural similarity index loss SSIM. To further preserve the content of images that already belong to the target domain, an identity loss function is introduced. When the input image comes from the target domain, this function encourages the generator to output results that are highly matched with the input, helping to avoid unnecessary modifications. Identity loss function is defined as follows: .

[0083] The complete training loss function of the generator is defined as follows: , wherein is the total loss function of the generator, is the weight coefficient of balancing the adversarial loss function, is the weight coefficient of balancing the cycle consistency loss function, is the weight coefficient of balancing the identity loss function.

[0084] For the discriminator and , the corresponding loss functions are designed to distinguish real optical target images from virtual optical target images, and real sonar target images from virtual sonar target images. The total loss functions of the discriminators and are defined as follows:

[0085] .

[0086] The generators and and the discriminators and adopt an adversarial training strategy, so that the generators and gradually enhance their ability to generate high-quality images that meet the domain consistency in continuous optimization. After training, only the generator is needed to complete the virtual sonar target image generation task. Specifically, given a real optical target image and a text prompt , the generator outputs a virtual sonar target image .

[0087] The virtual sonar image background reconstruction module is proposed to address the problem that the generated virtual sonar target image only contains the target and lacks a real background. This module is used to automatically supplement the background noise that is consistent with the distribution of the real sonar background image for the virtual sonar target image containing only the target region, achieving the natural fusion of the target and the background. Under the premise of not destroying the target structure, this module uses an unsupervised training strategy to make the texture features, spectral characteristics, and speckle noise distribution of the generated virtual sonar image with background consistent with those of the real sonar image.

[0088] As shown in Figure 5 , the training method of the virtual sonar image background reconstruction module is as follows: first, the virtual sonar target image generated by the virtual sonar target generation module is input. In order to enhance diversity and scene adaptability, the virtual sonar target image randomly scaling, scaling scale s , a scale factor is: , wherein, represents the original height of the virtual sonar target image, represents the original width of the virtual sonar target image; according to the scale factor, the scaled virtual sonar target image is obtained after using bilinear interpolation . In the real sonar background image , a random insertion position is selected, and the scaled virtual sonar target image is embedded into the real sonar background image to form a combined image . A binary mask is established for the combined image M , the area with a mask value of 1 represents the position of the scaled virtual sonar target image , and the area with a mask value of 0 corresponds to the background area. The combined image and the mask M are spliced in the channel dimension to constitute the input of the virtual sonar image background reconstruction network H : , wherein, is the input tensor of the virtual sonar image background reconstruction network H.

[0089] The virtual sonar image background reconstruction network H is used to compensate the background and adjust the brightness and texture of the combined image , and outputs a virtual sonar image with background , which is represented as: .

[0090] The virtual sonar image background reconstruction network HA generative structure based on convolutional neural networks is adopted, including a background reconstruction encoder, a background reconstruction decoder, and a multi-scale skip connection layer. The background reconstruction encoder consists of multiple layers of two-dimensional convolutional layers, batch normalization layers, and activation function layers connected in sequence. It is used to downsample the input image after channel stitching layer by layer and extract multi-scale features covering background structure information and noise distribution information. The background reconstruction decoder consists of deconvolutional layers and is used to restore the spatial resolution and texture details of the image layer by layer based on multi-scale features. The multi-scale skip connection layer is used to establish connections between the corresponding feature layers of the background reconstruction encoder and the background reconstruction decoder, directly passing multi-scale features to the corresponding layers of the background reconstruction decoder, realizing the fusion of global background structure and local texture features. Among them, the multiple layers of two-dimensional convolutional layers are used to extract spatial features and reduce the resolution of feature maps; the batch normalization layer is used for normalization processing to stabilize training; the activation function layer adopts the LeakyReLU function to introduce nonlinear transformation; the deconvolutional layer and the activation function layer are connected by the multi-scale skip connection layer for layer-by-layer upsampling to restore the spatial resolution of the image. The network input has 4 channels (a three-channel combined image plus a one-channel mask), and the output is a three-channel image.

[0091] Specifically, to improve the realism and stability of the background, a virtual sonar image background reconstruction network is used. H With discriminator This forms the adversarial training framework. Discriminator A convolutional neural network structure is used, with the input being a virtual sonar image with a background. Or real sonar background image The output is a virtual sonar image with a background. Or real sonar background image Local area authenticity scoring map or Each convolutional block contains a convolutional layer, a spectral normalization layer, and a LeakyReLU activation function layer, used to stabilize adversarial training and capture differences in local noise distribution. Discriminator Instead of using global pooling, it outputs a two-dimensional feature map to reflect the local discrimination results.

[0092] Specifically, during the training phase, the virtual sonar image background reconstruction network is constrained by multiple unsupervised loss methods. H The generation effect. Losses include:

[0093] (1) Adversarial loss: In order to make the discriminator Unable to distinguish virtual sonar images with background and real sonar background images Adversarial loss function The definition is as follows: .

[0094] (2) Spectral consistency loss: Comparison of virtual sonar images with background Background Compared with real sonar background images Background radial power spectral density , is represented as:

[0095]

[0096] in The radial spectral energy curve of the background region constrains the frequency distribution characteristics of the generated background; ⊙ represents pixel-by-pixel multiplication.

[0097] (3) Speckle noise statistical loss: Virtual sonar image with background Background Compared with real sonar background images Background Mean squared error on the Rayleigh distribution scaling parameter , can be represented as:

[0098] .

[0099] (4) Total variational smoothing loss Virtual sonar images with background Background Applying total variational regularization to suppress artifacts and preserve smoothness is expressed as:

[0100]

[0101] in, N for The total number of pixels; coordinates The value of the pixel adjacent to the right; coordinates The value of the adjacent pixel above; coordinates The current pixel value at that location.

[0102] (5) Target reconstruction loss, used to ensure that the shape and details of the original virtual sonar target image remain unchanged during background generation. It consists of three parts, including Difference, learning-aware image patch similarity index (LPIPS) and structural similarity index (SSIM). The loss function emphasizes pixel-level accuracy, helping to preserve overall fine details; LPIPS loss targets high-level semantics and perceptual quality, effectively addressing the problem of similar pixel values ​​but different visual perceptions; SSIM loss constrains local structure and texture consistency, ensuring that the generated image maintains a coherent structure in local regions. These components collectively provide a balanced constraint on image quality. (Target Reconstruction Loss Function) Defined as:

[0103]

[0104] In the formula, To balance the difference loss L A weighting factor of 1 To balance the weight coefficients of the learning-aware image patch similarity loss LPIPS, The weighting coefficients of the structural similarity index loss SSIM are used to balance the structural similarity index loss.

[0105] Virtual Sonar Image Background Reconstruction Network H The complete training loss function is defined as follows:

[0106]

[0107] In the formula, Network for reconstructing background of virtual sonar images H The total loss function, To balance the weighting coefficients of the adversarial loss function, To balance the weighting coefficients of the spectral consistency loss, To balance the weighting coefficients of the statistical loss of speckle noise, To balance the weighting coefficients of the total variation smoothing loss, Weighting coefficients for balancing the reconstruction loss of the target.

[0108] Specifically, for the discriminator Design a corresponding loss function to distinguish virtual sonar images with background. and real sonar background images Discriminator loss function The definition is as follows: .

[0109] Specifically, a network for reconstructing background data using virtual sonar images is employed. H With discriminator An adversarial training strategy is used to improve the fusion quality of the target and background. The training process is as follows: First, the virtual sonar target image generated by the virtual sonar target generation module is used... The composite image is constructed by pasting the images onto a real background image at random scales and positions. With corresponding maskM ; and then Background reconstruction network fed into virtual sonar image H To generate virtual sonar images with background and virtual sonar images with background Compared with real sonar background images Input discriminator respectively In each training step, the parameters of the reconstructed network are first fixed, and the discriminator is optimized. To enhance its discriminative ability, the discriminator is then fixed. The parameters are optimized to simultaneously minimize adversarial loss, spectral consistency loss, blob statistics loss, total variational smoothing term, and target reconstruction loss. Through alternating iterations, adversarial training enables the reconstruction network to gradually learn to generate noise and texture consistent with the statistical characteristics of real sonar images while preserving target details. After training convergence, the background reconstruction network can directly and naturally fuse virtual sonar target images with real sonar background images, and the output synthetic image closely resembles the real sample in terms of visual appearance, spectral density, blob statistics, and other aspects.

[0110] To verify the effectiveness and applicability of the proposed virtual sonar image generation method and system, a complete experimental dataset was constructed based on real underwater sonar imaging data and optical image data. Training and evaluation were carried out under a unified data preprocessing and strict data partitioning strategy. To ensure the representativeness and reliability of the experimental results, the data used not only covers multiple types of typical underwater targets but also includes large-scale real underwater sonar background images, which can fully reflect the generalization ability and target generation quality in complex underwater acoustic environments.

[0111] The real sonar images used for experimental verification come from an underwater acoustic target detection dataset, which contains 7600 real sonar images covering 10 categories of targets, including cube, ball, cylinder, human body, tyre, circle cage, square cage, metal bucket, plane and rov. According to the target detection annotation file provided in the dataset, the target regions in the real sonar images are cropped to obtain 12296 real sonar target images of 8 categories. For ease of training, all real sonar target images are uniformly adjusted to 256x256 pixels. In addition, real optical images of the same 10 categories of targets are collected, 16 for each category, and the real optical images are cropped to only retain the target region, finally obtaining 160 real optical target images. In the experiment, 6 categories of ball, cylinder, human body, tyre, square cage and metal bucket are selected from the real sonar target images and real optical target images as the training set, 2 categories of plane and rov are used as the verification set, and 2 categories of cube and circle cage are used as the test set for training and evaluation of the virtual sonar target generation module. At the same time, the 7600 real sonar images in the underwater acoustic target detection dataset are used for real sonar background image generation, 80% of the real sonar background images are selected to embed the generated virtual sonar target images, which are used for training of the virtual sonar image background reconstruction module; the remaining 10% and 10% are used for verification and testing. To ensure consistency, all real underwater sonar background images are standardized to 1024x1024 pixels.

[0112] Figure 6 The virtual sonar target generation module based on real optical target images generates virtual sonar target images that are consistent with real sonar target images in texture structure, brightness distribution and scattering characteristics. As shown in Figure 6 It can be seen that the virtual sonar target generation module proposed in the present application can generate virtual sonar target images that are highly consistent with real sonar target images in texture, brightness distribution and scattering characteristics based on real optical target images. The generation result maintains good consistency with the real sonar target image in target outline, energy attenuation characteristics and noise distribution, etc.

[0113] In addition, Figure 7 The fusion effect of the virtual sonar target image and the real sonar background image is shown. As shown in Figure 7It can be seen that the virtual sonar image background reconstruction module can realize natural fusion of the virtual sonar target image and the real sonar background image. The module dynamically adjusts the brightness contrast and scattering consistency between the virtual sonar target image and the real sonar background image while retaining the background texture and noise characteristics, so that the generated virtual sonar image with background is almost indistinguishable from the real sonar image in overall vision.

[0114] The above embodiments are used to explain the present application, but not to limit the present application, and any modifications and changes made to the present application within the spirit of the present application and the protection scope of the claims fall within the protection scope of the present application.

Claims

1. A method for generating virtual sonar images, characterized in that, The method includes: A visual-text fusion network is used to map real optical target images into virtual sonar target images; A composite image is generated by embedding a virtual sonar target image into a real sonar background image. A virtual sonar image background reconstruction network is used to perform background compensation and texture consistency adjustment on the combined image to obtain a virtual sonar image with a background. The step of embedding a virtual sonar target image into a real sonar background image to generate a combined image includes the following steps: The virtual sonar target image is scaled, and bilinear interpolation is used to obtain the scaled virtual sonar target image. Select an insertion point within a real sonar background image, and embed a scaled virtual sonar target image into that position to generate a composite image; Construct a binary mask for the combined image The area with a mask value of 1 corresponds to the area where the scaled virtual sonar target image is located, and the area with a mask value of 0 corresponds to the background area. Combine the image with a binary mask The images are stitched together along the channel dimension to form the input to the virtual sonar image background reconstruction network.

2. The virtual sonar image generation method according to claim 1, characterized in that, The virtual sonar image background reconstruction network includes: The background reconstruction encoder is used to downsample the input image after channel stitching layer by layer and extract multi-scale features that cover background structure information and noise distribution information. A background reconstruction decoder is used to recover the spatial resolution and texture details of an image layer by layer based on multi-scale features; The multi-scale skip connection layer is used to establish connections between the corresponding feature layers of the background reconstruction encoder and the background reconstruction decoder, and directly transmits multi-scale features to the corresponding layer of the background reconstruction decoder to achieve the fusion of global background structure and local texture features.

3. The virtual sonar image generation method according to claim 2, characterized in that, The background reconstruction encoder includes a series of two-dimensional convolutional layers, a batch normalization layer, and an activation function layer connected in sequence. The two-dimensional convolutional layers are used to extract spatial features and reduce the resolution of the feature map. The batch normalization layer is used for normalization processing to stabilize training. The activation function layer is used to introduce nonlinear transformation. The background reconstruction decoder includes a deconvolution layer, which is connected to the activation function layer through a multi-scale skip connection layer for upsampling layer by layer to restore the spatial resolution of the image.

4. The virtual sonar image generation method according to claim 2, characterized in that, The total loss function of the virtual sonar image background reconstruction network is expressed as follows: In the formula, Let $\frac{ ... To balance the weighting coefficients of the adversarial loss, To counteract the loss function, To balance the weighting coefficients of the spectral consistency loss, Let spectral consistency loss function be used. To balance the weighting coefficients of the statistical loss of speckle noise, Let be the statistical loss function for speckle noise. To balance the weighting coefficients of the total variation smoothing loss, The total variational smoothing loss function is... To balance the weighting coefficients of the target reconstruction loss, Reconstruct the loss function for the objective; The adversarial loss function is expressed as: In the formula, E[·] is the expected value operator. Virtual sonar images with background A local area authenticity scoring map; The spectrum consistency loss function is expressed as: In the formula, The radial spectral energy curve of the background region constrains the frequency distribution characteristics of the generated background; ⊙ represents pixel-by-pixel multiplication operation. This is a real sonar background image; The statistical loss function for the speckle noise is expressed as follows: ； The total variational smoothing loss function is expressed as: In the formula, N Virtual sonar images with background Total number of pixels in the background; coordinates The value of the adjacent pixel to the right; coordinates The value of the adjacent pixel above; coordinates The current pixel value at that location; The target reconstruction loss function is expressed as: In the formula, To balance the difference loss L A weighting factor of 1 To balance the weight coefficients of the learning-aware image patch similarity loss, The weighting coefficients are used to balance the loss of the structural similarity index.

5. The virtual sonar image generation method according to claim 1, characterized in that, The training method for the vision-text fusion network is as follows: Given real sonar target images and real optical target images, each with different text prompts, input them into a generator with the same structure. With generator In the process, virtual optical target images and virtual sonar target images are generated respectively; Construct a discriminator To distinguish between virtual optical target images and real optical target images, a discriminator is constructed. Used to distinguish between virtual sonar target images and real sonar target images; Define an adversarial loss function so that the discriminator It cannot distinguish between virtual optical target images and real optical target images, and the discriminator... Unable to distinguish between virtual sonar target images and real sonar target images; Define a cycle consistency loss function so that an image is mapped back to its original appearance after successive transformations; An identity loss function is introduced to further preserve image content that already belongs to the target domain; After training, use only the generator. This allows for the generation of virtual sonar target images.

6. The virtual sonar image generation method according to claim 4, characterized in that, The visual-text fusion network uses a pre-trained Stable Diffusion Turbo model as the generator, which includes a diffusion model encoder, a U-Net, and a diffusion model decoder. The visual-text fusion network is configured as follows: Input the real optical target image into the noise coding path of U-Net; Text embeddings are generated using a CLIP-based text encoder and injected into U-Net via a cross-attention mechanism; A skip connection is established between the diffusion model encoder and the diffusion model decoder through a zero-convolutional layer to preserve the detailed information of the real optical target image in the virtual sonar target image.

7. The virtual sonar image generation method according to claim 6, characterized in that, Low-rank adaptation modules are embedded in the diffusion model encoder, U-Net, and diffusion model decoder. During generator fine-tuning, the original weights of Stable DiffusionTurbo are kept frozen, and only the parameters of the low-rank adaptation modules, the initial convolutional layer of U-Net, and the zero convolutional layer are updated. At the same time, the text encoder remains completely frozen during fine-tuning.

8. A virtual sonar image generation system, used to implement the virtual sonar image generation method as described in any one of claims 1-7, characterized in that, The system includes: The virtual sonar target generation module is used to map real optical target images into virtual sonar target images; The virtual sonar image background reconstruction module is used to embed the virtual sonar target image into the real sonar background image and output a virtual sonar image with background.

9. The virtual sonar image generation system according to claim 8, characterized in that, The virtual sonar target generation module includes an image conversion discriminator, which uses a pre-trained CLIP model as a frozen backbone network to extract high-dimensional feature representations from the input target image. The feature representations are first processed through two consecutive convolutional downsampling blocks, then fed into a feedforward module, and finally mapped to discriminator scores through a fully connected layer.

Images