Virtual sonar image generation method and system

Abstract

The invention belongs to the technical field of underwater sonar imaging and intelligent identification, and relates to a virtual sonar image generation method and system. The method comprises the following steps: mapping a real optical target image into a virtual sonar target image by adopting a vision-text fusion network; embedding the virtual sonar target image into the real sonar background image to generate a combined image; and background compensation and texture consistency adjustment are carried out on the combined image by using a virtual sonar image background reconstruction network to obtain a virtual sonar image with a background. According to the invention, the bottleneck that real sonar data acquisition is limited by environment, cost and equipment conditions is broken through; meanwhile, the target type and number can be flexibly expanded, and the problems that sonar samples are insufficient in category and unbalanced in distribution are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of underwater sonar imaging and intelligent recognition, and particularly relates to a virtual sonar image generation method and system. BACKGROUND

[0002] As an important means of underwater detection and recognition, sonar imaging technology is widely used in underwater equipment detection, target search, ocean resource investigation, security monitoring and other fields. At present, intelligent recognition methods represented by deep learning have become the main direction to improve the accuracy of sonar image target analysis, but model training is heavily dependent on large-scale labeled samples. The actual underwater sonar data acquisition is limited by water conditions, equipment types, operation costs and sea conditions, and it is difficult, costly and limited in scene coverage. Moreover, the target categories are generally insufficient and unevenly distributed. On the other hand, the manual labeling of sonar targets is time-consuming, especially in low signal-to-noise ratio scenarios where target boundaries are blurred, making it difficult to ensure the stability and consistency of the labeling results, making high-precision labeled samples even more scarce.

[0003] In order to make up for the lack of data, many studies have tried to use data augmentation, transfer learning, simulated sonar imaging models and other methods to expand the sample size. However, traditional simulation-generated images often differ significantly from real sonar images in terms of texture details, scattering patterns and noise statistical characteristics, making it difficult to support model training in terms of image authenticity and diversity. Existing augmentation methods also mainly focus on geometric or lighting changes, and cannot solve the problem of target category scarcity and high labeling cost. The construction of complex virtual samples in natural scenes with multiple backgrounds, multiple poses and multiple scales is still insufficient.

[0004] In summary, how to efficiently generate high-quality virtual sonar images that are highly consistent with real image characteristics under the condition of lacking a large amount of real sonar data, to significantly increase the sample size, enrich the target categories and reduce the labeling cost, has become a technical problem that needs to be solved. SUMMARY

[0005] The present application provides a virtual sonar image generation method and system to address the problems of sonar image data scarcity, high acquisition cost and labeling difficulty. By constructing a virtual sonar target generation module, real optical target images are effectively converted into virtual sonar target images. And using a virtual sonar image background reconstruction module, the virtual sonar target image is naturally fused with the real sonar background image to obtain high-quality virtual sonar images with background. The present application uses adversarial training and unsupervised constraints to improve the authenticity and diversity of the generated results, which can significantly expand the number and categories of sonar samples, reduce the labeling cost, and provide reliable data support for underwater target detection and recognition tasks.

[0006] To achieve the above purpose, the first aspect of the present application provides a virtual sonar image generation method, comprising: mapping real optical target images to virtual sonar target images using a vision-text fusion network; embedding the virtual sonar target image into a real sonar background image to generate a combined image; reconstructing the background of the virtual sonar image using a virtual sonar image background reconstruction network to obtain a virtual sonar image with background; The embedding of the virtual sonar target image into the real sonar background image to generate the combined image comprises the following steps: performing a scaling operation on the virtual sonar target image and obtaining a scaled virtual sonar target image using bilinear interpolation; selecting an insertion position in the real sonar background image and embedding the scaled virtual sonar target image into the position to generate a combined image; constructing a binary mask for the combined image The region with a mask value of 1 corresponds to the region of the scaled virtual sonar target image, and the region with a mask value of 0 corresponds to the background region. combining the combined image and the binary mask pasting in the channel dimension to form the input of the virtual sonar image background reconstruction network.

[0007] In some embodiments, the virtual sonar image background reconstruction network comprises: a background reconstruction encoder for performing layer-by-layer down-sampling on the input image after channel pasting and extracting multi-scale features covering background structure information and noise distribution information; a background reconstruction decoder for recovering the spatial resolution and texture details of the image layer by layer based on the multi-scale features; a multi-scale skip connection layer for establishing a connection between the corresponding feature layers of the background reconstruction encoder and the background reconstruction decoder, directly transmitting the multi-scale features to the corresponding layers of the background reconstruction decoder to realize the fusion of global background structure and local texture features.

[0008] In some embodiments, the background reconstruction encoder comprises a plurality of two-dimensional convolution layers, batch normalization layers and activation function layers connected in sequence, the plurality of two-dimensional convolution layers are used to extract spatial features and reduce the resolution of feature maps, the batch normalization layers are used for normalization processing to stabilize training, and the activation function layers are used to introduce nonlinear transformation; the background reconstruction decoder comprises a deconvolution layer, and the deconvolution layer and the activation function layer are connected through a multi-scale skip connection layer for layer-by-layer up-sampling to recover the spatial resolution of the image.

[0009] In some embodiments, the total loss function of the virtual sonar image background reconstruction network is represented as:

[0010] wherein, is the total loss function of the virtual sonar image background reconstruction network, is the weight coefficient of the balance adversarial loss, is the adversarial loss function, is the weight coefficient of the balance spectral consistency loss, is the spectral consistency loss function, is the weight coefficient of the balance speckle noise statistics loss, is the speckle noise statistics loss function, is the weight coefficient of the balance total variation smoothing loss, is the total variation smoothing loss function, is the weight coefficient of the balance target reconstruction loss, is the target reconstruction loss function; The adversarial loss function is represented as:

[0011] wherein, E[·] is the expectation operator, is the local area authenticity score map of the virtual sonar image with background ; The spectral consistency loss function is represented as:

[0012] wherein, is the radial spectrum energy curve of the background area, which constrains the frequency distribution characteristics of the generated background; ⊙ represents the pixel-by-pixel multiplication operation; is the real sonar background image; The speckle noise statistics loss function is represented as:

[0013] The total variation smoothing loss function is represented as:

[0014] wherein, N is the total number of pixels of the background of the virtual sonar image with background ; is the value of the coordinate right adjacent pixel; is the value of the coordinate upper adjacent pixel; is the current pixel value at the coordinate ; The target reconstruction loss function is represented as:

[0015] wherein, a weight coefficient for balancing the difference loss L 1, a weight coefficient for balancing the difference loss a weight coefficient for balancing the learning perceptual image block similarity loss a weight coefficient for balancing the structural similarity index loss.

[0016] In some embodiments, the training method of the visual-text fusion network is: Given a real sonar target image and a real optical target image, input them into the same structure generator with different text prompts as conditions and the generator In it, respectively generate virtual optical target images and virtual sonar target images; Construct a discriminator for distinguishing virtual optical target images from real optical target images, construct a discriminator for distinguishing virtual sonar target images from real sonar target images; Define the adversarial loss function so that the discriminator cannot distinguish between virtual optical target images and real optical target images, and the discriminator cannot distinguish between virtual sonar target images and real sonar target images; Define the cycle consistency loss function so that the image maps back to its original appearance after continuous conversion; Introduce the identity loss function to further preserve the image content that already belongs to the target domain; After training, only the generator can complete the generation of virtual sonar target images.

[0017] In some embodiments, the visual-text fusion network uses a pre-trained Stable Diffusion Turbo model as the generator, which includes a diffusion model encoder, a U-Net, and a diffusion model decoder, and the visual-text fusion network is configured to: Input the real optical target image into the noise encoding path of the U-Net; Generate text embeddings through a CLIP-based text encoder, which are injected into the U-Net through cross-attention mechanisms; Establish a skip connection between the diffusion model encoder and the diffusion model decoder through a zero convolution layer to preserve the detail information of the real optical target image in the virtual sonar target image.

[0018] In some embodiments, a low-rank adaptation module is embedded in the diffusion model encoder, U-Net, and diffusion model decoder. During generator fine-tuning, the original weights of Stable Diffusion Turbo are kept frozen, and only the parameters of the low-rank adaptation module, the initial convolutional layer of U-Net, and the zero convolutional layer are updated. Meanwhile, the text encoder remains completely frozen during fine-tuning.

[0019] A second aspect of this application provides a virtual sonar image generation system for implementing the virtual sonar image generation method described in the first aspect of this application, comprising: The virtual sonar target generation module is used to map real optical target images into virtual sonar target images; The virtual sonar image background reconstruction module is used to embed the virtual sonar target image into the real sonar background image and output the virtual sonar image with the background.

[0020] In some embodiments, the virtual sonar target generation module includes an image conversion discriminator that uses a pre-trained CLIP model as a frozen backbone network to extract high-dimensional feature representations from the input target image. The feature representations are first processed through two consecutive convolutional downsampling blocks, then fed into a feedforward module, and finally mapped to discriminator scores through a fully connected layer.

[0021] Compared with the prior art, the advantages and positive effects of this application are as follows: (1) High-quality virtual sonar images can be generated using real optical images, breaking through the bottleneck of real sonar data acquisition being limited by environment, cost and equipment conditions; at the same time, the types and quantities of targets can be flexibly expanded, effectively solving the problems of insufficient sonar sample categories and unbalanced distribution. (2) The generated virtual sonar image automatically has accurate target location and category information, without the need for tedious manual annotation, avoiding the problems of difficulty in determining target boundaries, long time consumption and large error in traditional sonar image annotation; (3) By using visual-text conditional fusion and statistical constraints on spectrum, texture and noise, the virtual sonar target can approximate the real imaging effect in terms of photoacoustic characteristics, energy attenuation law and noise distribution, thereby enhancing the training contribution and generalization ability of virtual images to tasks such as detection and classification. (4) A virtual sonar image background reconstruction network is used to dynamically match virtual targets according to scene brightness and scattering characteristics, so that the generated image is highly consistent with the real sonar image in overall vision, and can be used to simulate underwater operation scenarios with different sea conditions and background conditions. (5) It can quickly generate multi-source sonar data for algorithm development, performance testing, data augmentation and simulation, providing stable information support for underwater security, marine exploration, equipment testing and other fields. Attached Figure Description

[0022] Figure 1 This is a flowchart of the virtual sonar image generation method in the embodiments of this application.

[0023] Figure 2 This is a schematic diagram of the generator network structure in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram of the image conversion discriminator structure in an embodiment of this application.

[0025] Figure 4 This is a schematic diagram of the training of the visual-text fusion network in an embodiment of this application.

[0026] Figure 5 This is a schematic diagram of the training of the virtual sonar image background reconstruction network in an embodiment of this application.

[0027] Figure 6 This is a schematic diagram of the result of generating a virtual sonar target image in an embodiment of this application.

[0028] Figure 7 This is a schematic diagram of the result of generating a virtual sonar image in an embodiment of this application. Detailed Implementation

[0029] The present application will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0030] like Figure 1 As shown, in one embodiment of the virtual sonar image generation method and system of this application, the virtual sonar image generation method includes: first, using a visual-text fusion network to map a real optical target image into a virtual sonar target image; second, embedding the virtual sonar target image into a real sonar background image to generate a combined image; and using a virtual sonar image background reconstruction network to perform background compensation and texture consistency adjustment on the combined image to obtain a virtual sonar image with a background.

[0031] In the above illustrative embodiments, the virtual sonar image generation method can directly output high-quality virtual sonar images with backgrounds, significantly enriching the number and categories of samples, reducing annotation costs, and providing reliable data support for subsequent target detection, classification, and simulation testing.

[0032] This application also provides a virtual sonar image generation system, which is used to implement the above-mentioned virtual sonar image generation method, including a virtual sonar target generation module and a virtual sonar image background reconstruction module. The virtual sonar target generation module is used to map a real optical target image into a virtual sonar target image; the virtual sonar image background reconstruction module is used to embed the virtual sonar target image into a real sonar background image and output a virtual sonar image with background.

[0033] Specifically, the virtual sonar target generation module leverages the single-step reasoning capability and strong prior knowledge of the pre-trained Stable Diffusion Turbo model to achieve efficient visual-text feature fusion and robust zero-shot generalization. By employing a non-paired training strategy—requiring only textual prompts and arbitrary real optical target images as references—this module can generate virtual sonar target images without relying on paired training data.

[0034] Specifically, the visual-text fusion network uses a pre-trained Stable Diffusion Turbo model as the generator. Unlike traditional diffusion models that typically require hundreds of iterations for denoising, Stable Diffusion Turbo supports single-step inference, significantly improving processing efficiency. Furthermore, leveraging the strong prior knowledge introduced by the pre-trained model, the generator possesses robust zero-shot generalization capabilities. Therefore, high-quality performance can be achieved with only minor fine-tuning.

[0035] Specifically, such as Figure 2 As shown, the generator comprises a diffusion model encoder, U-Net, and a diffusion model decoder. The generator integrates the diffusion model encoder, U-Net, and diffusion model decoder from the pre-trained Stable Diffusion Turbo model into a unified end-to-end framework. A text encoder based on a contrastive language-image pre-trained model (CLIP) is also employed to generate text embeddings for U-Net conditional tuning, and this text encoder remains completely frozen during fine-tuning. To ensure conflict-free input image and text conditions, real optical target images are input into the noise encoding path of U-Net; while CLIP text embeddings are injected into each U-Net module through a cross-attention mechanism, enabling the model to jointly attend to visual and linguistic cues, effectively mitigating the structural inconsistency between the input content and the generated noise map. To preserve fine-grained image details, skip connections are established between the diffusion model encoder and the diffusion model decoder through zero-initialized 1×1 convolutional layers (i.e., zero convolutional layers). These skip paths allow key spatial features of the real optical target images to be preserved in the output, thus effectively retaining the detailed information of the real optical target images.

[0036] Specifically, to achieve efficient adaptation with minimal computational overhead, low-rank adaptation (LoRA) modules are embedded in the diffusion model encoder, U-Net, and diffusion model decoder. During generator fine-tuning, all original weights of Stable Diffusion Turbo remain frozen, and only the parameters of the low-rank adaptation (LoRA) modules, the initial convolutional layers of U-Net, and the zero convolutional layers are updated. This selective training strategy effectively preserves the prior knowledge embedded in the pre-trained model while requiring only a small number of trainable parameters.

[0037] Specifically, the virtual sonar target generation module includes an image conversion discriminator. For the image conversion discriminator, such as... Figure 3 As shown, drawing inspiration from the design of Vision-Aided GANs, a pre-trained CLIP model is used as the frozen backbone network, which is generated from each input target image. I t High-dimensional feature representations are extracted from the data. These representations are then processed through two consecutive convolutional downsampling blocks: each downsampling block includes: a small-scale spatial convolutional layer (3×3 kernel) for channel projection, a LeakyReLU activation function layer, a blurred pooling layer for anti-aliasing downsampling, and a pointwise convolutional layer (1×1 convolution with a stride of 2) for further downsampling. The downsampled features are fed into a feedforward module consisting of a fully connected layer and subsequent LeakyReLU activation function layers. Finally, the fully connected layer maps the 256-dimensional vector to discriminator scores. D ( I t ).

[0038] Specifically, such as Figure 4 As shown, the training and enhancement process of the visual-text fusion network includes: given a real sonar target image and real optical target images Input them into the generator respectively and Generator and Using the same network structure, but with different text prompts. and As a condition. Generator and Generating virtual optical target images respectively and virtual sonar target images Specifically, it is expressed as: , ,in, and These respectively represent the text prompts "This is an optical image" and "This is a sonar image".

[0039] An adversarial loss function is employed to ensure that the discriminator cannot distinguish between virtual optical target images and real optical target images, or between virtual sonar target images and real sonar target images. Two discriminators are constructed. and : Among them, the discriminator Used to distinguish virtual optical target images and real optical target images Among them, the discriminator Used to distinguish virtual sonar target images and real sonar target images To enable the discriminator Unable to distinguish virtual optical target images and real optical target images And the discriminator Unable to distinguish virtual sonar target images and real sonar target images Adversarial loss function The definition is as follows: , where E[·] is the expectation value operator.

[0040] Furthermore, a cycle consistency loss function is used to ensure that images can be mapped back to their original appearance after successive transformations. This function forces forward and backward mappings to remain consistent, thereby improving the stability of the training process and the reliability of the generated results. Cycle Consistency Loss Function The definition is as follows:

[0041] in, The reconstruction loss function consists of three parts, including: L 1. Difference, learning-aware image patch similarity index (LPIPS) and structural similarity index (SSIM). L The 1 loss emphasizes pixel-level point-by-point accuracy, helping to preserve overall fine details; the LPIPS loss targets high-level semantics and perceptual quality, effectively addressing the problem of similar pixel values ​​but different visual perceptions; the SSIM loss constrains local structure and texture consistency, ensuring that the generated image maintains a coherent structure in local regions. These components together form a balanced constraint on image quality.

[0042] The reconstruction loss function is defined as follows:

[0043] In the formula, X The processed image, Y For the label image, express X and Y The comprehensive reconstruction error between the three dimensions of pixel accuracy, perceived quality, and structural consistency; express X and Y The sum of the absolute values ​​of the differences between corresponding pixel values; express X and Y Differences in high-level semantics and visual perception; express X and Y Similarity in local structure and texture; To balance the difference loss L A weighting factor of 1 To balance the weight coefficients of the learning-aware image patch similarity loss LPIPS, To balance the weighting coefficients of the Structural Similarity Index (SSIM) loss, an identity loss function is introduced to further preserve content from images already belonging to the target domain. When the input image comes from the target domain, this function prompts the generator to output a result that highly matches the input, helping to avoid unnecessary modifications. Identity Loss Function The definition is as follows: .

[0044] The complete training loss function for the generator is defined as follows: In the formula, Let the total loss function of the generator be . To balance the weighting coefficients of the adversarial loss function, To balance the weighting coefficients of the cycle consistency loss function, The weighting coefficients are used to balance the identity loss function.

[0045] For discriminator and A corresponding loss function was designed to distinguish between real optical target images and virtual optical target images, as well as real sonar target images and virtual sonar target images. Discriminator and The total loss function is defined as follows: .

[0046] generator and With discriminator and Adversarial training strategy is adopted to make the generator and The ability to generate high-quality, domain-consistent images is gradually enhanced through continuous optimization. After training, only the generator is needed. This allows for the generation of virtual sonar target images. Specifically, given a real optical target image... and text prompts generator It will output virtual sonar target images .

[0047] The virtual sonar image background reconstruction module addresses the problem that generated virtual sonar target images only contain the target and lack a realistic background. This module automatically supplements virtual sonar target images containing only the target region with background noise distributed in a manner consistent with the real sonar background image, achieving a natural fusion of the target and background. Without destroying the target structure, this module uses an unsupervised training strategy to ensure that the texture features, spectral characteristics, and speckle noise distribution of the generated virtual sonar image with background are consistent with those of the real sonar image.

[0048] like Figure 5 As shown, the training method for the virtual sonar image background reconstruction module is as follows: First, the virtual sonar target image generated by the virtual sonar target generation module is used as the training method for the virtual sonar image background reconstruction module. As input. To enhance diversity and scene adaptability, virtual sonar target images are used. Perform random scaling, scaling scale s Its scaling factor for: ,in, Represents the original height of the virtual sonar target image. The original width of the virtual sonar target image is represented; the scaled virtual sonar target image is obtained by bilinear interpolation based on the scaling factor. In real sonar background images Randomly select an insertion position within the image and scale the virtual sonar target image. Embedded into real sonar background image In the process, a combined image is formed. For combined images Create a binary mask M The area with a mask value of 1 represents the scaled virtual sonar target image. The area with a mask value of 0 corresponds to the background area. Combine the images. With mask M The images are stitched together along the channel dimension to form a virtual sonar image background reconstruction network. H Input: ,in, The input tensor for the virtual sonar image background reconstruction network H.

[0049] Virtual Sonar Image Background Reconstruction Network H Used for combining images Perform background compensation and brightness / texture adjustments to output a virtual sonar image with a background. , is represented as: .

[0050] Virtual Sonar Image Background Reconstruction Network H A generative structure based on convolutional neural networks is adopted, including a background reconstruction encoder, a background reconstruction decoder, and a multi-scale skip connection layer. The background reconstruction encoder consists of multiple layers of two-dimensional convolutional layers, batch normalization layers, and activation function layers connected in sequence. It is used to downsample the input image after channel stitching layer by layer and extract multi-scale features covering background structure information and noise distribution information. The background reconstruction decoder consists of deconvolutional layers and is used to restore the spatial resolution and texture details of the image layer by layer based on multi-scale features. The multi-scale skip connection layer is used to establish connections between the corresponding feature layers of the background reconstruction encoder and the background reconstruction decoder, directly passing multi-scale features to the corresponding layers of the background reconstruction decoder to achieve the fusion of global background structure and local texture features. Among them, the multiple layers of two-dimensional convolutional layers are used to extract spatial features and reduce the resolution of feature maps; the batch normalization layer is used for normalization processing to stabilize training; the activation function layer adopts the LeakyReLU function to introduce nonlinear transformation; the deconvolutional layer and the activation function layer are connected by the multi-scale skip connection layer for layer-by-layer upsampling to restore the spatial resolution of the image. The network input has 4 channels (a three-channel combined image plus a one-channel mask), and the output is a three-channel image.

[0051] Specifically, to improve the realism and stability of the background, a virtual sonar image background reconstruction network is used. H With discriminator This forms the adversarial training framework. Discriminator A convolutional neural network structure is used, with the input being a virtual sonar image with a background. Or real sonar background image The output is a virtual sonar image with a background. Or real sonar background image Local area authenticity scoring map or Each convolutional block contains a convolutional layer, a spectral normalization layer, and a LeakyReLU activation function layer, used to stabilize adversarial training and capture differences in local noise distribution. Discriminator Instead of using global pooling, it outputs a two-dimensional feature map to reflect the local discrimination results.

[0052] Specifically, during the training phase, the virtual sonar image background reconstruction network is constrained by multiple unsupervised loss methods. HThe generation effect. Losses include: (1) Adversarial loss: In order to make the discriminator Unable to distinguish virtual sonar images with background and real sonar background images Adversarial loss function The definition is as follows: .

[0053] (2) Spectral consistency loss: Comparison of virtual sonar images with background Background Compared with real sonar background images Background radial power spectral density , is represented as:

[0054] in The radial spectral energy curve of the background region constrains the frequency distribution characteristics of the generated background; ⊙ represents pixel-by-pixel multiplication.

[0055] (3) Speckle noise statistical loss: Virtual sonar image with background Background Compared with real sonar background images Background Mean squared error on the Rayleigh distribution scaling parameter , can be represented as: .

[0056] (4) Total variational smoothing loss Virtual sonar images with background Background Applying total variational regularization to suppress artifacts and preserve smoothness is expressed as:

[0057] in, N for The total number of pixels; coordinates The value of the pixel adjacent to the right; coordinates The value of the adjacent pixel above; coordinates The current pixel value at that location.

[0058] (5) Target reconstruction loss, used to ensure that the shape and details of the original virtual sonar target image remain unchanged during background generation. It consists of three parts, including... Difference, learning-aware image patch similarity index (LPIPS) and structural similarity index (SSIM). The loss function emphasizes pixel-level accuracy, helping to preserve overall fine details; LPIPS loss targets high-level semantics and perceptual quality, effectively addressing the problem of similar pixel values ​​but different visual perceptions; SSIM loss constrains local structure and texture consistency, ensuring that the generated image maintains a coherent structure in local regions. These components collectively provide a balanced constraint on image quality. (Target Reconstruction Loss Function) Defined as:

[0059] In the formula, To balance the difference loss L A weighting factor of 1 To balance the weight coefficients of the learning-aware image patch similarity loss LPIPS, The weighting coefficients of the structural similarity index loss SSIM are used to balance the structural similarity index loss.

[0060] Virtual Sonar Image Background Reconstruction Network H The complete training loss function is defined as follows:

[0061] In the formula, Network for reconstructing background of virtual sonar images H The total loss function, To balance the weighting coefficients of the adversarial loss function, To balance the weighting coefficients of the spectral consistency loss, To balance the weighting coefficients of the statistical loss of speckle noise, To balance the weighting coefficients of the total variation smoothing loss, Weighting coefficients for balancing the reconstruction loss of the target.

[0062] Specifically, for the discriminator Design a corresponding loss function to distinguish virtual sonar images with background. and real sonar background images Discriminator loss function The definition is as follows: .

[0063] Specifically, a network for reconstructing background data using virtual sonar images is employed. H With discriminator An adversarial training strategy is used to improve the fusion quality of the target and background. The training process is as follows: First, the virtual sonar target image generated by the virtual sonar target generation module is used... The composite image is constructed by pasting the images onto a real background image at random scales and positions. With corresponding mask M ; and then Background reconstruction network fed into virtual sonar image H To generate virtual sonar images with background and virtual sonar images with background Compared with real sonar background images Input discriminator respectively In each training step, the parameters of the reconstructed network are first fixed, and the discriminator is optimized. To enhance its discriminative ability, the discriminator is then fixed. The parameters are optimized to simultaneously minimize adversarial loss, spectral consistency loss, blob statistics loss, total variational smoothing term, and target reconstruction loss. Through alternating iterations, adversarial training enables the reconstruction network to gradually learn to generate noise and texture consistent with the statistical characteristics of real sonar images while preserving target details. After training convergence, the background reconstruction network can directly and naturally fuse virtual sonar target images with real sonar background images, and the output synthetic image closely resembles the real sample in terms of visual appearance, spectral density, blob statistics, and other aspects.

[0064] To verify the effectiveness and applicability of the proposed virtual sonar image generation method and system, a complete experimental dataset was constructed based on real underwater sonar imaging data and optical image data. Training and evaluation were carried out under a unified data preprocessing and strict data partitioning strategy. To ensure the representativeness and reliability of the experimental results, the data used not only covers multiple types of typical underwater targets but also includes large-scale real underwater sonar background images, which can fully reflect the generalization ability and target generation quality in complex underwater acoustic environments.

[0065] The experimental verification used real sonar images from an underwater acoustic target detection dataset, which contains 7600 real sonar images covering 10 target categories: cube, sphere, cylinder, human body, tire, circle cage, square cage, metal bucket, plane, and rov. Based on the target detection annotation files provided in the dataset, target regions were cropped from the real sonar images, resulting in 12296 real sonar target images across 8 categories. For ease of training, all real sonar target images were uniformly resized to 256×256 pixels. In addition, real optical images of the same 10 target categories were collected, 16 images per category, and cropped to retain only the target regions, ultimately yielding 160 real optical target images. In the experiment, six categories of objects—spheres, cylinders, human bodies, tires, square cages, and metal buckets—were selected from both real sonar target images and real optical target images as training sets. Two categories—planes and rovs—were used as validation sets, and two categories—cubes and circular cages—were used as test sets for training and evaluating the virtual sonar target generation module. Simultaneously, 7600 real sonar images from the underwater acoustic target detection dataset were used to generate real sonar background images. 80% of these real sonar background images were embedded into the generated virtual sonar target images for training the virtual sonar image background reconstruction module; the remaining 10% and 10% were used for validation and testing. To ensure input consistency, all real underwater sonar background images were normalized to 1024×1024 pixels.

[0066] Figure 6 This demonstrates the consistency between the virtual sonar target image generated by the virtual sonar target generation module based on a real optical target image and the real sonar target image in terms of texture structure, brightness distribution, and scattering characteristics. Figure 6 As can be seen, the virtual sonar target generation module proposed in this application can generate virtual sonar target images that are highly consistent with real sonar target images in terms of texture, brightness distribution, and scattering characteristics. The generated results maintain good consistency with real sonar target images in terms of target contour, energy attenuation characteristics, and noise distribution.

[0067] also, Figure 7 This demonstrates the fusion effect of virtual sonar target images and real sonar background images. (By...) Figure 7As can be seen, the virtual sonar image background reconstruction module proposed in this application can achieve a natural fusion of the virtual sonar target image and the real sonar background image. While preserving the background texture and noise features, the module dynamically adjusts the brightness contrast and scattering consistency between the virtual sonar target image and the real sonar background image, making the generated virtual sonar image with background almost indistinguishable from the real sonar image in overall visual appearance.

[0068] The above embodiments are used to explain this application, not to limit it. Any modifications and changes made to this application within the spirit and scope of the claims shall fall within the protection scope of this application.

Claims

1. A method for generating virtual sonar images, characterized in that, The method includes: A visual-text fusion network is used to map real optical target images into virtual sonar target images; A composite image is generated by embedding a virtual sonar target image into a real sonar background image. A virtual sonar image background reconstruction network is used to perform background compensation and texture consistency adjustment on the combined image to obtain a virtual sonar image with a background. The step of embedding a virtual sonar target image into a real sonar background image to generate a combined image includes the following steps: The virtual sonar target image is scaled, and bilinear interpolation is used to obtain the scaled virtual sonar target image. Select an insertion point within a real sonar background image, and embed a scaled virtual sonar target image into that position to generate a composite image; Construct a binary mask for the combined image The area with a mask value of 1 corresponds to the area where the scaled virtual sonar target image is located, and the area with a mask value of 0 corresponds to the background area. Combine the image with a binary mask The images are stitched together along the channel dimension to form the input to the virtual sonar image background reconstruction network.

2. The virtual sonar image generation method according to claim 1, characterized in that, The virtual sonar image background reconstruction network includes: The background reconstruction encoder is used to downsample the input image after channel stitching layer by layer and extract multi-scale features that cover background structure information and noise distribution information. A background reconstruction decoder is used to recover the spatial resolution and texture details of an image layer by layer based on multi-scale features; The multi-scale skip connection layer is used to establish connections between the corresponding feature layers of the background reconstruction encoder and the background reconstruction decoder, and directly transmits multi-scale features to the corresponding layer of the background reconstruction decoder to achieve the fusion of global background structure and local texture features.

3. The virtual sonar image generation method according to claim 2, characterized in that, The background reconstruction encoder includes a series of two-dimensional convolutional layers, a batch normalization layer, and an activation function layer connected in sequence. The two-dimensional convolutional layers are used to extract spatial features and reduce the resolution of the feature map. The batch normalization layer is used for normalization processing to stabilize training. The activation function layer is used to introduce nonlinear transformation. The background reconstruction decoder includes a deconvolution layer, which is connected to the activation function layer through a multi-scale skip connection layer for upsampling layer by layer to restore the spatial resolution of the image.

4. The virtual sonar image generation method according to claim 2, characterized in that, The total loss function of the virtual sonar image background reconstruction network is expressed as follows: In the formula, Let $\frac{ ... To balance the weighting coefficients of the adversarial loss, To counteract the loss function, To balance the weighting coefficients of the spectral consistency loss, Let spectral consistency loss function be used. To balance the weighting coefficients of the statistical loss of speckle noise, Let be the statistical loss function for speckle noise. To balance the weighting coefficients of the total variation smoothing loss, The total variational smoothing loss function is... To balance the weighting coefficients of the target reconstruction loss, Reconstruct the loss function for the target; The adversarial loss function is expressed as: In the formula, E[·] is the expected value operator. Virtual sonar images with background A local area authenticity scoring map; The spectrum consistency loss function is expressed as: In the formula, The radial spectral energy curve of the background region constrains the frequency distribution characteristics of the generated background; ⊙ represents pixel-by-pixel multiplication operation. This is a real sonar background image; The statistical loss function for the speckle noise is expressed as follows: ； The total variational smoothing loss function is expressed as: In the formula, N Virtual sonar images with background Total number of pixels in the background; coordinates The value of the adjacent pixel to the right; coordinates The value of the adjacent pixel above; coordinates The current pixel value at that location; The target reconstruction loss function is expressed as: In the formula, To balance the difference loss L A weighting factor of 1 To balance the weight coefficients of the learning-aware image patch similarity loss, The weighting coefficients are used to balance the loss of the structural similarity index.

5. The virtual sonar image generation method according to claim 1, characterized in that, The training method for the vision-text fusion network is as follows: Given real sonar target images and real optical target images, each with different text prompts, input them into a generator with the same structure. With generator In the process, virtual optical target images and virtual sonar target images are generated respectively; Construct a discriminator To distinguish between virtual optical target images and real optical target images, a discriminator is constructed. Used to distinguish between virtual sonar target images and real sonar target images; Define an adversarial loss function so that the discriminator It cannot distinguish between virtual optical target images and real optical target images, and the discriminator... Unable to distinguish between virtual sonar target images and real sonar target images; Define a cycle consistency loss function so that an image is mapped back to its original appearance after successive transformations; An identity loss function is introduced to further preserve image content that already belongs to the target domain; After training, use only the generator. This allows for the generation of virtual sonar target images.

6. The virtual sonar image generation method according to claim 4, characterized in that, The visual-text fusion network uses a pre-trained Stable Diffusion Turbo model as the generator, which includes a diffusion model encoder, a U-Net, and a diffusion model decoder. The visual-text fusion network is configured as follows: Input the real optical target image into the noise coding path of U-Net; Text embeddings are generated using a CLIP-based text encoder and injected into U-Net via a cross-attention mechanism; A skip connection is established between the diffusion model encoder and the diffusion model decoder through a zero-convolutional layer to preserve the detailed information of the real optical target image in the virtual sonar target image.

7. The virtual sonar image generation method according to claim 6, characterized in that, Low-rank adaptation modules are embedded in the diffusion model encoder, U-Net, and diffusion model decoder. During generator fine-tuning, the original weights of Stable DiffusionTurbo are kept frozen, and only the parameters of the low-rank adaptation modules, the initial convolutional layer of U-Net, and the zero convolutional layer are updated. At the same time, the text encoder remains completely frozen during fine-tuning.

8. A virtual sonar image generation system, used to implement the virtual sonar image generation method as described in any one of claims 1-7, characterized in that, The system includes: The virtual sonar target generation module is used to map real optical target images into virtual sonar target images; The virtual sonar image background reconstruction module is used to embed the virtual sonar target image into the real sonar background image and output the virtual sonar image with the background.

9. The virtual sonar image generation system according to claim 8, characterized in that, The virtual sonar target generation module includes an image conversion discriminator, which uses a pre-trained CLIP model as a frozen backbone network to extract high-dimensional feature representations from the input target image. The feature representations are first processed through two consecutive convolutional downsampling blocks, then fed into a feedforward module, and finally mapped to discriminator scores through a fully connected layer.

Images