An underwater image dataset augmentation method and system based on a generative model

Abstract

This invention discloses a method and system for expanding underwater image datasets based on a generative model. The method first constructs a pairing dataset of side-scan sonar images and text descriptions. After encoding the images into the latent space and adding noise, the LoRA module is used for fine-tuning while freezing the parameters of the basic diffusion model, enabling the model to learn the bright areas, shadow areas, and background texture features in the side-scan sonar images. Subsequently, new samples are generated based on preset prompts, and these generated samples are filtered and fused with the original samples to form an expanded dataset. This approach can improve the structural diversity and style consistency of side-scan sonar training samples, reduce data acquisition and annotation costs, and improve the training effect of downstream target detection models.

Description

Technical Field

[0001] This invention relates to the field of underwater information processing and intelligent sensing technology, and in particular to a method and system for expanding underwater image datasets based on generative models. Background Technology

[0002] Side-scan sonar is widely used in tasks such as seabed topography mapping, underwater target detection, subsea pipeline inspection, and shipwreck search due to its wide detection range, high imaging efficiency, and strong adaptability to complex underwater environments. Side-scan sonar images typically characterize the geometric and scattering properties of underwater targets through bright areas, shadow areas, and texture structures. Therefore, automated target detection based on side-scan sonar images has high engineering value.

[0003] However, compared to natural optical images, side-scan sonar images typically suffer from high noise, low contrast, complex textures, and significant imaging distortion, making target detection more challenging. Meanwhile, while deep learning-based detection methods can automatically extract features and achieve good detection results, they usually rely on large-scale, high-quality labeled data. For side-scan sonar scenarios, data acquisition requires specialized equipment and offshore operations, resulting in high costs and long lead times. Furthermore, labeling relies on professional personnel, leading to generally small sizes and uneven class distributions in publicly available datasets.

[0004] Common data augmentation methods include flipping, rotating, scaling, cropping, and brightness adjustment. While these methods are simple to implement, they essentially involve geometric transformations or pixel perturbations of existing samples, making it difficult to introduce new target structural patterns. In side-scan sonar image scenarios, due to the strong imaging patterns between bright and shadow areas of the target and background texture, relying solely on traditional augmentation methods is insufficient to effectively compensate for the missing structural distribution in the training data.

[0005] Existing research has attempted to use generative models such as GANs for image synthesis, but GANs typically suffer from problems such as training instability, pattern collapse, and insufficient sample diversity. Diffusion models approximate the real data distribution through progressive noise addition and reverse denoising, resulting in better performance in terms of image generation quality and diversity. On the other hand, diffusion models have a large parameter scale, and direct full parameter fine-tuning leads to high computational costs and may cause overfitting under small sample data conditions.

[0006] Therefore, there is an urgent need to propose an image dataset expansion scheme that can adapt to the imaging characteristics of side-scan sonar images, reduce training costs, and improve the diversity of data structures. Summary of the Invention

[0007] This invention proposes a method for expanding underwater image datasets based on generative models, which can effectively solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A method for augmenting underwater image datasets based on generative models includes the following steps: 1. Construct a dataset of image-text pairing consisting of side-scan sonar images and text descriptions; 2. Use a variational autoencoder to encode side-scan sonar images into a latent space representation; 3. Add noise to the latent space representation and encode the text description into a text conditional vector; 4. Input the noisy latent representation and text conditional vector into the basic diffusion model. With the parameters of the basic diffusion model frozen, train only the LoRA module to obtain a fine-tuned model for side-scan sonar scenarios. 5. During the inference phase, input prompt words and randomly initialize noise latent variables, and gradually denoise and generate the latent representation of the target image through the fine-tuning model; 6. Decode the latent representation of the target image to obtain the generated side-scan sonar image; 7. Filter the generated images and merge them with the original images to form an expanded training dataset.

[0009] Preferably, the text description includes at least two of the following: target semantics, sonar image type, bright area features, shadow features, and background texture.

[0010] Preferably, the text description includes trigger words for establishing sonar style constraints.

[0011] Preferably, the LoRA module is inserted into the critical linear layer, attention layer, or cross-modal fusion layer of the basic diffusion model.

[0012] Preferably, the step of filtering the generated image includes manual and / or automatic filtering of the target sharpness, shadow rationality and background consistency of the generated image.

[0013] Preferably, the extended training dataset is used to train the underwater target detection model.

[0014] Advantages compared to existing technologies: This invention provides a dataset augmentation technique for side-scan sonar image scenarios. Instead of simply flipping, rotating, or altering the brightness of existing images, this technique establishes a correspondence between side-scan sonar images and text descriptions. It utilizes a diffusion model to learn the structural distribution and stylistic features of side-scan sonar images in the latent space, and then combines this with an efficient LoRA parameter fine-tuning method. This allows the model to adapt to the side-scan sonar domain under limited sample conditions, ultimately generating synthetic image samples with consistent target semantics, bright area structure, shadow features, and background texture. These synthetic samples are then fused with the original samples to construct an expanded training dataset. This achieves the following significant effects: 1. Generating new samples through diffusion models rather than traditional geometric enhancement methods can introduce new target structural patterns and improve the structural diversity of side-scan sonar training data.

[0015] 2. The LoRA method is used to efficiently fine-tune the parameters of the basic diffusion model, requiring only a small number of additional parameters to be trained, thus reducing the GPU memory and computing resources required for training.

[0016] 3. By using image-text pairing and trigger word constraints, the generated samples maintain good consistency between the target semantics and the side-scan sonar style.

[0017] 4. By screening the generated samples, we can ensure that the samples included in the extended dataset have good usability.

[0018] 5. By applying the extended dataset to downstream detection tasks, the training effect of the object detection model can be improved, especially in scenarios with small sample sizes and class imbalance.

[0019] This invention is particularly suitable for the following scenarios: First, the cost of acquiring real side-scan sonar images is high, making it difficult to obtain large-scale samples in a short period; second, the number of samples for certain target categories is significantly insufficient, leading to class bias during the training of the detection model; third, existing traditional data augmentation methods are unable to generate new target structure patterns and cannot effectively compensate for gaps in the structural distribution of the training set. Target recognition in side-scan sonar images depends not only on brightness intensity but also significantly on the spatial relationship between bright areas, shadow areas, and background textures of the target. Therefore, the key to this invention is not merely "generating images," but rather generating images that conform to the imaging rules of side-scan sonar. Attached Figure Description

[0020] Figure 1 This is a flowchart of the overall process for the side-scan sonar image dataset augmentation method based on diffusion model and LoRA fine-tuning of the present invention. Figure 2 This is a flowchart illustrating the steps of the underwater image dataset augmentation method based on a generative model according to the present invention. Detailed Implementation

[0021] Example 1, refer to Appendix Figure 1-2 An underwater image dataset augmentation method based on a generative model includes the overall system structure, acquisition and preprocessing of original side-scan sonar image samples, construction of paired datasets, training of a diffusion model based on latent space, efficient fine-tuning of LoRA parameters, design and inference generation of cue words, quality screening of generated samples, construction of extended training datasets, application of extended datasets in downstream tasks, implementation of out-of-class target structure transfer, method flow examples, and alternative implementation methods.

[0022] The overall system structure includes a data preparation unit, an image-text pairing construction unit, a latent space coding unit, a condition construction unit, a model fine-tuning unit, an image generation unit, a sample selection unit, and an extended dataset construction unit.

[0023] The data preparation unit is used to acquire or import raw side-scan sonar images; the image-text pairing construction unit is used to generate corresponding text descriptions for each side-scan sonar image; the latent space encoding unit is used to map images to latent space representations; the condition construction unit is used to convert text descriptions into text condition vectors; the model fine-tuning unit is used to train the LoRA module under the condition of freezing the basic model parameters; the image generation unit is used to generate side-scan sonar style images under the condition of prompt words; the sample screening unit is used to perform quality control on the generated images; and the extended dataset construction unit is used to merge the screened samples with the original samples to form an extended training set.

[0024] The above units can be deployed on the same computing device or distributed across multiple server nodes.

[0025] The computing device may include a GPU workstation, a server, an edge computing device, or a cloud training platform. Examples of training environments include PyTorch 2.2.0, CUDA 12.1, Python 3.11, Intel Core i9-14900HX, and NVIDIA GeForce RTX 5070. This environment can be used as one implementation environment for the present invention, but the present invention is not limited to this specific hardware and software version.

[0026] Acquisition and preprocessing of raw side-scan sonar image samples: First, raw side-scan sonar image samples are acquired. These raw image samples can be derived from public datasets, self-acquisition mission data, historical project databases, or other authorized side-scan sonar image sources. Preferably, the samples include at least typical targets such as ships, aircraft wreckage, and human bodies to cover representative categories commonly encountered in underwater detection.

[0027] In one example, the original dataset contains 1915 images, including three types of targets: ships, aircraft wreckage, and human bodies.

[0028] Because side-scan sonar images from different sources may vary in imaging resolution, grayscale range, background noise level, and target scale, preprocessing is preferably performed before they are used for model training. This preprocessing may include one or more of the following: Image format standardization: Convert images of different formats to PNG, JPG, or BMP format. The image size is standardized, and the input size is adjusted to the fixed resolution required for model training. Grayscale range normalization is used to reduce amplitude differences caused by different equipment or different flight acquisition conditions; Noise sample removal: Remove images that are severely out of focus, have severe motion blur, lack echoes, or have unidentifiable targets. Background anomaly removal removes images that are not representative or contain large areas of invalidity.

[0029] When manually screening images, the focus is on retaining samples with clear target bright areas, complete shadow structures, representative background textures, and easily identifiable target edges. This is because the target structure in side-scan sonar images often depends on the combination relationship between the bright echo area and the shadow area behind it. Therefore, if this combination relationship is unclear, it will be detrimental to the model learning stable domain features.

[0030] Construction of the image-text pairing dataset: Instead of directly inputting images into the diffusion model for unconditional training, this invention first constructs a dataset of image-text pairings with text descriptions to achieve conditional generation.

[0031] Specifically, for each side-scan sonar image, a corresponding text description is generated. The text description preferably includes at least two of the following information, and more preferably includes three or more: Target category semantic information, including ships, airplanes, human bodies, submarine cables, vehicles, etc.; Image type information, including side-scan sonar images; Bright area structure information, including bright echo areas and strong scattering areas; Shadow structure information, including dark shadows behind the target, long shadows, or short shadows; Background texture information, including seabed background, sediment texture, flat background or rough background; Target geometric state information, including slender targets, blocky targets, humanoid targets, and debris targets; Information about the target's orientation or spatial relationships, such as its extension along the flight path or the shadow cast behind it.

[0032] A side-scan sonar image containing a shipwreck target can be described as: "Side-scan sonar image of a shipwreck target with bright echo areas and distinct dark shadows on the seabed background."

[0033] An image of a human body can be described as follows: "Side-scan sonar image: a humanoid target against the seabed background, the main body is brighter, and there is a thin shadow behind it."

[0034] With this descriptive method, the text conditions no longer merely express "what the target is," but further express "how the target should be presented in side-scan sonar."

[0035] A unified trigger word is added to the text description to explicitly identify that the current sample belongs to the side-scan sonar style domain. For example, "sonar image" or its equivalent Chinese expression can be used as a style cue, enabling the model to establish a stable mapping relationship between the trigger word and the side-scan sonar imaging style during fine-tuning, thereby improving the consistency of subsequent generated results in terms of domain style. This type of trigger word can be used in both the training and inference phases.

[0036] Training of diffusion models based on latent space: The basic generative model used in this invention is the FLUX.1 diffusion model. This model does not directly perform diffusion and denoising in the pixel space. Instead, it first uses a variational autoencoder to compress and map the image to a low-dimensional latent space, and then performs noise modeling and inverse denoising in the latent space. This approach has two significant advantages: first, it reduces computational complexity and training overhead; second, it preserves the main structural information in the image, making it suitable for learning the spatial relationships between targets, shadows, and background in side-scan sonar images.

[0037] The specific training process is as follows: First, the image is input into the Flux VAE encoder, which outputs the corresponding latent representation. This latent representation compresses the original image information but retains key imaging features such as target contours, brightness relationships, and background texture. Then, noise is added to this latent representation according to a preset noise scheduling strategy, resulting in noisy latent variables at different time steps. Simultaneously, the corresponding text description is input into the text encoder to obtain a text conditional vector. Next, the noisy latent variables and the text conditional vector are input together into the FLUX.1 base model, which learns to perform a noise prediction task under given noise levels and text conditions, thereby gradually mastering the distribution of side-scan sonar images.

[0038] The noise scheduling strategy can employ linear scheduling, cosine scheduling, or other scheduling methods suitable for training diffusion models; the training loss can be noise prediction loss, latent space reconstruction loss, or a combination thereof. Although forward diffusion and backward denoising of diffusion models have general mathematical expressions, this invention is not limited to a specific noise scheduling formula, as long as it can achieve conditional diffusion training based on latent space.

[0039] Efficient LoRA parameter fine-tuning method: Since a complete basic diffusion model typically contains a large number of parameters, directly fine-tuning all parameters would require significant computing power and GPU memory, and is prone to overfitting in side-scan sonar scenarios with limited sample sizes. Therefore, this invention employs a LoRA fine-tuning strategy.

[0040] Specifically, the LoRA module is inserted into the key linear layer, attention layer, or cross-modal information interaction layer of the FLUX.1 model. During training, the original base model weights remain unchanged, and only the low-rank matrix parameters introduced by LoRA are trained. This allows for domain adaptation of the model with fewer parameter updates. The core idea of ​​LoRA is to introduce low-rank increment terms on the original weight matrix, rather than directly updating all model weights, making it particularly suitable for tasks with "limited sample size but clear style adaptation requirements".

[0041] The rank r of LoRA can be set to 4, 8, 16, 32 or other reasonable range values; the same or different rank values ​​can be set for different layers during training; the rank value can also be dynamically adjusted according to GPU resources and training set size.

[0042] The LoRA module primarily learns the distributional differences between the natural image domain and the side-scan sonar image domain, enabling the model to transition from its original pre-trained semantic capabilities to the ability to express the patterns of bright areas, shadow areas, and background textures in side-scan sonar. In other words, this invention does not enable the model to "generate side-scan sonar images from scratch," but rather, based on its existing generation capabilities, allows the model to master the features of the side-scan sonar domain.

[0043] The process of prompt word design and reasoning generation: After LoRA fine-tuning, the model proceeds to the inference generation stage. During inference, prompt words are first input, and a corresponding text conditional vector is generated by the text encoder. Then, initial noise latent variables are randomly sampled from a Gaussian distribution as the starting point for image generation. Next, under the combined effect of text conditions and LoRA weights, the model performs multi-step inverse denoising on the noise latent variables, gradually obtaining a target latent representation that meets semantic requirements and has side-scan sonar-style characteristics. Finally, the target latent representation is restored to the output image using a Flux VAE decoder.

[0044] To improve sample diversity, one or more of the following methods can be used to control the generated results: Change the random seed; Change the order of the prompt words; Replace with synonyms or near-synonyms; Add target size description; Add target orientation description; Add a description of the shadow length; Add background texture description; Add descriptions of image intensity or brightness level.

[0045] This allows the model to differentiate in terms of target scale, orientation, shadow morphology, and background coupling, thereby generating richer candidate samples. By changing the random seed and cue word descriptions, the diversity of generated samples in terms of scale, orientation, shadow length, and background texture can be improved.

[0046] The prompts can be designed as a combination of "category semantics + image structure + style trigger words". For example: "ship + bright highlight + dark shadow + sonar image"; “plane wreck + side-scan sonar + seafloor texture + shadow”; "human-shaped target + sonar image + bright target + dark trailingshadow".

[0047] The prompt words can both constrain the target semantics and enhance the consistency of domain style.

[0048] Quality screening of generated samples: Considering that the results generated by the diffusion model have a certain degree of randomness, and not all generated images are suitable as training samples, this invention sets up a sample selection step.

[0049] Sample screening can be conducted manually. Screening personnel will make decisions based on the following criteria: Is the target entity clear? Is the target outline complete? Does the shaded area exist? Is the direction of the shadow reasonable? Does the correspondence between the shadow and the subject appear natural? Is the background texture continuous? Does the overall grayscale style closely resemble a real side-scan sonar image?

[0050] If the generated image has issues such as a clear subject but missing shadows, obviously incorrect shadow positions, shadows being detached from the target, abrupt changes in background texture, or a severe deviation in image style, then the image will be removed and not included in the extended training dataset.

[0051] Sample selection can also incorporate automated screening mechanisms. For example, pre-trained side-scan sonar classifiers, style discriminators, or structural consistency scoring models can be used to automatically score candidate generated images, retaining only samples with scores above a preset threshold. Automated and manual screening can be used individually or in combination, i.e., automated screening followed by manual review.

[0052] The sample screening results can be divided into three levels: "retain", "review", and "reject". "Retain" samples are directly included in the expanded training set, "review" samples are manually reviewed, and "reject" samples are no longer included in subsequent training.

[0053] Construction of extended training dataset: The generated samples, selected through screening, are fused with the original real samples to construct an expanded training dataset. During fusion, strategies such as class balancing, random sampling, sample weighting, or stratified sampling can be combined to avoid an excessive number of generated samples that could cause the training set distribution to deviate from the real data distribution.

[0054] The original dataset contains 1915 side-scan sonar images; the traditional augmentation dataset consists of 5745 images; the diffusion augmentation dataset also consists of 5745 images to ensure comparability between different augmentation strategies. This approach can be used to compare and verify the facilitating effect of the samples generated by this invention on downstream tasks.

[0055] The construction of expanded training datasets considers not only increasing the number of samples but also class balance. For example, for classes with fewer samples in the original dataset, the proportion of generated samples for that class can be increased; for classes with already abundant original data, the proportion of generated samples can be appropriately reduced to achieve both class balance and structural diversity.

[0056] Application of extended datasets in downstream tasks: The completed extended training dataset can be further used to train side-scan sonar target detection models. These target detection models can be YOLO, or other detection models based on convolutional neural networks or Transformer structures. The extended dataset generated based on the diffusion model has been used to validate the effectiveness of YOLOv5n, YOLOv8n, and YOLOv11n.

[0057] The expanded dataset was divided into training, validation, and test sets. During training, the model underwent standard object detection training. Finally, detection performance was evaluated using metrics such as Precision, Recall, and mAP. Introducing the diffusion model to generate data significantly improved object detection performance compared to using only the original dataset or traditional augmented datasets, especially for small sample classes.

[0058] Implementation methods for out-of-class target structure migration: Although only three types of target images—ships, aircraft, and human bodies—are used during training, the model can still generate side-scan sonar-style images of other target structures not present in the training set, such as vehicles, submarine cables, and bicycles, under new cue words during inference.

[0059] The fine-tuning model of this invention learns not only the local appearance of a specific target category, but also the partial imaging structural relationships between the target's bright and shadow areas and the background. Based on this characteristic, this invention can also be used for pre-expansion of new category samples. For example, when there are very few real samples of a new target category, a batch of candidate images with a side-scan sonar style can be generated first using semantic cue words, and then manually verified before being included in the training set, thereby reducing the initial sampling cost of adding new categories.

[0060] Method flow example: The method flow of the present invention may specifically include: Step 1: Collect 1915 raw side-scan sonar images and complete the category labeling; Step 2: Perform image preprocessing including size unification, quality screening, and grayscale normalization; Step 3: Write a text description for each image, including the target category, highlight areas, shadows, and background information, and uniformly add style trigger words; Step 4: Input the image into the Flux VAE encoder to obtain the latent space representation; Step 5: Add noise to the latent space representation and encode the text description into a text conditional vector; Step 6: With the FLUX.1 base model parameters frozen, train the LoRA module to obtain the side-scan sonar style fine-tuning model; Step 7: Input new prompt words and random noise latent variables to generate candidate side-scan sonar image samples; Step 8: Manually screen the candidate samples and remove those with unreasonable shading or distorted style; Step 9: Merge the selected generated samples with the original samples to construct an expanded dataset of 5745 images; Step 10: Train the object detection model based on the extended dataset and output the detection results.

[0061] Alternative implementation methods: This invention is not limited to the FLUX.1 base model used in the examples. Other conditional generation models with latent space diffusion capabilities can also be used without departing from the core idea of ​​this invention; the LoRA module can also be replaced with other low-rank adaptation structures, as long as they are essentially efficient parameter fine-tuning methods for training a small number of additional parameters under the condition of freezing the main parameters of a large model, they can all be regarded as equivalent technical means of this invention.

[0062] Similarly, the sample selection method, prompt word template, image preprocessing method, expanded training set fusion ratio, and downstream model type can all be adjusted according to actual needs without affecting the core scheme of this invention to expand the side-scan sonar dataset through "image-text pairing + latent space diffusion + LoRA fine-tuning + sample selection".

[0063] In summary, this invention constructs an image-text pairing dataset for side-scan sonar images, combining the generative capabilities of diffusion models with the efficient adaptation capabilities of LoRA fine-tuning. It learns the bright area, shadow, and background texture patterns of side-scan sonar images under limited training sample conditions, and ensures the quality of generated samples through a screening mechanism, ultimately forming an expanded training dataset that can be used for downstream object detection tasks. Compared with traditional augmentation methods, this invention can introduce new target structure patterns and improve sample diversity; compared with full-parameter fine-tuning schemes, this invention has lower training costs and is more suitable for engineering application scenarios with limited sample size.

[0064] Example 2, refer to Appendix Figure 1-2 A method for augmenting underwater image datasets based on generative models includes the following steps: S1. Obtain raw image samples from the side-scan sonar and construct corresponding text descriptions for the raw image samples to form an image-text pairing dataset. The original image samples include at least one or more target categories such as ships, aircraft wreckage, and human bodies, and the fine-tuned model is also able to generate side-scan sonar-style images of other target structures that do not appear in the training samples under new cue words.

[0065] The text description includes image type information, structural prompts, and trigger words.

[0066] The text description includes target category semantic information and sonar image type information, and also includes at least one structural cue information among target bright area information, shadow area information, and background texture information.

[0067] The text description includes trigger words for explicitly indicating the style domain of the side-scan sonar. These trigger words are used to establish a mapping relationship between the training phase and the side-scan sonar imaging style, and to constrain the style consistency of the generated results during the inference phase.

[0068] S2. The side-scan sonar raw image samples are mapped to the latent space using a variational autoencoder to obtain the image latent representation; S3. Perform noise sampling on the image latent representation to obtain a noisy latent representation; simultaneously, input the text description into a text encoder to obtain a text conditional vector; S4. Input the noisy latent representation and the text conditional vector into the basic diffusion model. Under the condition of freezing the original parameters of the basic diffusion model, train only the LoRA module inserted in the key layer of the basic diffusion model to obtain a fine-tuned model suitable for side-scan sonar image generation. The basic diffusion model is a conditional image generation model based on latent space modeling. Its generation process is not performed directly in pixel space, but first the image is mapped to the latent space through a variational autoencoder, and then noise modeling and denoising generation are performed in the latent space.

[0069] The key layer includes at least one of an attention layer, a linear transformation layer, or a cross-modal feature fusion layer for processing latent image representations and textual conditional information.

[0070] The LoRA module achieves efficient parameter fine-tuning by introducing a low-rank increment term into the linear mapping layer weight matrix of the basic diffusion model. During training, the original weight matrix is ​​fixed, and only the low-rank matrix parameters are updated.

[0071] S5. Input prompt words and randomly initialize noise latent variables into the fine-tuning model, perform stepwise denoising generation, and obtain the latent representation of the target image; S5.1 generates side-scan sonar image samples with different target scales, target orientations, shadow lengths, or background texture features by adjusting at least one of the "cue words and random initialization noise latent variables" in the input fine-tuning model.

[0072] The "cue words and randomly initialized noise latent variables" include random seeds, cue word representation, target semantic description, shadow description, or background texture description.

[0073] S5.2 performs stepwise denoising to generate the latent representation of the target image.

[0074] S6. The latent representation of the target image is decoded by the decoder of the variational autoencoder to obtain the generated side-scan sonar image sample; S7. Perform quality screening on the generated side-scan sonar image samples; The quality screening includes judging at least one of the following in the generated samples: the clarity of the target subject, the correspondence between the target shadows, the rationality of the shadow position, and the continuity of the background texture, and removing generated samples that do not meet the preset conditions.

[0075] S8. The filtered generated samples are fused with the original image samples to construct an extended training dataset.

[0076] A side-scan sonar image dataset augmentation system based on diffusion model and LoRA fine-tuning includes: The data construction module is used to build a dataset of image-text pairings between side-scan sonar images and text descriptions; The latent space coding module is used to map the original image to the latent space; The condition building module is used to generate text condition vectors; The model fine-tuning module is used to train the LoRA module while freezing the parameters of the underlying diffusion model; The image generation module is used to generate side-scan sonar image samples based on prompt words and randomly initialized noise latent variables; The sample filtering module is used to filter the generated samples; The dataset building module is used to merge the filtered generated samples with the original image samples to form an expanded training dataset.

[0077] An electronic device includes a processor and a memory, wherein a computer program is stored in the memory.

[0078] A computer-readable storage medium having a computer program stored thereon.

Claims

1. A method for augmenting underwater image datasets based on generative models, characterized in that: The process includes the following steps: S1. Obtain raw image samples from the side-scan sonar and construct corresponding text descriptions for the raw image samples to form an image-text pairing dataset; S2. The side-scan sonar raw image samples are mapped to the latent space using a variational autoencoder to obtain the image latent representation; S3. Perform noise sampling on the image latent representation to obtain a noisy latent representation; simultaneously, input the text description into a text encoder to obtain a text conditional vector; S4. Input the noisy latent representation and the text conditional vector into the basic diffusion model. Under the condition of freezing the original parameters of the basic diffusion model, train only the LoRA module inserted in the key layer of the basic diffusion model to obtain a fine-tuned model suitable for side-scan sonar image generation. S5. Input prompt words and randomly initialize noise latent variables into the fine-tuning model, perform stepwise denoising generation, and obtain the latent representation of the target image; S6. The latent representation of the target image is decoded by the decoder of the variational autoencoder to obtain the generated side-scan sonar image sample; S7. Perform quality screening on the generated side-scan sonar image samples; S8. The filtered generated samples are fused with the original image samples to construct an extended training dataset.

2. The underwater image dataset augmentation method based on a generative model according to claim 1, characterized in that: The text description includes target category semantic information and sonar image type information, and also includes at least one structural cue information among target bright area information, shadow area information, and background texture information; the text description contains trigger words for explicitly indicating the side-scan sonar style domain, the trigger words are used to establish a mapping relationship between the training phase and the side-scan sonar imaging style, and to constrain the style consistency of the generated results during the inference phase.

3. The underwater image dataset augmentation method based on a generative model according to claim 1, characterized in that: The basic diffusion model in S4 is a conditional image generation model based on latent space modeling. Its generation process is not performed directly in pixel space, but first the image is mapped to the latent space through a variational autoencoder, and then noise modeling and denoising generation are performed in the latent space.

4. The underwater image dataset augmentation method based on a generative model according to claim 1, characterized in that: The key layer in S4 includes at least one of an attention layer, a linear transformation layer, or a cross-modal feature fusion layer for processing latent image representation and text conditional information. The LoRA module achieves efficient parameter fine-tuning by introducing a low-rank increment term into the linear mapping layer weight matrix of the basic diffusion model. During training, the original weight matrix is ​​fixed, and only the low-rank matrix parameters are updated.

5. The underwater image dataset augmentation method based on a generative model according to claim 1, characterized in that: The quality screening in S7 includes judging at least one of the following in the generated samples: the clarity of the target subject, the correspondence between the target shadows, the rationality of the shadow position, and the continuity of the background texture, and removing generated samples that do not meet the preset conditions.

6. The underwater image dataset augmentation method based on a generative model according to claim 1, characterized in that: S5 includes the following steps: S5.1 By adjusting at least one of the "cue words and random initialization noise latent variables" in the input fine-tuning model, side-scan sonar image samples with different target scales, target orientations, shadow lengths, or background texture features are generated. The "cue words and randomly initialized noise latent variables" include random seeds, cue word representation, target semantic description, shadow description or background texture description; S5.2 performs stepwise denoising to generate the latent representation of the target image.

7. The underwater image dataset augmentation method based on a generative model according to claim 1, characterized in that: The original image samples include at least one or more target categories such as ships, aircraft wreckage, and human bodies, and the fine-tuned model is also able to generate side-scan sonar-style images of other target structures that do not appear in the training samples under new cue words.

8. A side-scan sonar image dataset augmentation system based on diffusion model and LoRA fine-tuning, characterized in that, include: The data construction module is used to build a dataset of image-text pairings between side-scan sonar images and text descriptions; The latent space coding module is used to map the original image to the latent space; The condition building module is used to generate text condition vectors; The model fine-tuning module is used to train the LoRA module while freezing the parameters of the underlying diffusion model; The image generation module is used to generate side-scan sonar image samples based on prompt words and randomly initialized noise latent variables; The sample filtering module is used to filter the generated samples; The dataset building module is used to merge the filtered generated samples with the original image samples to form an expanded training dataset.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the method according to any one of claims 1 to 10.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images