A deep-sea rare biological target detection method based on a stable diffusion model

Abstract

The application discloses a deep-sea rare biological target detection method based on a stable diffusion model and belongs to the technical field of deep-sea biological target detection. Rare biological categories are determined from a deep-sea benthic biological real scene data set, and a foreground region image is extracted to construct a real target data set, a foreground image generation model is trained, and diversified synthetic foreground images are generated. A conditional control network is used to generate an external drawing of the foreground image according to a layout mask, the synthetic foreground image is fused with a deep-sea background, and a target position label is automatically generated according to the layout mask to obtain a synthetic scene image. After the synthetic quality is screened and the label frame is refined, the synthetic scene image is combined with the real scene data set to construct an enhanced training data set, and a target detection model is trained. The application solves the problem that the target detection performance is limited due to the long-tail distribution effect of the deep-sea exploration data set, and improves the detection capability of the target detection model for the rare biological categories.

Description

Technical Field

[0001] This invention belongs to the field of deep-sea biological target detection technology, specifically relating to a method for detecting rare deep-sea biological targets based on a stable diffusion model. Background Technology

[0002] Deep-sea benthic animals are an important component of the deep-sea ecosystem, possessing unique ecological and genetic value. With the development of deep-sea mining technology, the detection and protection of deep-sea benthic animals are becoming increasingly important. By employing deep-sea biological target detection algorithms, researchers can efficiently extract biological category and abundance information from deep-sea exploration data. This information can support deep-sea ecological environment monitoring and the planning and management of ecological protection zones, reducing the risk of damage to the deep-sea ecosystem from deep-sea mining activities. Deep learning-based target detection technology has become a key technology for deep-sea biological identification, with its detection accuracy and efficiency far exceeding traditional algorithms. As a data-driven approach, the training of deep learning models heavily relies on large-scale, diverse datasets. However, the collection of deep-sea optical data depends on ocean voyages, requiring the deployment of equipment such as optical towed vehicles or autonomous underwater vehicles to depths of thousands of meters for exploration and collection, resulting in extremely high collection costs. Furthermore, the density of deep-sea benthic organisms is low in most sea areas, with a few species occupying high ecological niches while most species appear very infrequently in specific environments, leading to significant long-tail distribution effects and small sample size problems in the dataset.

[0003] The scarcity of rare category data significantly limits the detection performance of deep-sea organism target detection models. Inter-class competition during model training prevents the model from iteratively optimizing for rare organism categories, causing it to tend to predict rare organisms as common ones during inference. Insufficient training samples lead to overfitting to organism morphology under specific backgrounds and shooting angles, failing to fully learn the fine-grained semantic features of rare organism categories and limiting their recognition accuracy. Existing data augmentation techniques mainly include traditional methods and generative model-based methods. Traditional data augmentation techniques primarily involve geometric transformations such as random erasure, cropping, perspective transformation, random flipping, and scaling, as well as photometric transformations such as adjusting contrast, sharpness, blur, hue, saturation, and brightness. However, traditional data augmentation methods have limited image transformation space, making it difficult to fundamentally solve the long-tail problem and the small sample size problem. While generative adversarial network-based methods can generate realistic image samples, they struggle to achieve complex scene combinations and control. While stable diffusion models have made significant progress in the field of image generation, they have hundreds of millions of parameters, making it difficult to fine-tune all parameters under the condition of scarce deep-sea biological data. Furthermore, the generated images lack spatial layout control capabilities, making them difficult to apply directly to target detection tasks.

[0004] Therefore, designing a stable diffusion model framework that combines low-rank adaptive training and conditional control to solve the above problems for the detection of rare deep-sea organisms is of great significance. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention provides a method for detecting rare deep-sea organisms based on a stable diffusion model, comprising the following steps:

[0006] S1. Data preparation steps: Count the number of samples for each biological category from the real scene dataset of deep-sea benthic organisms, identify the biological categories with a sample number lower than a preset threshold as rare biological categories, and crop the foreground area image of the rare biological category from the original image according to the coordinates of the labeled box to construct the real target dataset.

[0007] S2. Foreground image generation steps: For each rare organism category, a low-rank adaptive method is used to fine-tune the pre-trained stable diffusion model to obtain a foreground image generation model; a hierarchical text prompt word containing organism category, morphological description, lighting conditions and shooting angle is constructed to drive the foreground image generation model to generate synthetic foreground images of rare organism categories through an iterative denoising process.

[0008] S3. Background fusion and layout control steps: A stable diffusion model is fine-tuned using diverse deep-sea background images, including deep-sea sedimentary backgrounds, nodule mining area backgrounds, and basalt backgrounds, to obtain a background image generation model; the conditional control network is initialized using common pre-trained weights and kept frozen during subsequent training; only the underlying stable diffusion model is fine-tuned to enable it to learn to generate background images of a specified type under the additional guidance provided by the control network; a layout mask is generated based on the size distribution of bounding boxes in the real scene dataset; using the layout mask, the synthetic foreground image, and background cue words as conditional inputs, the conditional control network performs external drawing on the synthetic foreground image based on the layout mask, fuses the synthetic foreground image with the deep-sea background, and automatically generates target position annotations based on the bounding box coordinates in the layout mask to obtain a synthetic scene image;

[0009] S4. Enhanced dataset construction steps: The synthetic scene images are quality-screened to remove images with false morphological features, the automatically generated bounding box annotations are refined, and the screened and refined synthetic scene images are merged with the real scene dataset to construct the enhanced training dataset.

[0010] S5. Target detection model training and inference steps: Train the target detection model using the enhanced training dataset, and use the trained target detection model to detect biological targets in the deep-sea image to be detected, outputting biological category and location information.

[0011] Furthermore, in step S2, a low-rank adaptive method is used to fine-tune the pre-trained stable diffusion model, including the following sub-steps:

[0012] S21. Model initialization: Load the pre-trained stable diffusion model and freeze the original weight parameters of the U-Net network in the model.

[0013] S22. Low-rank matrix injection: In the self-attention layer and cross-attention layer of the U-Net network, trainable low-rank decomposition matrices are injected into the weight matrices of the query, key, and value, respectively; the fine-tuned query vector is calculated according to the following formula:

[0014] ;

[0015] ;

[0016] ;

[0017] in, The calculated query vector; This is the fine-tuned query weight matrix; The input image features; This is the frozen original query weight matrix; The incremental matrix introduced for low-rank adaptation; Let be the first low-rank matrix with dimension . ; This is the second low-rank matrix with dimension . ; This represents the row dimension of the original weight matrix; The column dimensions of the original weight matrix; It is a low-rank number, with a value ranging from 16 to 32. Its specific value is determined according to the complexity of the image to be generated. This represents matrix multiplication.

[0018] S23. Model Fine-tuning Training: Using rare biological foreground images and their corresponding category text descriptions from the real target dataset as training samples, the parameters of the low-rank factorization matrix are optimized by minimizing the noise prediction loss to obtain the foreground image generation model; the training parameters are set as follows: the U-Net learning rate is... The text encoder learning rate is The training batch size is 2, the number of training epochs is 6, and the training image size is [missing information]. Pixel.

[0019] Furthermore, in step S2, generating a synthetic foreground image of rare biological categories through an iterative denoising process includes the following sub-steps:

[0020] S24. Initial Noise Generation: Generate a random Gaussian noise matrix that follows a standard normal distribution. Input the variational autoencoder of the stable diffusion model to transform the Gaussian random noise into initial noise latent variables. ;

[0021] S25. Single-step noise reduction calculation: at each time step The U-Net network is based on the current noise latent variable. and text embedding Predict the noise in the current step and calculate the latent variables from the previous time step using the following formula:

[0022] ;

[0023] in, These are the latent variables corresponding to the previous time step, obtained through calculation. For the noise latent variable at the current time step; The noise scheduling parameters for the current time step. ; For the predefined first Step noise intensity parameters; From step 1 to step 2 The cumulative product of the noise scheduling parameters of each step; For the U-Net network based on the current noise latent variable Time step and text embedding Predicted noise; For the first The scaling factor for the randomness intensity of the step; This represents a random noise term that follows a standard normal distribution. The text embedding is obtained by encoding hierarchical text prompts using a text encoder.

[0024] S26. Iterative Denoising: From Time Steps Begin, iteratively execute step S25 for a total of Next, the initial noise latent variable Gradually convert to denoised latent variables ;

[0025] S27, Image Reconstruction: Denoising Latent Variables The decoder of the variational autoencoder is input to reconstruct the synthetic foreground image.

[0026] Furthermore, in step S3, generating a layout mask based on the size distribution of the bounding boxes in the real-world scene dataset includes the following sub-steps:

[0027] S31. Size Range Statistics: Statistically analyze the width and height distribution of bounding boxes for each rare organism category in the real-world dataset, and determine the minimum width of the bounding box for each category. and maximum value and the minimum height and maximum value ;

[0028] S32. Bounding box parameter sampling: The position coordinates and size parameters of the bounding box are randomly sampled and generated in the following manner:

[0029] bounding box width In the interval Uniform random sampling within the bounding box; bounding box height In the interval Uniform random sampling within the bounding box; x-coordinate of the top left corner of the bounding box In the interval Uniform random sampling within the bounding box; top left corner ordinate In the interval Uniform random sampling within the area;

[0030] in, The width of the bounding box obtained from the sampling; The height of the bounding box obtained from sampling; The x-coordinate of the top-left corner of the bounding box; The ordinate of the top-left corner of the bounding box; The width of the target image; The height of the target image; and These are the minimum and maximum statistical values ​​for the bounding box width of this category, respectively; and These are the minimum and maximum values ​​of the bounding box height for this category, respectively.

[0031] S33. Mask Image Generation: Create a layout mask image with the same size as the target image, set the pixel value of the bounding box region to 1, and set the pixel value of the background region to 0.

[0032] Furthermore, in step S3, the synthesized foreground image is fused with the deep-sea background using a conditional control network, including the following sub-steps:

[0033] S34. Conditional Encoding: Input the layout mask into the conditional encoding branch of the conditional control network to extract multi-layer layout conditional features; input the background cue words into the text encoder to obtain the background text embedding;

[0034] S35, Feature Fusion: The layout condition features output from each layer of the conditional coding branch are mapped through a zero convolutional layer and then concatenated with the features of the corresponding layer of the stable diffusion model U-Net encoder in the channel dimension.

[0035] S36. Background Generation and Fusion: The synthesized foreground image is placed at the bounding box position specified by the layout mask. The U-Net decoder is driven by the stitched features and background text embeddings to generate a deep-sea background and achieve natural fusion of the foreground and background. To alleviate the problem of abrupt edges, the conditional control network supports feathering the layout mask. That is, by transforming the hard boundary mask composed of 0 / 1 into a soft boundary mask with a 0-1 gradient transition, a smoother foreground-background boundary is obtained. During the generation process, the initial filling method of the mask area adopts the "fill" strategy, that is, the background mask area is pre-filled with the blurred content of the foreground. Compared with the setting without filling preprocessing, this strategy can make the generated background show higher consistency with the foreground in terms of lighting and tone. Only masked padding is set to 32, that is, during conditional generation, the mask is extended outward by 32 pixels to participate in denoising, thereby enhancing the continuity of shadows and textures. The resolution of the synthesized image of the conditional control network is uniformly set to 1280×960 pixels, which is consistent with the aspect ratio of the original image and the resolution meets the requirements of subsequent object detection.

[0036] S37. Automatic annotation: Based on the coordinate parameters of the bounding box in the layout mask. , , and It automatically generates target location annotation files, resulting in a composite scene image and its corresponding annotation information.

[0037] Furthermore, in step S5, the target detection model adopts the RTDETR model, and the training and inference process includes the following sub-steps:

[0038] S51. Model Construction: Construct an RTDETR object detection model with ResNet-18 as the backbone network;

[0039] S52. Loss function configuration: Set the bounding box regression loss gain to 7.5, the classification loss gain to 0.5, and the distribution focus loss gain to 1.5.

[0040] S53. Model Training: Set the initial learning rate to 0.0001, use the AdamW optimizer, and use the augmented training dataset as input to iteratively optimize the model parameters to obtain the trained object detection model.

[0041] S54, Target Detection Inference: Input the deep-sea image to be detected into the trained target detection model, and output the category prediction results and bounding box position coordinates of each biological target in the image.

[0042] Furthermore, the rare organism category includes one or more of the following: crinoids, corals, bryozoans, sea urchins, hydras, enterobacteria, and octopuses.

[0043] Furthermore, in step S2, the hierarchical text prompts are constructed as follows:

[0044] Text prompts = description of biological category + morphological description + description of lighting conditions + description of shooting angle;

[0045] The description of the organism category includes the name of the rare organism category; the description of the morphology includes the color, posture and size characteristics of the organism; the description of the lighting conditions includes whether it is a bright or dark environment; and the description of the shooting angle includes whether it is a top view, a side view or a front view.

[0046] Furthermore, in step S4, the quality screening criteria are: retaining images that can accurately represent key biological identification features and have qualified visual fidelity, and removing images with false morphological features; the refinement operation is: adjusting the boundaries of the annotation boxes that fail to closely fit the biological contour, so that the annotation boxes closely fit the biological contour.

[0047] Furthermore, the images in the real-world scene dataset are taken from benthic organism images captured by underwater towed vehicles or autonomous underwater vehicles in deep-sea mining areas; the deep-sea background includes one or more of the following: deep-sea sedimentary substrate, nodule-dense areas, sparse nodule areas, and basalt basement.

[0048] The beneficial effects achieved by this invention are as follows:

[0049] This invention employs a low-rank adaptive method to fine-tune a pre-trained stable diffusion model. By injecting trainable low-rank decomposition matrices into the self-attention and cross-attention layers of the U-Net network, the model's parameters are updated efficiently while freezing the original weight parameters. This enables the model to effectively learn the key visual features of various organisms even with a very limited number of deep-sea rare organism samples. This solves the technical challenge of fine-tuning all parameters of a large-scale pre-trained model under small sample conditions. At the same time, it preserves the general visual prior knowledge of the pre-trained model, avoids the catastrophic forgetting problem, and significantly reduces the computational resources and training data scale required for model fine-tuning.

[0050] This invention constructs hierarchical text prompts that include biological category, morphological description, lighting conditions, and shooting angle. By integrating the semantic information of the text prompts into the image generation process through the cross-attention mechanism of a stable diffusion model, different combinations of text prompts can guide the model to generate images with different attributes. This systematically expands the feature space of the training data and generates new feature combinations of biological poses, lighting conditions, and background environments that are not present in the original dataset. This overcomes the limitations of the feature distribution of the original dataset and effectively improves the diversity of generated samples.

[0051] This invention employs a conditional control network to achieve natural fusion of the synthesized foreground image and the deep-sea background. By using a layout mask as a conditional input to guide the image generation process, it achieves precise control over the spatial layout of the target. At the same time, it automatically generates target location annotation information based on the bounding box coordinates in the layout mask, greatly reducing the workload of manual annotation. This solves the problem that the foreground image lacks spatial context information and cannot be directly applied to target detection tasks, ensuring that the synthesized scene image maintains the same style as real deep-sea exploration data in terms of lighting, shadows, and texture.

[0052] This invention constructs an enhanced training dataset by merging synthetic scene images with real scene datasets, effectively increasing the number of samples for rare organisms. This alleviates the class imbalance problem and long-tail distribution effect of deep-sea organism datasets, enabling the target detection model to perform more thorough iterative optimization of rare organisms during training. This improves the model's detection performance for rare organisms and has a stronger performance improvement effect compared to traditional data augmentation methods such as geometric transformation and photometric transformation. It provides reliable technical support for deep-sea ecological environment monitoring and the planning and management of ecological protection areas.

[0053] Attached illustrations

[0054] Figure 1 This is a flowchart of a method for detecting rare deep-sea organisms based on a stable diffusion model according to the present invention.

[0055] Figure 2 The images shown are actual application test results of Example 4, where sub-image (a) is the test result of exploration data from an underwater towed vehicle (OTV) and sub-image (b) is the test result of exploration data from an underwater autonomous vehicle (AUV).

[0056] Figure 3 This is a comparison chart of the detection performance of various biological categories in Example 3, where sub-chart (a) is a comparison chart of the mAP50 index of the OTV dataset, and sub-chart (b) is a comparison chart of the accuracy index of the AUV dataset.

[0057] Figure 4This is a comparison chart of the training process in Example 3, where sub-chart (a) is the training curve of the OTV dataset and sub-chart (b) is the training curve of the AUV dataset.

[0058] Figure 5 This is a foreground image generation result diagram of Example 1, where sub-image (a) is a real image example diagram and sub-image (b) is a generated image example diagram;

[0059] Figure 6 This is a comparison of the effect of hierarchical text prompts on improving the diversity of generated images in Example 1, where sub-image (a) is a sample image of a deep-sea octopus and sub-image (b) is a sample image of a generated deep-sea octopus.

[0060] Figure 7 This is a diagram showing the background blending and layout control results of Example 1. Detailed Implementation

[0061] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are only examples. The present invention is not limited to the specific structural forms described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Reference Figure 1 This invention provides a method for detecting rare deep-sea organisms based on a stable diffusion model. This method addresses the problems of high cost of acquiring deep-sea exploration data, scarcity of rare organism samples, and long-tailed distribution effects in datasets. It expands the training dataset by introducing generative artificial intelligence techniques, thereby improving the detection performance of the target detection model for rare organisms. The stable diffusion model is an image generation model based on the diffusion process. Its core idea is to model the image generation process as a reverse denoising process that gradually restores ordered structures from disordered noise. This invention fine-tunes the pre-trained stable diffusion model using low-rank adaptive techniques, enabling it to generate images of rare deep-sea organisms. It also combines a conditional control network to control the layout of the generated images and achieve background fusion. Finally, an enhanced training dataset is constructed to train the target detection model.

[0063] The method of the present invention includes a data preparation step S1, a foreground image generation step S2, a background fusion and layout control step S3, an augmented dataset construction step S4, and an object detection model training and inference step S5.

[0064] Step S1 is the data preparation step. The purpose of this step is to identify rare biological categories from the real-world deep-sea benthic organism dataset and extract foreground images of these categories for subsequent fine-tuning of the generative model. Real-world deep-sea benthic organism datasets typically originate from images of benthic organisms taken by towed underwater vehicles or autonomous underwater vehicles (AUVs) in deep-sea mining areas. These images often feature complex environments including deep-sea sedimentary bottoms, areas with dense nodules, areas with sparse nodules, and basaltic basements. Since the acquisition of deep-sea optical data relies on ocean voyages, it requires deploying equipment such as optical towed vehicles or AUVs to depths of thousands of meters for exploration and collection, resulting in extremely high acquisition costs. Furthermore, the density of deep-sea benthic organisms is low in most sea areas, with a few species occupying high ecological niches while most species appear very infrequently in specific environments, leading to a significant long-tail distribution effect in the dataset.

[0065] In step S1, the sample count of each biological category in the real-world scene dataset is first counted. This invention identifies biological categories with sample counts below a preset threshold as rare biological categories. This threshold can be set according to the overall size of the dataset and the category distribution. Preferably, categories with sample counts ranking near the bottom of the dataset can be identified as rare biological categories. In deep-sea benthic organism detection tasks, rare biological categories may include one or more of crinoids, corals, bryozoans, sea urchins, hydras, enterobacteria, and octopuses. After identifying rare biological categories, foreground regions of these categories are cropped from the original image based on the bounding box coordinates in the real-world scene dataset. The bounding box coordinates are typically stored as the center point coordinates, width, and height of the bounding box. The cropping operation extracts rectangular regions containing biological individuals from the original image based on these coordinates. The cropped foreground region images of each rare biological category are stored separately to construct a real target dataset for fine-tuning the stable diffusion model. Since the sample count of rare biological categories is inherently small, and the size of the real target dataset is usually also small, this is precisely why a low-rank adaptive method is needed for efficient parameter fine-tuning.

[0066] Step S2 is the foreground image generation step. The purpose of this step is to train a dedicated foreground image generation model for each rare organism category and use this model to generate diverse synthetic foreground images. The stable diffusion model has been pre-trained on a large-scale image dataset and possesses the ability to generate images of general categories, but it lacks prior knowledge in the field of deep-sea organisms. To enable the stable diffusion model to generate images of rare deep-sea organisms, it needs fine-tuning. However, due to its complex conditional coding structure and large-scale attention module, the stable diffusion model has hundreds of millions of parameters. Full parameter fine-tuning of such a large model requires substantial computational resources and training data. Given the scarcity of deep-sea biological data, full parameter fine-tuning of a large model is not feasible.

[0067] This invention employs a low-rank adaptive method to fine-tune a pre-trained stable diffusion model. Low-rank adaptive is a parameter-efficient fine-tuning technique. Its core idea is to update the weights by injecting a set of trainable low-rank decomposition matrices into the pre-trained model. It can achieve performance similar to full-parameter fine-tuning with only a small number of parameters.

[0068] Step S2 includes sub-steps S21 to S27. In sub-step S21, the pre-trained stable diffusion model is loaded, and the original weight parameters of the U-Net network in the model are frozen. The U-Net network is the core denoising network of the stable diffusion model, which adopts an encoder-decoder structure and fuses feature information at different scales through skip connections. Freezing the original weight parameters can preserve the general visual prior knowledge learned by the pre-trained model and avoid catastrophic forgetting during fine-tuning with small samples.

[0069] In substep S22, trainable low-rank decomposition matrices are injected into the self-attention layer and cross-attention layer of the U-Net network, respectively, to the weight matrices of the query, key, and value. The self-attention layer is used to capture the spatial dependencies within image features, and the cross-attention layer is used to incorporate textual conditional information into the image generation process. Taking the query weight matrix as an example, the fine-tuned query vector is calculated according to the following formula:

[0070] ;

[0071] ;

[0072] ;

[0073] In the above formula, The calculated query vector; This is the fine-tuned query weight matrix; The input image features; This is the frozen original query weight matrix; The incremental matrix introduced for low-rank adaptation; Let be the first low-rank matrix with dimension . ; This is the second low-rank matrix with dimension . ; This represents the row dimension of the original weight matrix; The column dimensions of the original weight matrix; For low-rank numbers, the preferred value range is 16 to 32; This represents matrix multiplication. By decomposing the increment matrix into the product of two low-rank matrices, the number of parameters that need to be trained can be reduced from... Reduce to When the rank much smaller and In this way, the number of parameters can be reduced to less than one percent of the original model's number of parameters. The weight matrices of keys and values ​​are processed using the same low-rank decomposition method.

[0074] In sub-step S23, rare biological foreground images and their corresponding category text descriptions from the real target dataset are used as training samples. The parameters of the low-rank factorization matrix are optimized by minimizing the noise prediction loss to obtain the foreground image generation model. During training, the encoder of the variational autoencoder first compresses the input image into a latent variable, then randomly samples time steps in a uniform distribution, and adds Gaussian noise of the corresponding number of steps to the latent variable to obtain a noise latent variable. The U-Net network predicts the noise added to the noise latent variable under the joint guidance of text encoding and time steps. The parameters of the low-rank factorization matrix are optimized by minimizing the mean square error between the predicted noise and the real noise. The training parameters can be set to the U-Net learning rate. The magnitude of the text encoder's learning rate is [missing information]. The training batch size can be set to 2 to 4 based on the video memory capacity, the number of training epochs can be set to 5 to 10, and the training image size can be set to... Pixels. Because low-rank adaptive methods only require optimizing a small number of parameters, the training time for a single-class foreground image generation model can usually be controlled within tens of minutes.

[0075] To enhance the diversity of generated samples, this invention constructs hierarchical text prompts that include biological category, morphological description, lighting conditions, and shooting angle. The text prompts are constructed as follows: a text prompt equals the biological category description plus the morphological description plus the lighting condition description plus the shooting angle description. Specifically, the biological category description includes the name of rare biological categories; the morphological description includes the organism's color, posture, and size characteristics; the lighting condition description includes bright or dark environments; and the shooting angle description includes top-down, side-view, or front-view angles. By designing diverse combinations of text prompts, the biological color, lighting conditions, and shooting angle attributes of the generated samples can be guided and controlled, generating feature combinations not present in the original data, thereby expanding the diversity of the data.

[0076] Substeps S24 to S27 describe the specific process of generating synthetic foreground images of rare biological categories through an iterative denoising process. In substep S24, a random Gaussian noise matrix following a standard normal distribution is generated, and a variational autoencoder transforms the Gaussian random noise into latent variables. The dimension of the noise matrix matches the dimension of the latent space of the input Unet network.

[0077] In sub-step S25, a single-step denoising calculation is performed. At each time step... The U-Net network is based on the current noise latent variable. and text embedding Predict the noise in the current step and calculate the latent variables from the previous time step using the following formula:

[0078] ;

[0079] In the above formula, These are the latent variables corresponding to the previous time step, obtained through calculation. For the noise latent variable at the current time step; The noise scheduling parameters for the current time step. ; For the predefined first Step noise intensity parameters; From step 1 to step 2 The cumulative product of the noise scheduling parameters of each step; For the U-Net network based on the current noise latent variable Time step and text embedding Predicted noise; For the first The scaling factor for the randomness intensity of the step; This represents a random noise term that follows a standard normal distribution. The text embedding is obtained by encoding hierarchical text prompts using a text encoder. This represents the square root operation. Noise scheduling parameters. Typically, linear or cosine scheduling strategies are predefined and used as time steps... Increase, cumulative product The monotonically decreasing noise tends towards 0, causing the latent noise variable to gradually transform into pure random noise. The reverse denoising process, on the other hand, starts from pure random noise and gradually restores it to a latent variable containing semantic information. A random noise term is introduced. It can increase the diversity of generated images and avoid the generated results being too homogeneous.

[0080] In sub-step S26, from time step Begin by iteratively executing sub-step S25. Next, the initial noise latent variable Gradually convert to denoised latent variables Number of iterations The number of iterations is typically set to tens to hundreds. More iterations usually result in higher image quality, but the computation time also increases accordingly. The preferred setting is 20;

[0081] In sub-step S27, the denoised latent variables are... The decoder of the variational autoencoder is input to reconstruct the synthetic foreground image. The decoder of the variational autoencoder maps the latent variables in the low-dimensional latent space back to the high-dimensional pixel space to obtain the final synthetic foreground image. For each rare biological category, several synthetic foreground images can be generated using the trained foreground image generation model. The number of images can be adjusted according to data augmentation needs and computational resources.

[0082] Step S3 is the background fusion and layout control step. The purpose of this step is to naturally blend the synthetic foreground image generated in step S2 with the diverse deep-sea background, while simultaneously achieving precise control and automated annotation of the target spatial layout. Although the synthetic foreground image generated in step S2 can be directly used for image classification tasks, its application in object detection tasks is limited due to the lack of background and spatial context information. Object detection tasks not only need to identify biological categories but also need to locate the organisms' positions within the image; therefore, it is necessary to embed the foreground image into a realistic deep-sea background environment.

[0083] This invention employs a conditional control network to achieve background fusion and layout control. A conditional control network is a network structure that adds an additional conditional coding branch to a stable diffusion model. Its core idea is to freeze the U-Net weights of the original stable diffusion model to preserve its visual priors and to replicate its encoder as a trainable conditional coding branch. After the layout mask image is processed by this branch to extract multiple layers of features, each layer's features are mapped using a 1×1 zero convolution and then concatenated with the corresponding layer's features extracted by the original U-Net encoder along the channel dimension. This concatenated feature is then input into the original U-Net decoder to drive image generation, thereby achieving layout-controlled image rendering.

[0084] First, a stable diffusion model is trained using diverse deep-sea background images, including deep-sea sedimentary backgrounds, nodule mining area backgrounds, and basalt backgrounds. The training dataset should cover various bottom environments commonly encountered in deep-sea exploration to ensure that the generated synthetic scene images have sufficient background diversity.

[0085] Then, a layout mask is generated based on the size distribution of the bounding boxes in the real-world scene dataset. The layout mask generation process includes sub-steps S31 to S33. In sub-step S31, the width and height distribution of the bounding boxes for each rare organism category in the real-world scene dataset are statistically analyzed to determine the minimum width of the bounding boxes for each category. and maximum value and the minimum height and maximum value This statistical process ensures that the dimensions of the subsequently generated bounding boxes conform to the prior distribution of the actual exploration data, thus avoiding the generation of targets with unreasonable scales.

[0086] In sub-step S32, the position coordinates and size parameters of the bounding box are randomly sampled and generated as follows: Bounding box width. In the interval Uniform random sampling within the bounding box; bounding box height In the interval Uniform random sampling within the bounding box; x-coordinate of the top left corner of the bounding box In the interval Uniform random sampling within the bounding box; top left corner ordinate In the interval Uniform random sampling within the area. During the above sampling process, The width of the sampled bounding box, in pixels; The height of the sampled bounding box, in pixels; The x-coordinate of the top-left corner of the bounding box, in pixels; The ordinate of the top-left corner of the bounding box, in pixels; Width of the target image, in pixels; The height of the target image, in pixels; and These are the minimum and maximum values ​​of the bounding box width for this category, in pixels. and These are the minimum and maximum statistical values ​​of the bounding box height for this category, in pixels. This is achieved by limiting the sampling range of the position coordinates to... and This ensures that the generated bounding box does not exceed the image boundary.

[0087] In sub-step S33, a layout mask image with the same size as the target image is created, setting the pixel values ​​of the bounding box region to 1 and the pixel values ​​of the background region to 0. The layout mask image represents the spatial location information of the target in the form of a binary image, which is used to guide the conditional control network to generate or fuse the foreground target at a specified location.

[0088] Sub-steps S34 to S37 describe the specific process of fusing the synthesized foreground image with the deep-sea background using a conditional control network. In sub-step S34, the layout mask is input into the conditional encoding branch of the conditional control network to extract multi-layer layout conditional features; background cue words are input into the text encoder to obtain the background text embedding. The background cue words are used to describe the desired type of deep-sea background, such as a deep-sea sediment background, a nodule mining area background, or a basalt background.

[0089] In sub-step S35, the layout conditional features output from each layer of the conditional coding branch are mapped through a zero-convolutional layer and then concatenated with the features of the corresponding layer of the stable diffusion model U-Net encoder along the channel dimension. A zero-convolutional layer is a convolutional layer with weights and biases initialized to zero. Its function is to minimize the influence of conditional signals in the early stages of training and gradually learn appropriate feature fusion methods as training progresses, thereby ensuring training stability.

[0090] In sub-step S36, the synthesized foreground image is placed at the bounding box position specified by the layout mask. The U-Net decoder is driven by the stitched features and background text embeddings to perform denoising generation, achieving the generation of a deep-sea background and the natural fusion of the foreground and background. The conditional control network uses image rendering techniques to generate a deep-sea background environment that transitions naturally with the foreground while keeping the content of the foreground region unchanged, avoiding obvious boundary artifacts between the foreground and background. To alleviate the problem of abrupt edges, the conditional control network supports feathering the layout mask. That is, by transforming the hard boundary mask composed of 0 / 1 into a soft boundary mask with a gradual transition of 0-1, a smoother foreground-background boundary is obtained. During the generation process, the initial filling method of the mask region adopts a "fill" strategy, that is, pre-filling the background mask region with the blurred content of the foreground. Compared with the setting without filling preprocessing, this strategy can make the generated background exhibit higher consistency with the foreground in terms of lighting and tone. The masked padding was set to 32, meaning that the mask was extended outward by 32 pixels during conditional generation to participate in denoising, thereby enhancing the continuity of shadows and textures. The resolution of the synthesized image from the conditional control network was uniformly set to 1280×960, maintaining the same aspect ratio as the original image, and the resolution met the requirements for subsequent object detection.

[0091] In substep S37, the coordinate parameters of the bounding box in the layout mask are used. , , and The system automatically generates target location annotation files, resulting in a synthetic scene image and its corresponding annotation information. Since the position and size of the bounding boxes are determined during the layout mask generation stage, these parameters can be directly converted into the annotation format required for the object detection task, achieving automated annotation and greatly reducing the workload of manual annotation.

[0092] Step S4 is the augmentation dataset construction step. The purpose of this step is to perform quality control on the synthetic scene images generated in step S3 and merge them with the real scene dataset to construct the augmentation training dataset. In step S4, the synthetic scene images are first subjected to quality screening, removing images with false morphological features. The quality screening criteria are: retaining images that accurately represent key biological recognition features and have acceptable visual fidelity, and removing images with false morphological features. False morphological features include defects that may be generated by the generative model, such as abnormal biological structures, incorrect limb numbers, and disproportionate morphology. Quality screening can be performed through manual visual inspection or automatic screening based on image quality assessment metrics.

[0093] Next, the automatically generated bounding box annotations are refined. This refinement involves adjusting the boundaries of annotation boxes that do not closely align with the biological contour, ensuring a tight fit. Since small gaps exist between some foreground masks and the biological contour, this refinement further improves the quality of the annotation boxes, enhancing their localization accuracy for target detection tasks.

[0094] Finally, the quality-filtered and bounding box-refined synthetic scene images are merged with the real-world scene dataset to construct an augmented training dataset. This augmented training dataset contains both original real images and newly generated synthetic images. By increasing the number of samples from rare organism categories, the class imbalance problem in the dataset can be alleviated, allowing the model to perform sufficient iterative optimization on rare organism categories during training.

[0095] Step S5 is the object detection model training and inference step. The purpose of this step is to train the object detection model using the augmented training dataset and apply the trained model to the biological object detection task in deep-sea images. This invention preferably uses the RTDETR model as the object detection model. RTDETR is an end-to-end object detection model based on the Transformer architecture, which decouples the detection head into classification and regression branches and introduces Focal loss to assign higher loss weights to hard examples, giving it advantages in handling class imbalance problems.

[0096] Step S5 includes sub-steps S51 to S54. In sub-step S51, an RTDETR object detection model with ResNet-18 as the backbone network is constructed. ResNet-18 is a residual network with 18 layers, which solves the problem of training difficulties for deep networks by introducing residual connections. As a backbone network, it can effectively extract multi-scale feature information of images. The choice of backbone network can be adjusted according to the computational resources and detection accuracy requirements, or deeper network structures such as ResNet-50 can be used.

[0097] In substep S52, the weights of each component of the loss function are set. The bounding box regression loss optimizes the positional matching between the predicted and ground truth boxes, the classification loss optimizes the accuracy of class prediction, and the distribution focus loss addresses the problem of imbalanced sample difficulty. The bounding box regression loss gain can be set between 7 and 8, the classification loss gain between 0.4 and 0.6, and the distribution focus loss gain between 1 and 2.

[0098] In sub-step S53, the initial learning rate is set to... The model employs the AdamW optimizer, using an augmented training dataset as input to iteratively optimize model parameters, resulting in a trained object detection model. The AdamW optimizer is an optimization algorithm that combines momentum and adaptive learning rate, and improves upon the Adam optimizer by addressing weight decay; it is widely used in deep learning model training. During training, learning rate decay strategies, such as cosine annealing, can be used to gradually reduce the learning rate in the later stages of training to achieve more stable convergence.

[0099] In sub-step S54, the deep-sea image to be detected is input into the trained target detection model, which outputs the category prediction results and bounding box coordinates of each biological target in the image. The target detection model performs forward inference on the input image, outputting the category label, confidence score, and bounding box coordinates of each detected target. The bounding box coordinates are usually represented as the coordinates of the top-left corner and width and height, or as the coordinates of the top-left corner and bottom-right corner. By setting a confidence threshold, low-confidence detection results can be filtered out, and overlapping redundant detection boxes can be removed using a non-maximum suppression algorithm, ultimately obtaining the category and location information of each biological target in each image.

[0100] This invention provides an effective technical solution to address the challenges of high data acquisition costs, scarce samples of rare organisms, and long-tailed distribution effects in deep-sea biological data augmentation by introducing a stable diffusion model. The low-rank adaptive method enables fine-tuning of large-scale pre-trained models under small sample conditions, while the conditional control network ensures controllable image layout and natural foreground-background fusion. The automated annotation mechanism reduces the workload of manual annotation. By expanding the number of training samples for rare organisms, the target detection model can learn more comprehensively from these categories, thereby improving its detection performance and providing technical support for deep-sea ecological environment monitoring and the planning and management of ecological protection zones.

[0101] Example 1: Foreground image generation and background fusion; This example uses a dataset of deep-sea benthic organism images collected by an underwater towed vehicle (OTV) and an underwater autonomous vehicle (AUV) in the Beijing Pioneer Mining Area of ​​Chinese polymetallic nodules in the western Pacific Ocean as an example to illustrate the specific implementation process of step S1 data preparation, step S2 foreground image generation, and step S3 background fusion and layout control in the method of the present invention.

[0102] In step S1, the data preparation step, the OTV dataset contains 2499 benthic organism images, each with a size of 4000×3000 pixels, of which 1973 are used for training and 526 for testing. The AUV dataset contains 3680 benthic organism images, each with a size of 4864 pixels. The datasets contain 3648 pixels each, with 2928 images used for training and 752 for testing. The backgrounds of the images in both datasets encompass complex environments including deep-sea sediment bottoms, areas with dense nodules, areas with sparse nodules, and basaltic basements. The two datasets contain a total of 16 biological categories, including sea cucumbers, sponges, shrimp, sea anemones, polychaetes, fish, isopods, starfish, brittle stars, crinoids, corals, bryozoans, sea urchins, hydroids, enterobacteria, and octopuses. After analyzing the sample size for each biological category, significant class imbalance was observed in all datasets. In this embodiment, the seven biological categories with the fewest samples were identified as rare biological categories: crinoids, corals, bryozoans, sea urchins, hydroids, enterobacteria, and octopuses. Based on the bounding box coordinates, foreground images of the seven rare organism categories were cropped from the original images. A total of 175 foreground images of rare organism categories were extracted from the two datasets to construct the real target dataset, including 48 images of crinoids, 26 images of corals, 28 images of bryozoans, 26 images of sea urchins, 20 images of hydroids, 17 images of enterobacteria, and 10 images of octopuses. Since the number of training samples for rare organism categories is extremely limited, with only 10 training images for the octopus category, this verifies the severe long-tail distribution effect in deep-sea organism datasets. It is this data scarcity problem that prompts this invention to adopt a low-rank adaptive method for efficient parameter fine-tuning.

[0103] In step S2, the foreground image generation step, this embodiment trains seven low-rank adaptive models for the seven rare biological categories mentioned above. In sub-step S21, the pre-trained stablediffusionxl base model is loaded. This model has been pre-trained on a large-scale image dataset with a resolution of 1024×1024, and the original weight parameters of the U-Net network in the model are frozen. In sub-step S22, a trainable low-rank decomposition matrix is ​​injected into the self-attention layer and cross-attention layer of the U-Net network, with the low-rank number r set to 32 and the network alpha value set to 16.

[0104] In sub-step S23, rare biological foreground images and their corresponding category text descriptions from the real target dataset are used as training samples for fine-tuning. The training parameter settings for the low-rank adaptive model used in this embodiment are shown in Table 1.

[0105] Table 1. Training parameter settings for low-rank adaptive models

[0106]

[0107] The specific training configuration parameters of the method of this invention can be found in Table 1. The U-Net learning rate is set to 1×10. -4 The text encoder learning rate is set to 1×10. -5 There is an order of magnitude difference between the two. The principle behind this parameter setting is that the U-Net network is responsible for denoising and reconstructing image features, requiring a large learning rate to quickly adapt to new biological category features; while the text encoder has been fully pre-trained on large-scale text-image pairs, and its semantic understanding ability is relatively complete. Using a smaller learning rate allows for fine-tuning while retaining its general semantic encoding capabilities. The optimizer uses AdamW8bit, which maintains the adaptive learning rate advantage of the Adam optimizer while reducing memory usage through 8-bit quantization, making it possible to fine-tune large models on consumer-grade graphics cards. The learning rate scheduler uses the cosine_with_restarts strategy, which periodically adjusts the learning rate using a cosine function. In the later stages of training, the learning rate is gradually reduced to achieve more stable convergence, and periodic restarts help the model escape local optima. The training batch size is set to 2, the number of training epochs is set to 6, and the training image size is set to 512×512 pixels. The low-rank adaptive model is trained in a Linux environment with 16GB of GPU memory, and the average training time for each category of the low-rank adaptive model is approximately 20 minutes. As can be concluded from the parameter settings in Table 1, the method of this invention can complete the training on ordinary consumer-grade graphics cards without the need for enterprise-grade computing resources.

[0108] In sub-steps S24 to S27, the trained foreground image generation model generates 200 synthetic foreground images for each rare organism category, generating a total of 1400 images across 7 categories, with an average inference time of approximately 2 seconds per image. To enhance the diversity of generated samples, this embodiment constructs hierarchical text prompts containing organism category, morphological description, lighting conditions, and shooting angle. For example, prompts such as Darkenvironment, lateralaspect, and Differentorientations are used to guide and control the attributes of the generated samples, such as organism color, lighting conditions, and shooting angle.

[0109] The foreground image generation result in this embodiment is as follows: Figure 5 As shown. Figure 5 (a) Examples of real images for 7 rare creature categories. Figure 5 (b) shows generated image examples for seven rare creature categories, from left to right: crinoids, corals, bryozoans, sea urchins, hydroids, enterobacteria, and octopuses. Figure 5 This allows for a direct comparison of the visual effects of real and generated images, revealing the degree of similarity between the generated image and the real image in terms of morphological features, texture details, and contour sharpness. From... Figure 5 It can be concluded that the generated images successfully captured the basic identifying features of deep-sea organisms, including the feathery tentacles of crinoids, the branching structure of corals, the colony morphology of bryozoans, the spherical shell and spines of sea urchins, the tentacle structure of hydroids, the worm morphology of enterobacteria, and the brachial structure of octopuses, with clear outlines and rich details. This result verifies that the technique of fine-tuning the stable diffusion model using the low-rank adaptive method in this invention is effective. Even with only a few dozen training samples, the model can still learn the key visual features of various organisms and generate realistic synthetic images. The pre-trained stable diffusion model already possesses powerful general image generation capabilities and rich visual prior knowledge. The low-rank adaptive method, by injecting a small number of trainable parameters, retains the general capabilities of the pre-trained model while enabling the model to learn the domain-specific features of deep-sea organisms, thereby achieving effective transfer learning under small sample conditions.

[0110] This embodiment further verifies the effect of hierarchical text prompts on improving the diversity of generated images, as shown in the following results. Figure 6 As shown. Figure 6 (a) is a deep-sea cephalopod sample obtained from a real dataset. Figure 6 (b) is an example of a deep-sea cephalopod sample generated based on diverse text prompts. From Figure 6 As can be seen, the original dataset only recorded a limited number of octopus morphologies, including a white octopus in a contracted state and a yellow octopus in an umbrella-like open state, while the synthetic data generated a richer combination of morphologies. Figure 6 It can be concluded that the generative model synthesizes feature combinations not present in the original data, such as a white octopus in an umbrella-like open state, creating new combinations of postures and forms. This result verifies the effectiveness of the hierarchical text prompt design technique employed in this invention. By controlling the combination of attributes such as biological color, lighting conditions, and shooting angle, the diversity of data can be systematically expanded, generating new feature combinations not present in the training set. The cross-attention mechanism of the stable diffusion model integrates the semantic information of the text prompts into the image generation process. Different combinations of text prompts guide the model to generate images with different attributes, thereby overcoming the limitations of the original dataset and providing richer training samples for the object detection model.

[0111] In step S3, background fusion and layout control, this embodiment uses diverse deep-sea background images, including deep-sea sedimentary backgrounds, deep-sea nodule mining area backgrounds, and deep-sea basalt backgrounds, to train the image generation network. The conditional control network is initialized using common pre-trained weights and remains frozen during subsequent training. Only the underlying stable diffusion model is fine-tuned to learn to generate background images of a specified type under the additional guidance provided by the control network. In sub-steps S31 to S33, the width and height distribution of the bounding boxes for each rare organism category in the real-world scene dataset are statistically analyzed to determine the sampling interval for the bounding box size of each category. The position coordinates of the bounding boxes are randomly generated within the target image size range to generate a layout mask image. In sub-steps S34 to S37, the layout mask, the synthesized foreground image, and the background cue words are used as conditional inputs. The conditional control network fuses the synthesized foreground image with the deep-sea background and automatically generates target location labels based on the bounding box coordinates in the layout mask. The inference parameter settings of the conditional control model used in this embodiment are shown in Table 2.

[0112] Table 2. Conditional Control Model Inference Parameter Settings

[0113]

[0114] The background blending and layout control results in this embodiment are as follows: Figure 7 As shown. Figure 7 Examples of deep-sea exploration images generated based on a stable diffusion model and layout control are shown. From Figure 7 The visual effects of synthetic scene images generated by conditional control networks can be obtained, including the degree of integration between foreground creatures and the background environment, the diversity of the background, and the overall realism of the image. Figure 7The following conclusions can be drawn: First, this method can generate deep-sea exploration images with controllable spatial layout, and biological targets are accurately placed at the positions specified by the layout mask, verifying the effectiveness of the layout control function. Second, the organisms are seamlessly integrated with the diverse seabed exploration environment, with a natural transition between the foreground and background and no obvious boundary artifacts, verifying the ability of the conditional control network in image rendering. Third, the background of the synthesized image is consistent with the style of real deep-sea exploration data in terms of lighting, shadows, and textures, covering various environments such as deep-sea sediment bottom, nodule-dense areas, sparse nodule areas, and basalt basement. This result verifies that the technical means of background fusion and layout control using the conditional control network in this invention is effective. The conditional control network retains its visual prior by freezing the weights of the original stable diffusion model, while adding a conditional encoding branch to process the layout mask signal. The introduction of zero convolutional layers ensures that the influence of the conditional signal is minimized in the early stage of training, enabling the model to progressively learn the fusion mode of the foreground and background. The layout mask explicitly specifies the foreground and background regions in the form of a binary image. Based on this, the conditional control network generates a background environment that transitions naturally while keeping the foreground content unchanged. At the same time, the bounding box coordinates can be directly converted into the object detection annotation format to achieve automated annotation.

[0115] Example 2: Quality Assessment of Synthetic Images; This example assesses the quality of the synthetic images generated in Example 1, using multiple image quality assessment metrics such as FID, KID, IS, CMMD, and CLIP scores to quantitatively analyze the quality of synthetic images from the underwater towed vehicle dataset and the underwater autonomous vehicle dataset.

[0116] The evaluation metrics used in this embodiment include: the FID metric, which extracts the feature vectors output by the last pooling layer of the Inception-V3 network from the synthetic and real images, and measures the similarity between the two by calculating the Fréchet distance between the feature distributions; the lower the value, the closer the distributions are. The KID metric removes the normal distribution assumption in FID and measures the difference between the two distributions by calculating the MMD distance between the features. The IS metric measures the sharpness and diversity of the generated image set by calculating the KL divergence between the predicted class of each image and the overall distribution; the higher the value, the better the quality. The CMMD metric and CLIP score measure the semantic matching degree by measuring the MMD distance and cosine similarity of the CLIP embeddings between the generated and real images, respectively. The image quality evaluation results of the synthetic dataset in this embodiment are shown in Table 3.

[0117] Table 3. Quantitative evaluation results of image quality in the synthetic dataset.

[0118]

[0119] Table 3 shows a comparison of quantitative indicators of the synthesized images from the two datasets across multiple image quality assessment dimensions. The following conclusions can be drawn from Table 3: First, the KID index of the OTV synthetic dataset is 0.028, which is relatively low, indicating that the generated images and real images have a high degree of matching in feature distribution, suggesting that the synthetic images are statistically close to the real images. Second, the CMMD index of the OTV synthetic dataset is 0.73 and the CLIP score is 0.71, indicating that the synthetic images and real images have strong semantic correlation and good semantic consistency, suggesting that the synthetic images accurately express the target biological category in terms of semantic content. Third, the IS index of the OTV synthetic dataset is 4.97, which is an improvement compared to 3.77 in the original rare biological dataset, indicating that the generative model effectively expands the diversity of the data. Fourth, compared with the OTV dataset, the FID index of the AUV synthetic dataset is 160.17 and the KID index is 0.053, both higher than the OTV synthetic dataset, indicating that there is a larger feature distribution gap between the synthetic images and real images in the AUV dataset. This may be related to the inherent characteristics of the AUV dataset, such as higher imaging height and a higher proportion of small targets. These quantitative evaluation results validate the effectiveness of the technical solution of this invention from multiple dimensions, demonstrating that the stable diffusion model based on low-rank adaptive fine-tuning can generate high-quality synthetic images that meet the requirements in terms of feature distribution, semantic content, and diversity. This embodiment further evaluates the quality of synthetic images for each rare biological category in detail, and the results are shown in Table 4.

[0120] Table 4. Quantitative evaluation results of data volume and image quality of synthetic datasets for different biological categories.

[0121]

[0122] Table 4 shows the actual amount of data used for training the low-rank adaptive model for each rare organism category, as well as the detailed evaluation results of each category on various image quality metrics. The following conclusions can be drawn from Table 4: First, the number of training samples for each rare organism category is extremely limited. The octopus category, which has the fewest training images in the OTV dataset, has only 5, and the coral category, which has the fewest in the AUV dataset, has only 4. This verifies the severe long-tail distribution effect in the deep-sea organism dataset and also verifies the necessity of using a low-rank adaptive method for small-sample fine-tuning in this invention. Second, the crinoid has the best FID index of 157.5 on the OTV dataset, indicating that the synthetic crinoid image has the closest feature distribution to the real image. Third, the coral has the highest IS index of 3.10 on the OTV dataset, indicating that the synthetic coral image has good clarity and diversity. Fourth, the octopus has the best CMMD index of 0.70 and CLIP score of 0.81, both of which are the best values ​​among all categories in the OTV dataset, indicating that the synthetic octopus image has the best semantic consistency. Fifth, the hydroid has the highest FID index of 329.5 on the OTV dataset, indicating that there is a significant difference in feature distribution between the synthetic hydroid image and the real image. Table 3 further reveals that the synthesis results differ among different biological categories. Biological categories with relatively clear morphological structures and stable features, such as crinoids, corals, and octopuses, show better synthesis results, while those with semi-transparent morphology, small size, and complex features, such as hydras, show relatively weaker synthesis results. The mechanism behind this phenomenon is that the stable diffusion model tends to learn and generate images with clear structures during pre-training. For semi-transparent, low-contrast, and morphologically variable targets, the model struggles to fully capture their subtle features, leading to a domain gap between the synthesized and real images. This analysis provides an explanation for the differentiated performance of target detection in subsequent embodiments.

[0123] Example 3: Object Detection Model Training and Performance Comparison; This example, based on the synthetic images generated in Example 1, verifies the construction of the augmented dataset in step S4 and the training and inference of the object detection model in step S5, and compares it with traditional data augmentation methods to verify the advantages of the present invention over traditional data augmentation methods. In step S4, the synthetic scene images generated in Example 1 undergo quality screening. Images with generated false features are removed through manual inspection. The visual fidelity pass rate of each type of generated image exceeds 80%, meaning that most images can accurately represent the key features of organisms. Among the few false morphological features, the double-tailed structure appearing in intestinal gills is the most significant morphological artifact. The automatically generated bounding boxes are refined to ensure that the bounding boxes closely fit the organism's outline. The synthetic scene images after quality screening and bounding box refinement are merged with the real scene dataset. Based on the original real dataset, approximately 700 new training images are added for the 7 rare categories to construct the augmented training dataset.

[0124] In step S5, the object detection model training and inference step, this embodiment uses the RTDETR model as the object detection model and ResNet-18 as the backbone network. In sub-step S52, the bounding box regression loss gain is set to 7.5, the classification loss gain to 0.5, and the distribution focus loss gain to 1.5. In sub-step S53, the initial learning rate is set to 0.0001, and the AdamW optimizer is used to iteratively optimize the model parameters using the augmented training dataset as input.

[0125] The comparison results of the training process in this embodiment are as follows: Figure 4 As shown. Figure 4 The diagram illustrates a comparison between training using synthetic data and training using only real data. Subplot (a) shows the results for the OTV dataset, and subplot (b) shows the results for the AUV dataset. Figure 4 (a) The subgraph shows a comparison of the training curves of the two training strategies on the OTV dataset, including information such as the convergence speed of the loss function and the trend of changes in accuracy metrics. From Figure 4 (a) Subgraph shows that, firstly, the model trained with the augmented dataset converges more smoothly and with less fluctuation during training, indicating that the augmented dataset makes the training process of the object detection model relatively more stable; secondly, the model trained with the augmented dataset can converge to a better performance level faster, indicating that the synthetic data effectively compensates for the lack of training samples for rare categories, enabling the model to learn rare categories more fully. Figure 4(b) The subgraph shows a comparison of the training curves of the two training strategies on the AUV dataset. The results indicate that the model trained using the augmented dataset also exhibits a more stable training process and better convergence performance on the AUV dataset, verifying the universality of the method in this invention across different data sources. This result verifies that the technical means of improving object detection performance using data augmentation strategies in this invention is effective. The training process of a deep learning model is essentially a parameter optimization process based on gradient descent. When there is a severe shortage of rare class samples in the training data, the contribution of these classes to the loss function is overwhelmed by the head class, causing the model parameters to tilt towards the head class and making it impossible to effectively optimize the rare classes. By increasing the number of training samples of rare classes, the contribution ratio of these classes in the loss function can be increased, enabling the model to learn the discriminative features of rare classes, thereby improving the detection capability of rare classes.

[0126] This embodiment further compares the detection performance differences between the method of the present invention and traditional data augmentation methods, and the results are shown in Table 5.

[0127] Table 5 Comparison of the improvement effects of different data augmentation methods on the object detection model on the OTV dataset.

[0128]

[0129] Table 5 shows the performance comparison of different data augmentation methods under the same training data volume, including multiple metrics such as precision, recall, F1 score, mAP50 index, and mAP50-95 index. The following conclusions can be drawn from Table 5: First, the baseline model using only real images achieves precision, recall, mAP50 index, and mAP50-95 index of 76.5%, 70.6%, 71.6%, and 44.8%, respectively. This baseline performance reflects the upper limit of the object detection model's detection capability under class imbalance conditions. Second, the model trained using the method of this invention, combining real and generated images, achieves precision, recall, F1 score, mAP50 index, and mAP50-95 index of 81.2%, 78.6%, 79.3%, 78.4%, and 47.0%, respectively, representing improvements of 4.7%, 8.0%, 7.9%, 6.8%, and 2% compared to the baseline model. The mAP50 index of the method in this invention is 78.4%, which is significantly improved across all metrics. Third, compared with traditional data augmentation methods, the mAP50 index is 71.8% for fuzzy augmentation, 74.7% for scaling augmentation, 72.6% for shearing augmentation, 73.7% for saturation augmentation, 74.9% for a combination of scaling and shearing augmentation, and 72.6% for a combination of scaling, shearing, and saturation augmentation. Under the same augmented data volume, this method consistently outperforms various traditional data augmentation operations and their combinations, with an improvement of 3.5% to 6.6%. Traditional data augmentation methods transform the original image through geometric and photometric transformations. Essentially, they perform limited deformation operations on the original image, and the image transformation space is limited by the feature distribution of the original data, making it difficult to generate new feature combinations that do not exist in the original data. In contrast, data augmentation methods based on stable diffusion models sample from the probability distribution to generate new images. They can synthesize combinations of biological poses, lighting conditions, and background environments that do not appear in the original data, fundamentally expanding the feature space of the training data. Therefore, they are more advantageous in improving the generalization ability of the model.

[0130] The comparison results of the detection performance of each biological category in this embodiment are as follows: Figure 3 As shown. Figure 3 The left figure shows a comparison of the mAP50 metric on the OTV dataset. Figure 3 The right figure shows a comparison of accuracy metrics on the AUV dataset. From Figure 3 The left figure shows the changes in detection accuracy for each biological category on the OTV dataset before and after using augmented data. From Figure 3The left figure leads to the following conclusions: First, while the detection accuracy of the enhanced tail category was significantly improved, the accuracy of the head category was maintained, indicating that the data augmentation strategy improved the detection performance of rare categories without negatively impacting common categories. Second, the mAP50 indexes of sea cucumbers, sponges, sea anemones, polychaetes, fish, isopods, starfish, and brittle stars in the head category remained basically unchanged, while the mAP50 index of shrimp increased from 64.1% to 75.5%, indicating that the data augmentation strategy has a positive effect on overall detection performance. Third, the detection performance improvement of rare organisms was much greater than that of the head category, verifying that the technical approach of data augmentation for rare categories in this invention is correct. Figure 3 The figure on the right shows the changes in the accuracy of each biological category on the AUV dataset before and after using augmented data. The results show that the data augmentation strategy can also improve the detection performance of rare biological categories on the AUV dataset.

[0131] Example 4: Detection Performance Gain Analysis and Practical Application; This example provides a detailed analysis of the detection performance gain of the method of the present invention on different datasets and different biological categories, and demonstrates the application effect of the trained target detection model in a real deep-sea exploration scenario. Detailed results of the detection performance gain for each rare biological category in this example are shown in Table 6.

[0132] Table 6. Detection performance gains of stable diffusion data enhancement strategies across different biological categories.

[0133]

[0134] Table 6 details the changes in detection performance of each rare organism category before and after data augmentation on the OTV and AUV datasets, including changes in precision, recall, and AP50 scores, and the magnitude of these gains. Table 4 shows that on the OTV dataset, the AP50 score for crinoids improved from 38.5% to 59.9%, an increase of 21.4%; for bryozoans, it improved from 51.4% to 80.1%, an increase of 28.7%; and for octopuses, it improved from 62.5% to 99.5%, an increase of 37.0%. These three categories showed particularly significant performance improvements. The recall score for octopuses increased from 25.0% to 100.0%, an increase of 75.0%. This means that before augmentation, the model could only detect a quarter of the octopus targets, while after augmentation, the model could detect almost all of them, validating the effectiveness of data augmentation in improving the problem of missed detections in rare categories. When there is a severe shortage of samples for a certain category in the training data, the model struggles to learn the discriminative features of that category and tends to predict it as a similar category with more samples during inference, resulting in a high false negative rate. By increasing the number of training samples for that category, the model can learn richer intra-class feature variations, thereby improving the recall rate. The hydra category showed a decline in various metrics on the OTV dataset, with precision decreasing by 14.7%, recall by 11.1%, and AP50 by 7.1%. Combined with the analysis in Table 3 of Example 2, the hydra's FID index on the OTV dataset is 329.5, the highest among all categories, indicating a significant difference in feature distribution between the synthetic and real hydra images. This phenomenon suggests that if there is a significant domain difference between the feature distribution of synthetic data and real data, it is difficult to bring effective performance gains to downstream detection tasks; instead, it may introduce noise and lead to performance degradation. This analysis reveals the applicability boundaries of the method of this invention: for categories with high-quality synthetic images, the data augmentation effect is significant; for categories with a large domain difference between synthetic and real images, further improvements to the generative model are needed to narrow the domain gap. The enhancement effects for the same category vary across different datasets. Taking hydra as an example, performance degrades on the OTV dataset, while on the AUV dataset, precision increases from 19.1% to 84.7%, a 65.6% improvement; recall increases from 42.9% to 71.4%, a 28.5% improvement; and the AP50 score increases from 32.5% to 82.3%, a 49.77% improvement, demonstrating a highly significant performance boost. Table 4 shows that the FID score for hydra on the AUV dataset is 303.7, which, although high, represents the largest improvement in detection performance. This phenomenon indicates that the relationship between synthetic image quality metrics and downstream detection task performance is not a simple linear one; the improvement in detection performance is also influenced by various factors such as the baseline performance of the original dataset, inter-category competition, and data distribution characteristics.When the baseline performance of a certain category in the original dataset is extremely low, even with some domain disparity in the synthesized images, the additional training samples they provide can still lead to performance improvements. For enterobacteria, relatively stable performance improvements were achieved on both datasets: a 10.6% improvement in AP50 on the OTV dataset and a 16.8% improvement on the AUV dataset, with precision improvements of 26.0% and 49.08% on the two datasets, respectively. This result validates the effectiveness and stability of the proposed method on different datasets.

[0135] This embodiment applies the trained target detection model to a real-world deep-sea exploration scenario, and the detection results are as follows: Figure 2 As shown. Figure 2 This document presents an example of the results of applying an enhanced target detection model to biological detection in the BPC mining area of ​​the western Pacific Ocean. Figure 2 (a) shows the results corresponding to the OTV exploration data. Figure 2 (b) shows the corresponding results of AUV exploration data.

[0136] from Figure 2 (a) It allows us to determine the performance of the trained target detection model on actual exploration images from an underwater towed vehicle, including the accuracy of bounding box positions, the correctness of category predictions, and the ability to identify different biological categories. Figure 2 (a) It can be concluded that the model can accurately identify the categories of various deep-sea benthic organisms in the image and locate their positions. The detection box is closely attached to the biological target, and the category label is correct, which verifies the effectiveness of the method of the present invention on actual OTV exploration data.

[0137] from Figure 2 (b) It allows us to determine the detection performance of the trained target detection model on actual exploration images from underwater autonomous vehicles. Figure 2 (a) In comparison, Figure 2 (b) shows that the AUV exploration data exhibits significant differences in imaging quality, target scale, and background complexity, but the model can still adapt to the unique imaging conditions of the AUV dataset and accurately detect deep-sea benthic organisms in the images. Figure 2 (b) It can be concluded that the method of the present invention has good applicability on different data acquisition platforms, and the trained model can generalize to deep-sea biological image data from different sources.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting rare deep-sea organisms based on a stable diffusion model, characterized in that, Includes the following steps: S1. Data preparation steps: Count the number of samples for each biological category from the real scene dataset of deep-sea benthic organisms, identify the biological categories with a sample number lower than a preset threshold as rare biological categories, and crop the foreground area image of the rare biological category from the original image according to the coordinates of the labeled box to construct the real target dataset. S2. Foreground image generation steps: For each rare organism category, a low-rank adaptive method is used to fine-tune the pre-trained stable diffusion model to obtain a foreground image generation model; a hierarchical text prompt word containing organism category, morphological description, lighting conditions and shooting angle is constructed to drive the foreground image generation model to generate synthetic foreground images of rare organism categories through an iterative denoising process. S3. Background fusion and layout control steps: A stable diffusion model is fine-tuned using diverse deep-sea background images, including deep-sea sedimentary backgrounds, nodule mining area backgrounds, and basalt backgrounds, to obtain a background image generation model; the conditional control network is initialized using general pre-trained weights and kept frozen during subsequent training; only the underlying stable diffusion model is fine-tuned to enable it to learn to generate background images of a specified type under the additional guidance provided by the control network; a layout mask is generated based on the size distribution of the bounding boxes in the real-world scene dataset. Using the layout mask, the composite foreground image, and the background cue words as input conditions, the conditional control network generates the composite foreground image by drawing it based on the layout mask. The composite foreground image is then merged with the deep-sea background, and the target position is automatically generated based on the bounding box coordinates in the layout mask to obtain the composite scene image. S4. Enhanced dataset construction steps: The synthetic scene images are quality-screened to remove images with false morphological features, the automatically generated bounding box annotations are refined, and the screened and refined synthetic scene images are merged with the real scene dataset to construct the enhanced training dataset. S5. Target detection model training and inference steps: Train the target detection model using the enhanced training dataset, and use the trained target detection model to detect biological targets in the deep-sea image to be detected, outputting biological category and location information.

2. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, In step S2, a low-rank adaptive method is used to fine-tune the pre-trained stable diffusion model, which includes the following sub-steps: S21. Model initialization: Load the pre-trained stable diffusion model and freeze the original weight parameters of the U-Net network in the model. S22. Low-rank matrix injection: In the self-attention layer and cross-attention layer of the U-Net network, trainable low-rank decomposition matrices are injected into the weight matrices of the query, key, and value, respectively; the fine-tuned query vector is calculated according to the following formula: ； ； ； in, The calculated query vector; This is the fine-tuned query weight matrix; The input image features; This is the frozen original query weight matrix; The incremental matrix introduced for low-rank adaptation; Let be the first low-rank matrix with dimension . ; This is the second low-rank matrix with dimension . ; This represents the row dimension of the original weight matrix; The column dimensions of the original weight matrix; It is a low-rank number, ranging from 16 to 32, and its specific value is determined according to the complexity of the image to be generated; This represents matrix multiplication. S23. Model Fine-tuning Training: Using rare biological foreground images and their corresponding category text descriptions from the real target dataset as training samples, the parameters of the low-rank factorization matrix are optimized by minimizing the noise prediction loss to obtain the foreground image generation model; the training parameters are set as follows: the U-Net learning rate is... The text encoder learning rate is The training batch size is 2, the number of training epochs is 6, and the training image size is [missing information]. Pixel.

3. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 2, characterized in that, In step S2, a synthetic foreground image of rare biological categories is generated through an iterative denoising process, including the following sub-steps: S24. Initial Noise Generation: Generate a random Gaussian noise matrix that follows a standard normal distribution. Input the variational autoencoder of the stable diffusion model to transform the Gaussian random noise into initial noise latent variables. ; S25. Single-step noise reduction calculation: at each time step The U-Net network is based on the current noise latent variable. and text embedding Predict the noise in the current step and calculate the latent variables from the previous time step using the following formula: ； in, These are the latent variables corresponding to the previous time step, obtained through calculation. For the noise latent variable at the current time step; The noise scheduling parameters for the current time step. ; For the predefined first Step noise intensity parameters; From step 1 to step 2 The cumulative product of the noise scheduling parameters of each step; For the U-Net network based on the current noise latent variable Time step and text embedding Predicted noise; For the first The scaling factor for the randomness intensity of the step; This represents a random noise term that follows a standard normal distribution. The text embedding is obtained by encoding hierarchical text prompts using a text encoder. S26. Iterative Denoising: From Time Step Begin, iteratively execute step S25 for a total of Next, the initial noise latent variable Gradually convert to denoised latent variables ; S27. Image Reconstruction: Denoising Latent Variables The decoder of the variational autoencoder is input to reconstruct the synthetic foreground image.

4. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, In step S3, a layout mask is generated based on the size distribution of the bounding boxes in the real scene dataset, including the following sub-steps: S31. Size Range Statistics: Statistically analyze the width and height distribution of bounding boxes for each rare organism category in the real-world dataset, and determine the minimum width of the bounding box for each category. and maximum value and the minimum height and maximum value ; S32. Bounding box parameter sampling: The position coordinates and size parameters of the bounding box are randomly sampled and generated in the following manner: bounding box width In the interval Uniform random sampling within the bounding box; bounding box height In the interval Uniform random sampling within the bounding box; x-coordinate of the top left corner of the bounding box In the interval Uniform random sampling within the bounding box; top left corner ordinate In the interval Uniform random sampling within the area; in, The width of the bounding box obtained from the sampling; The height of the bounding box obtained from sampling; The x-coordinate of the top-left corner of the bounding box; The ordinate of the top-left corner of the bounding box; The width of the target image; The height of the target image; and These are the minimum and maximum statistical values ​​for the bounding box width of this category, respectively; and These are the minimum and maximum values ​​of the bounding box height for this category, respectively. S33. Mask Image Generation: Create a layout mask image with the same size as the target image, set the pixel value of the bounding box region to 1, and set the pixel value of the background region to 0.

5. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, In step S3, the synthesized foreground image is fused with the deep-sea background using a conditional control network, including the following sub-steps: S34. Conditional Encoding: Input the layout mask into the conditional encoding branch of the conditional control network to extract multi-layer layout conditional features; Input the background cue words into the text encoder to obtain the embedded background text; S35, Feature Fusion: The layout condition features output from each layer of the conditional coding branch are mapped through a zero convolutional layer and then concatenated with the features of the corresponding layer of the stable diffusion model U-Net encoder in the channel dimension. S36. Background generation and fusion: The synthesized foreground image is placed at the bounding box position specified by the layout mask. The U-Net decoder is driven by the spliced ​​features and background text embedding to generate noise, thereby realizing the generation of deep-sea background and the natural fusion of foreground and background. S37. Automatic annotation: Based on the coordinate parameters of the bounding box in the layout mask. , , and It automatically generates target location annotation files, resulting in a composite scene image and its corresponding annotation information.

6. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, In step S5, the target detection model adopts the RTDETR model, and the training and inference process includes the following sub-steps: S51. Model Construction: Construct an RTDETR object detection model with ResNet-18 as the backbone network; S52. Loss function configuration: Set the bounding box regression loss gain to 7.5, the classification loss gain to 0.5, and the distribution focus loss gain to 1.

5. S53. Model Training: Set the initial learning rate to 0.0001, use the AdamW optimizer, and use the augmented training dataset as input to iteratively optimize the model parameters to obtain the trained object detection model. S54, Target Detection Inference: Input the deep-sea image to be detected into the trained target detection model, and output the category prediction results and bounding box position coordinates of each biological target in the image.

7. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, The rare organism category includes one or more of the following: crinoids, corals, bryozoans, sea urchins, hydras, enterobacteria, and octopuses.

8. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, In step S2, the hierarchical text prompts are constructed as follows: Text prompts = description of biological category + morphological description + description of lighting conditions + description of shooting angle; The description of the organism category includes the name of the rare organism category; the description of the morphology includes the color, posture and size characteristics of the organism; the description of the lighting conditions includes whether it is a bright or dark environment; and the description of the shooting angle includes whether it is a top view, a side view or a front view.

9. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, In step S4, the quality screening criteria are: retaining images that can accurately represent key biological identification features and have qualified visual fidelity, and removing images with false morphological features; the refinement operation is: adjusting the boundaries of the annotation boxes that fail to closely fit the biological contours so that the annotation boxes closely fit the biological contours.

10. The method for detecting rare deep-sea organisms based on a stable diffusion model according to claim 1, characterized in that, The images in the real-world scene dataset are taken from images of benthic organisms in deep-sea mining areas by underwater towed vehicles or autonomous underwater vehicles; the deep-sea background includes one or more of the following: deep-sea sedimentary substrate, nodule-dense areas, sparse nodule areas, and basalt basement.

Images