Underwater scene generation method based on multi-modal fusion and label guiding prompt and related equipment

Abstract

The application discloses a method for generating an underwater scene based on multi-modal fusion and label guidance and related equipment, and the method comprises the following steps: obtaining an underwater image and constructing a multi-modal data representation; based on a semantic graph, a depth map and an edge map, a multi-modal residual condition fusion module is used to sequentially perform independent coding, spatial channel collaborative attention enhancement and semantic dominant residual fusion to obtain a fusion condition feature; based on the underwater image, water type division is performed, and then an enhanced text prompt is constructed in combination with the semantic graph and an initial text description; based on the fusion condition feature and the enhanced text prompt, a latent space diffusion model is used to generate a synthetic underwater image. The multi-modal residual condition fusion module is used to introduce the semantic graph, the depth map and the edge map, and the latent space diffusion model is used to generate a high-quality underwater image, so that the dependence on expensive measured data and artificial fine labeling can be significantly reduced, and the method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an underwater scene generation method and related equipment based on multimodal fusion and label guidance prompts. Background Technology

[0002] With the increasing demand for marine resource development and ecological protection, underwater environmental perception technology is becoming increasingly important in scientific research, robot navigation, and resource exploration. Among these technologies, semantic segmentation and depth estimation, with computer vision at its core, are key means to achieve underwater scene understanding. Although deep learning has achieved breakthroughs in terrestrial vision tasks with large-scale datasets, underwater vision research remains plagued by data scarcity. On the one hand, the acquisition of high-quality depth data is extremely costly due to expensive specialized equipment and complex dynamic environments. On the other hand, the absorption and scattering of light by water causes severe color shifts and contrast degradation in images, greatly increasing the difficulty of pixel-level semantic annotation and accurate depth information acquisition. Existing datasets are insufficient to support the training needs of deep models. Specifically, existing technologies suffer from at least the following drawbacks: 1. Extremely high cost and limited annotation accuracy for real-world data acquisition: The acquisition of real underwater images is highly dependent on professional diving equipment or underwater vehicles, and is easily affected by water turbidity, light attenuation, and current environment, resulting in long acquisition cycles and high costs for high-quality depth data samples. Furthermore, due to severe color shifts, scattering blurring, and contrast degradation in underwater imaging, edge detection is blurry when manually performing pixel-level semantic annotation, and depth information acquired through sensors often contains holes or alignment errors, making it difficult to support the needs of high-precision supervised learning models.

[0003] 2. Insufficient generalization ability of traditional physical imaging models: Existing synthesis methods based on physical degradation models (such as the Jaffe-McGlamery model) are often based on simplified optical assumptions, which makes it difficult to simulate the complex and varied water body types in reality (such as different levels of chlorophyll content, suspended particulate matter and backscattering effects). This results in a huge "domain gap" between the generated synthetic images and the real underwater scene. After the model is trained on synthetic data, its generalization performance in practical applications is poor.

[0004] 3. Existing generative models lack fine controllability and consistency: Traditional generative adversarial networks (GANs) are prone to mode collapse when synthesizing underwater images and struggle to maintain a high degree of consistency between semantic labels and image content at high resolutions. While existing diffusion models improve image quality, they often lack effective guidance mechanisms when dealing with underwater-specific multimodal constraints (such as depth guidance and spatial-channel feature coupling), resulting in generated images that still do not conform to the real underwater imaging patterns in terms of physical logic, structural layout, and detailed texture. Summary of the Invention

[0005] The main objective of this invention is to propose an underwater scene generation method, apparatus, electronic device, storage medium, and program product based on multimodal fusion and tag-guided prompts, aiming to solve at least one problem in the prior art.

[0006] To achieve the above objectives, one aspect of this invention proposes an underwater scene generation method based on multimodal fusion and tag-guided prompts, the method comprising: Acquire underwater images and construct multimodal data representations; the multimodal data representations include underwater images and their corresponding semantic maps, depth maps, edge maps, and initial text descriptions; Based on semantic maps, depth maps, and edge maps, the fusion condition features are obtained by sequentially processing independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion using a multimodal residual condition fusion module. Water body types are classified based on underwater images, and then enhanced text prompts are constructed by combining semantic maps and initial text descriptions. Based on fusion conditional features and enhanced textual prompts, synthetic underwater images are generated using a latent space diffusion model.

[0007] In some embodiments, constructing a multimodal data representation includes the following steps: A semantic graph is obtained by aggregating instance-level annotation information based on underwater images. A depth map is obtained by forward inference of underwater images using a pre-trained depth estimation model. The edge map is obtained by extracting the image boundary information of the underwater image through an edge detection network; Based on underwater images, corresponding initial text descriptions are generated using a visual language model.

[0008] In some embodiments, based on semantic maps, depth maps, and edge maps, a multimodal residual conditional fusion module is used to sequentially process the data through independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion to obtain fused conditional features, including the following steps: Based on the semantic graph, high-dimensional semantic features are extracted using the first encoder. Based on the depth map, high-dimensional depth features are extracted using a second encoder. Based on the edge map, high-dimensional edge features are extracted using a third encoder; Feature preprocessing is performed on deep high-dimensional features and edge high-dimensional features, and then feature enhancement is performed through the spatial channel collaborative attention module to obtain deep enhancement features and edge enhancement features respectively; Using semantic high-dimensional features as the dominant benchmark, residual fusion is performed by combining deep enhancement features and edge enhancement features to obtain fused conditional features.

[0009] In some embodiments, feature preprocessing is performed on the deep high-dimensional features and edge high-dimensional features, and then feature enhancement is performed through a spatial channel collaborative attention module to obtain corresponding depth-enhanced features and edge-enhanced features, including the following steps: Based on deep high-dimensional features, the deep preprocessed features are obtained by summing the results of all parallel processing element by element through parallel first convolution, second convolution and batch normalization. Based on the high-dimensional edge features, the edge preprocessing features are obtained by summing the results of all parallel processing element by element through parallel first convolution, second convolution and batch normalization. Based on deep preprocessed features, feature enhancement is performed using a spatial channel collaborative attention module to obtain deep enhanced features; Based on edge preprocessing features, feature enhancement is performed using a spatial channel collaborative attention module to obtain edge enhanced features.

[0010] In some embodiments, semantic high-dimensional features are used as the dominant benchmark, and residual fusion is performed by combining deep enhancement features and edge enhancement features to obtain fused conditional features, including the following steps: The semantic high-dimensional features, deep enhancement features, and edge enhancement features are added element-wise to obtain the preliminary fusion features; The initial fused features are aligned and compressed using a pre-defined mapping function to obtain joint features; the mapping function is constructed by stacking multiple layers of convolutional kernels. By performing a residual connection that adds the semantic high-dimensional features and joint features element-wise, we obtain the fused conditional features.

[0011] In some embodiments, water body type classification is performed based on underwater images, and then enhanced text prompts are constructed by combining semantic maps and initial text descriptions, including the following steps: The underwater image and the preset water body classification prompt template are input into the visual big language model to determine the water body type and obtain the initial water body type. Obtain the voting results of all annotators regarding the initial water body type; If the proportion of those in agreement in the evaluation voting results is greater than a preset threshold, the initial water body type will be used as the target water body type; otherwise, the underwater images will be sent to domain experts to obtain their feedback and final decision as the target water body type. Based on the target water body type, the initial text description, and the label categories corresponding to the semantic map, enhanced text prompts are obtained by fusing them using a fixed template.

[0012] In some embodiments, a synthetic underwater image is generated using a latent space diffusion model based on fusion conditional features and enhanced textual cues, including the following steps: A noise distribution is constructed by progressively adding Gaussian noise to underwater images through a forward process using a latent space diffusion model. In the reverse process of the latent space diffusion model, the fusion conditional features and enhanced textual prompts are used as joint conditions to progressively remove noise from the noise distribution in order to restore the image content and obtain a synthetic underwater image.

[0013] To achieve the above objectives, another aspect of this invention proposes an underwater scene generation device based on multimodal fusion and tag-guided prompts, the device comprising: The first module is used to acquire underwater images and construct multimodal data representations; wherein, the multimodal data representations include underwater images and their corresponding semantic maps, depth maps, edge maps and initial text descriptions; The second module is used to process semantic maps, depth maps, and edge maps, and then use a multimodal residual conditional fusion module to obtain fusion conditional features through independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion. The third module is used to classify water body types based on underwater images, and then combine semantic maps and initial text descriptions to construct enhanced text prompts. The fourth module is used to generate synthetic underwater images based on fusion conditional features and enhanced textual cues using a latent space diffusion model.

[0014] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0016] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0017] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method, apparatus, electronic device, storage medium, and program product for generating underwater scenes based on multimodal fusion and tag-guided prompts. This solution acquires underwater images and constructs a multimodal data representation. The multimodal data representation includes the underwater image and its corresponding semantic map, depth map, edge map, and initial text description. Based on the semantic map, depth map, and edge map, a multimodal residual conditional fusion module is used to process the images through independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion to obtain fusion conditional features. Water body type classification is performed based on the underwater image, and then enhanced text prompts are constructed by combining the semantic map and the initial text description. Based on the fusion conditional features and enhanced text prompts, a synthetic underwater image is generated using a latent space diffusion model. This invention introduces semantic maps, depth maps, and edge maps through a multimodal residual conditional fusion module, and combines them with a latent space diffusion model to generate high-quality underwater images. This significantly reduces the reliance on expensive measured data and meticulous manual annotation. The introduction of the edge map effectively enhances semantic boundary consistency and alleviates pixel-level annotation ambiguity. Furthermore, based on water body type classification and enhanced text prompts, this invention can flexibly simulate the optical characteristics of different water bodies, effectively compensating for the limitations of simplified traditional physical models and significantly narrowing the domain gap between synthesized images and real-world scenes. Similarly, this invention utilizes spatial channel collaborative attention and semantically driven residual fusion to ensure a high degree of coupling between the generated image and the input depth, semantics, and other multimodal conditions. The latent space diffusion model avoids mode collapse and effectively maintains the realism of physical logic and texture details at high resolution. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an implementation environment for an underwater scene generation method based on multimodal fusion and tag-guided prompting provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the underwater scene generation method based on multimodal fusion and tag-guided prompts provided in an embodiment of the present invention. Figure 3 This is a comparative schematic diagram of Atlantis, TIDE, and the method of the present invention provided in the embodiments of the present invention; Figure 4 This is an example illustration of how text alone cannot model the complex underwater world; Figure 5 This is a schematic diagram of the overall process architecture of the underwater scene generation method based on multimodal fusion and tag-guided prompts provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of the structure of the multimodal residual condition fusion module provided in an embodiment of the present invention; Figure 7This is a schematic diagram of the water body type determination process provided in an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the visual effect of an image example provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the qualitative results on the SQUID dataset provided in an embodiment of the present invention; Figure 10 This is a schematic diagram illustrating the visualization effects of testing different models using different datasets, provided by an embodiment of the present invention. Figure 11 This is a schematic diagram of the underwater scene generation device based on multimodal fusion and tag-guided prompts provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0020] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0021] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0022] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0023] Among the related technologies, the shortcomings of existing technologies include: extremely high cost of acquiring measured data and limited annotation accuracy; insufficient generalization ability of traditional physical imaging models; and lack of fine controllability and consistency in existing generative models.

[0024] In view of this, embodiments of the present invention provide an underwater scene generation method and related equipment based on multimodal fusion and tag-guided prompts. This scheme acquires underwater images and constructs a multimodal data representation. The multimodal data representation includes the underwater image and its corresponding semantic map, depth map, edge map, and initial text description. Based on the semantic map, depth map, and edge map, a multimodal residual conditional fusion module is used to process the data through independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion to obtain fusion conditional features. The underwater image is used to classify water body types, and then combined with the semantic map and initial text description to construct enhanced text prompts. Based on the fusion conditional features and enhanced text prompts, a synthetic underwater image is generated using a latent space diffusion model. This invention introduces semantic maps, depth maps, and edge maps through a multimodal residual conditional fusion module, and combines them with a latent space diffusion model to generate high-quality underwater images. This significantly reduces the reliance on expensive measured data and meticulous manual annotation. The introduction of the edge map effectively enhances semantic boundary consistency and alleviates pixel-level annotation ambiguity. Furthermore, based on water body type classification and enhanced text prompts, this invention can flexibly simulate the optical characteristics of different water bodies, effectively compensating for the limitations of simplified traditional physical models and significantly narrowing the domain gap between synthesized images and real-world scenes. Similarly, this invention utilizes spatial channel collaborative attention and semantically driven residual fusion to ensure a high degree of coupling between the generated image and the input depth, semantics, and other multimodal conditions. The latent space diffusion model avoids mode collapse and effectively maintains the realism of physical logic and texture details at high resolution.

[0025] It is understood that the underwater scene generation method based on multimodal fusion and tag-guided prompts provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0026] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0027] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0028] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0029] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0030] For example, based on Figure 1 The implementation environment shown in this embodiment of the invention provides an underwater scene generation method based on multimodal fusion and tag guidance prompts. The following description uses the application of this underwater scene generation method based on multimodal fusion and tag guidance prompts in server 101 as an example. It can be understood that this underwater scene generation method based on multimodal fusion and tag guidance prompts can also be applied to terminal 102.

[0031] Reference Figure 2 , Figure 2 This is an optional flowchart of an underwater scene generation method based on multimodal fusion and tag guidance provided in an embodiment of the present invention. The execution subject of this underwater scene generation method based on multimodal fusion and tag guidance can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S400.

[0032] Step S100: Acquire underwater images and construct multimodal data representation; The multimodal data representation includes underwater images and their corresponding semantic maps, depth maps, edge maps, and initial text descriptions; It should be noted that, in some embodiments, constructing a multimodal data representation may include the following steps: obtaining a semantic map by aggregating instance-level annotation information based on underwater images; obtaining a depth map by performing forward inference on underwater images using a pre-trained depth estimation model; obtaining an edge map by extracting image boundary information of underwater images through an edge detection network; and generating corresponding initial text descriptions based on underwater images using a visual language model.

[0033] For example, in some specific implementations, during the data construction phase, the present invention uses the largest underwater instance segmentation dataset currently available. As training data, first use underwater images. As input, a semantic graph is obtained by aggregating instance-level annotation information. This is used to characterize the spatial distribution of different types of targets in a scene; simultaneously, a pre-trained depth estimation model is utilized. Inference is performed on the input image to obtain a depth map. It is used to describe pixel-level geometric relationships; further, it is used through an edge detection network. Extract image boundary information to obtain edge map To enhance the ability to express structural contours; in addition, a corresponding initial text description is generated through a visual language model (BLIP-2 model). Thus, a unified multimodal data representation can be constructed. .

[0034] Step S200: Based on the semantic map, depth map and edge map, the multimodal residual conditional fusion module is used to process the fusion conditional features by sequentially performing independent encoding, spatial channel collaborative attention enhancement and semantic-dominated residual fusion. It should be noted that in some embodiments, step S200 may include the following steps: extracting semantic high-dimensional features using a first encoder based on a semantic map; extracting deep high-dimensional features using a second encoder based on a depth map; extracting edge high-dimensional features using a third encoder based on an edge map; performing feature preprocessing on the deep high-dimensional features and edge high-dimensional features, and then performing feature enhancement through a spatial channel collaborative attention module to obtain corresponding deep enhancement features and edge enhancement features; using the semantic high-dimensional features as the dominant benchmark, combining the deep enhancement features and edge enhancement features to perform residual fusion to obtain fusion conditional features.

[0035] For example, in some specific embodiments, the present invention constructs a multimodal residual fusion module (MCRF) to jointly model semantic segmentation, depth and edge information.

[0036] Let the input be a semantic graph. Depth map and edge map These three modalities first extract initial features through independent encoding networks. Their mathematical expression is as follows:

[0037] in, There are three independent encoders, each responsible for extracting high-dimensional features from the semantic map, depth map, and edge map. It provides discrete and stable semantic category labels (i.e., high-dimensional semantic features), which determine the macroscopic layout of the generated scene; The spatial topological features (i.e., deep high-dimensional features) of the scene were extracted, which characterize the relative depth between objects; This extracts high-frequency gradient information (i.e., high-dimensional edge features), which emphasizes the target boundary and local details. These three elements are highly complementary in physical representation, but their feature distributions differ significantly.

[0038] It should be pointed out that, and Relying on pre-trained models, most existing pre-trained models are trained on clear land images. Underwater environments suffer from severe domain shift issues. Therefore, during cross-domain extraction, both types of features inevitably contain erroneous responses. To address this, this invention employs a Spatial Channel Collaborative Attention (SCSA) module to uniformly model depth and edge features. Before entering the SCSA module, the features are preprocessed through convolution operations to enhance their expressive power. Subsequently, the processed features are input into the SCSA module.

[0039] After feature enhancement, the process enters the multimodal fusion stage. In underwater scenarios, semantic tags are unaffected by illumination degradation and possess the highest physical reliability. Therefore, this invention establishes high-dimensional semantic features. As the dominant benchmark for the integration process.

[0040] It should be noted that in some embodiments, feature preprocessing is performed on deep high-dimensional features and edge high-dimensional features, and then feature enhancement is performed through a spatial channel collaborative attention module to obtain deep enhanced features and edge enhanced features. This can include the following steps: based on the deep high-dimensional features, the results of all parallel processing are summed element-wise through parallel first convolution, second convolution, and batch normalization to obtain deep preprocessed features; based on the edge high-dimensional features, the results of all parallel processing are summed element-wise through parallel first convolution, second convolution, and batch normalization to obtain edge preprocessed features; based on the deep preprocessed features, feature enhancement is performed using a spatial channel collaborative attention module to obtain deep enhanced features; based on the edge preprocessed features, feature enhancement is performed using a spatial channel collaborative attention module to obtain edge enhanced features.

[0041] For example, in some specific implementations, features are preprocessed through convolution operations before entering SCSA to enhance their expressive power. Specifically, this is done in parallel. Convolution (i.e., the first convolution) and Convolution (i.e., second convolution), combined with batch normalization:

[0042]

[0043] Similarly, the preprocessing process for edge branches is expressed as follows:

[0044]

[0045] in, Convolution is used to extract local spatial structure information, giving features a stronger neighborhood representation ability; Convolution is used to enable information exchange between channels; batch normalization is used to standardize the feature distribution, thereby improving the stability of subsequent processing. and These are the features extracted initially (i.e., deep preprocessed features and edge preprocessed features). The purpose of this step is to map the original features to a representation space that is more suitable for attention modeling.

[0046] The processed features are then input into the SCSA module:

[0047] Among them, SCSA stands for Spatial Channel Collaborative Attention. It is the feature map after the original depth map has been enhanced (i.e., depth-enhanced feature). This is the enhanced feature map (i.e., edge enhancement feature) of the original edge detection map. The SCSA module collaboratively reconstructs the features in both spatial and channel dimensions. In the spatial dimension, the network calculates the positional weight matrix. It adaptively assigns higher weights to structurally clear regions and suppresses interference from blurred water bodies. In the channel dimension, the network evaluates the information entropy of different feature maps. It actively amplifies the channel responses that are sensitive to geometric structure. Through this two-dimensional dynamic gating mechanism, SCSA can accurately filter out false edges and erroneous depths. Compared to traditional static weighting, SCSA provides more refined and controllable adjustments to the local feature distribution.

[0048] It should be noted that in some embodiments, semantic high-dimensional features are used as the dominant benchmark, and residual fusion is performed by combining deep enhancement features and edge enhancement features to obtain fusion conditional features. This may include the following steps: adding semantic high-dimensional features, deep enhancement features and edge enhancement features element-wise to obtain preliminary fusion features; performing feature alignment and compression on the preliminary fusion features through a preset mapping function to obtain joint features; wherein, the mapping function is constructed by stacking multiple convolutional kernels; and performing residual connection of semantic high-dimensional features and joint features element-wise to obtain fusion conditional features.

[0049] For example, in some specific implementations, the network first adds the three types of feature tensors element by element:

[0050] in, This is the feature map resulting from the summation of three features. This initial summation operation breaks down the isolation between modalities. It initially aggregates heterogeneous information into a feature tensor of the same dimension. Subsequently, through a mapping function... Align and compress the fused features:

[0051] This is the output feature map after initial fusion, and the mapping function. Typically composed of multiple stacked convolutional kernels, its core task is to bridge the semantic gap between multimodalities and extract high-dimensional joint representations. To prevent the attenuation of high-level semantic information during complex convolutional operations, this invention introduces a residual connection mechanism. The network integrates the fused joint features with the original semantic features. Perform element-wise summation:

[0052] in This is the feature map that is ultimately input into the conditional control. This structure ensures that semantic information dominates the final output, while allowing depth and edge information to participate in the representation in a complementary form. From an overall structural perspective, each module has a clear division of labor. The encoder is responsible for extracting basic features. Convolution and normalization are used to adjust the feature distribution. SCSA is used to suppress noise and enhance structural regions. The fusion mapping achieves multimodal alignment. Residual connections are used to stabilize information transmission. The cooperation between these steps ensures good controllability of the fusion process.

[0053] Finally, output features This feature serves as a conditional input in the denoising process of the diffusion model. It strikes a balance between semantic consistency and structural representation, providing more stable constraints in complex underwater environments, thereby improving the structural rationality and detail representation of the generated images.

[0054] Step S300: Based on the underwater image, water body type is classified, and then enhanced text prompts are constructed by combining the semantic map and the initial text description; It should be noted that in some embodiments, step S300 may include the following steps: inputting the underwater image and a preset water body classification prompt template into a visual large language model to determine the water body type and obtain an initial water body type; obtaining the evaluation voting results of all annotators on the initial water body type; if the proportion of agreement in the evaluation voting results is greater than a preset proportion threshold, the initial water body type is taken as the target water body type; otherwise, the underwater image is sent to a domain expert to obtain the final decision of the domain expert as the target water body type; based on the target water body type, the initial text description, and the label category corresponding to the semantic map, the enhanced text prompt is obtained by fusing them through a fixed template.

[0055] In some exemplary embodiments, to further improve the controllability and repeatability of the generation process, this invention proposes a Label-Guided Prompt Enhancement (LPE) module. This module employs a lightweight design, with all calculations performed during the data preprocessing stage, thereby avoiding the introduction of additional computational overhead during the training and inference of the diffusion model. This design ensures the stability of the model's main structure while enhancing the expressive power of conditional information.

[0056] In its implementation, LPE uses training samples As the basic processing unit, where Indicates degraded underwater images, The corresponding semantic segmentation is labeled (i.e., the label category corresponding to the semantic graph). First, to obtain the initial semantic description, for each input image... Generate text descriptions using a pre-trained BLIP-2 model. (That is, the initial text description, which has been completed in step S100). This description can provide basic scene semantic information, such as the target category and its spatial relationships.

[0057] Building upon this foundation, this invention introduces an explicit label enhancement mechanism to correct and supplement the initial semantics. Specifically, it constructs a four-level underwater degradation classification system based on three independent physical dimensions: sharpness, color distortion, and turbidity. This system covers continuous changes from clear environments to high turbidity and low illumination conditions, and is expressed in the form of discrete labels. In this way, the complex underwater imaging degradation process can be transformed into structured semantic information, thereby enhancing the model's ability to perceive environmental attributes.

[0058] Regarding the annotation process, this invention adopts a strategy combining machine prediction and manual verification. First, a Large Language Model (LLM) is used to perform a preliminary evaluation of each image across three dimensions, establishing a unified initial annotation benchmark. Based on this, a multi-stage manual verification mechanism is introduced. All samples are independently evaluated by three annotators with underwater vision research experience, and the annotation results are determined by majority vote. For samples with differing opinions, domain experts make the final decision. This process effectively reduces subjective bias, improves the consistency and reliability of the labels, and ensures that the annotation results have strong physical plausibility.

[0059] Subsequently, the validated degenerate labels are mapped to structured natural language descriptions and stored in lightweight JSON format. This design eliminates the need for additional inference or label generation during the training phase, thereby improving overall training efficiency. In the prompt word construction phase, the degenerate labels and weak semantic descriptions are fused using a fixed template to obtain the final prompt words.

[0060] Final generated prompt words The text encoder input into the diffusion model serves as conditional information during training and inference. Compared to methods that rely solely on text or a single semantic description, this strategy can simultaneously express "environmental attributes" and "target information" at the semantic level. Specifically, degenerate labels provide global environmental constraints, while weak semantic descriptions provide target-level information, forming a complementary relationship.

[0061] Step S400: Based on fusion conditional features and enhanced text prompts, a synthetic underwater image is generated using a latent space diffusion model; It should be noted that in some embodiments, step S400 may include the following steps: using a latent space diffusion model to gradually add Gaussian noise to the underwater image through a forward process to construct a noise distribution; in the reverse process of the latent space diffusion model, using fusion conditional features and enhanced text prompts as joint conditions, gradually removing noise from the noise distribution to restore the image content, thereby obtaining a synthetic underwater image.

[0062] For example, in some specific embodiments, during the image generation stage, the present invention constructs a conditional generation process based on a latent space diffusion model. In the forward process, Gaussian noise is progressively added to the real image to construct a noise distribution; in the backward process, multimodal fusion features and enhanced text semantics are used as joint conditions to progressively remove noise and restore image content. At each diffusion time step, conditional information is injected into the main generator network through a control branch, ensuring that the generated result progressively approximates the real distribution while satisfying multiple constraints of semantic consistency, structural rationality, and clear boundaries. Through this method, the generated image not only possesses high visual realism but also maintains strict alignment with the input conditions in terms of semantic structure and geometric relationships.

[0063] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0064] First, it should be noted that, as Figure 3 As shown, existing diffusion model-based methods directly generate underwater scenes, such as Atlantis ( Figure 3 (a) marked part) and TIDE ( Figure 3 (b) Marked section). Atlantis generates relatively realistic underwater images by pairing land depth information with corresponding land images. However, due to significant differences in category distribution and visual features between land and underwater scenes, a large domain gap still exists. In contrast, TIDE alleviates this problem to some extent, but it mainly relies on text-driven generation and lacks effective constraints on the structure of the underwater scene. Figure 4 As shown, when faced with complex scenes, TIDE often generates images with messy structures and unreasonable layouts, which not only affects the generation quality but also introduces additional noise to downstream tasks (such as semantic segmentation).

[0065] In view of this, such as Figure 3 As shown ( Figure 3 (c) Marked part), this invention proposes a multimodal conditional diffusion underwater scene generation method based on the collaborative constraints of semantic information, geometric information and structural information. By constructing a unified multimodal expression system and semantic enhancement mechanism, structural consistency and semantic alignment constraints are introduced in the diffusion generation process, thereby realizing highly realistic and controllable image generation in complex underwater environments, and used to support data construction for downstream visual tasks.

[0066] In this invention, underwater scene images are first used as the basic input, and multimodal prior information is constructed based on the semantic composition, spatial structure, and geometric relationships of the images. Specifically, a semantic map is obtained by aggregating instance-level annotations on the original images to describe the spatial distribution of various target categories in the scene; simultaneously, a pre-trained depth estimation model is used to infer from the images to obtain a depth map reflecting three-dimensional structural relationships; further, an edge detection model is used to extract the boundary and contour information of the images to obtain an edge map, thereby enhancing the expression of structural details. Based on this, an initial text description related to the image content is generated by combining a visual language model, thus forming a multimodal condition set including semantics, depth, edges, and text semantics, providing unified prior constraints for the subsequent generation process.

[0067] To ensure consistency of information from different modalities in subsequent processing, this invention performs unified preprocessing and standardization operations on multimodal data, including spatial scale alignment, resolution unification, and normalization, so that each modality can be represented in the same coordinate system. Simultaneously, to address potential noise issues in depth maps and edge maps under complex underwater degradation conditions, smoothing and anomaly suppression are applied to improve the stability of the subsequent fusion process.

[0068] In the multimodal fusion stage, this invention constructs a residual conditional fusion mechanism dominated by semantic information. Specifically, firstly, the semantic map, depth map, and edge map are feature-mapped separately using independent encoders, transforming them into a unified feature space. Then, a joint channel and spatial attention mechanism is introduced for depth and edge features to enhance effective structural information and suppress noise interference. During the fusion process, semantic features are used as the backbone representation, and the enhanced depth and edge features are injected into the semantic branch through a residual method. This maintains semantic consistency while supplementing geometric and boundary information, ultimately obtaining a unified multimodal conditional feature representation. This fusion method avoids the information redundancy problem caused by simple splicing and effectively improves the collaborative expressive ability between different modalities.

[0069] In terms of semantic constraint construction, this invention further introduces a label-guided text enhancement mechanism to improve the controllability of the diffusion model in complex underwater degradation environments. Specifically, firstly, the underwater scene is classified into degradation levels based on factors such as image turbidity and lighting conditions, and corresponding category label information is extracted from the semantic map. Subsequently, the degradation level, semantic category, and initial text description are structurally fused to construct an enhanced text prompt containing explicit semantics and scene attributes, which is then converted into a semantic vector representation by a text encoder. This semantic vector participates in the subsequent generation process as conditional information, thereby introducing explicit category constraints and scene control capabilities at the generation stage, effectively alleviating the problem of inaccurate expression of traditional text descriptions in underwater degradation scenarios.

[0070] In the image generation stage, this invention constructs a conditional generation process based on a latent space diffusion model. In the forward pass, Gaussian noise is progressively added to the real image to construct a noise distribution. In the backward pass, multimodal fusion features and enhanced text semantics are used as joint conditions to progressively remove noise and restore image content. At each diffusion time step, conditional information is injected into the main generator network through a control branch, ensuring that the generated result gradually approximates the real distribution while satisfying multiple constraints: semantic consistency, structural rationality, and clear boundaries. Through this approach, the generated image not only possesses high visual realism but also maintains strict alignment with the input conditions in terms of semantic structure and geometric relationships.

[0071] During model training, this invention optimizes network parameters by minimizing the difference between predicted and actual noise, and simultaneously introduces multimodal conditions and semantic cues during the training phase, enabling the model to learn the real data distribution in complex underwater environments. Through a joint constraint mechanism, the model maintains stable generation capabilities even when facing scenes with varying turbidity, lighting changes, and complex structures.

[0072] After the model converges, the trained generative model can be used for large-scale underwater data synthesis. Specifically, by inputting different combinations of semantic maps, depth maps, edge maps, and text prompts, underwater images that strictly correspond to the conditions can be generated, thus constructing a synthetic dataset with high consistency and diversity. This dataset maintains consistency with the input prior in terms of semantic labels and geometric information and can be directly used for supervised learning tasks.

[0073] Finally, the generated data is applied to downstream visual tasks such as underwater semantic segmentation and depth estimation. By replacing or supplementing real data, it effectively alleviates the data scarcity problem and significantly improves the model's generalization ability and robustness in complex underwater environments. Furthermore, this invention supports controllable generation by adjusting the degradation level and semantic input, adapting to different application scenarios and showing promising application prospects in underwater exploration, unmanned underwater vehicle visual perception, and marine environmental monitoring.

[0074] In some specific application scenarios, such as Figure 5 As shown, the underwater scene generation method proposed in this invention, based on multimodal fusion and label-guided prompts, is built upon a conditional diffusion model framework. Through the collaborative modeling of semantic, geometric, structural, and textual semantic information, multiple constraint mechanisms are introduced during the generation process, thereby achieving highly consistent, realistic, and controllable image generation in complex underwater environments. Its complete process includes stages such as multimodal prior construction, multimodal feature fusion, textual semantic enhancement, and conditional diffusion generation, with each stage coupled to form a unified closed loop. Specifically, the implementation includes: During the data construction phase, this invention uses the largest underwater instance segmentation dataset currently available. As training data, first use underwater images. As input, a semantic graph is obtained by aggregating instance-level annotation information. This is used to characterize the spatial distribution of different types of targets in a scene; simultaneously, a pre-trained depth estimation model is utilized. Inference is performed on the input image to obtain a depth map. It is used to describe pixel-level geometric relationships; further, it is used through an edge detection network. Extract image boundary information to obtain edge map To enhance the ability to express structural contours; in addition, a corresponding initial text description is generated through a visual language model (BLIP-2 model). Thus, a unified multimodal data representation can be constructed. .

[0075] In underwater image generation tasks, single-constraint modalities often fall short. The underwater environment is extremely complex. Intense light scattering, severe color shifts, and a sharp decrease in image contrast all contribute to degradation. The physical structure of images is crucial. However, these structures also introduce significant visual noise. Relying solely on semantic segmentation maps leads to the loss of local texture details. Conversely, relying solely on geometric depth maps prevents models from accurately distinguishing object categories. A single prior is insufficient to balance structural integrity and semantic accuracy. Therefore, this invention constructs a Multimodal Residual Fusion Module (MCRF). This module is as follows... Figure 6 As shown, semantic segmentation, depth, and edge information are jointly modeled.

[0076] Let the input be a semantic graph. Depth map and edge map These three modalities first extract initial features through independent encoding networks. Their mathematical expression is as follows:

[0077] in, There are three independent encoders, each responsible for extracting high-dimensional features from the semantic map, depth map, and edge map. It provides discrete and stable semantic category labels, which determine the macroscopic layout of the generated scene; The spatial topological features of the scene were extracted, which characterize the relative depth between objects; This extracts high-frequency gradient information, emphasizing target boundaries and local details. These three elements are highly complementary in physical representation, but their feature distributions differ significantly.

[0078] It should be pointed out that, and Relying on pre-trained models, most existing pre-trained models are trained on clear land images. Underwater environments suffer from severe domain shift problems. Therefore, both types of features inevitably contain erroneous responses during cross-domain extraction. For example, in low-contrast waters, depth estimation often shows large-area smoothing faults. In turbid water areas, suspended particles induce edge detection networks to generate false boundaries. Directly inputting these noisy features into the backbone network severely interferes with the denoising trajectory of the diffusion model. Therefore, rigorous screening and reconstruction of geometric features are essential.

[0079] To address this, this invention employs a Spatial Channel Collaborative Attention (SCSA) module to uniformly model depth and edge features. Before entering the SCSA, the features are preprocessed through convolutional operations to enhance their expressive power. Specifically, this is done in parallel. Convolution and Convolution, combined with batch normalization:

[0080]

[0081] Similarly, the preprocessing process for edge branches is expressed as follows:

[0082]

[0083] in, Convolution is used to extract local spatial structure information, giving features a stronger neighborhood representation ability; Convolution is used to enable information exchange between channels; batch normalization is used to standardize the feature distribution, thereby improving the stability of subsequent processing. and These are the features after initial extraction. The purpose of this step is to map the original features to a representation space that is more suitable for attention modeling.

[0084] The processed features are then input into the SCSA module:

[0085] Among them, SCSA stands for Spatial Channel Collaborative Attention. It is the feature map after enhancement of the original depth map. This is the enhanced feature map of the original edge detection map. The SCSA module collaboratively reconstructs features in both spatial and channel dimensions. Spatially, the network calculates the location weight matrix. It adaptively assigns higher weights to structurally clear regions, suppressing interference from blurred water bodies. In the channel dimension, the network evaluates the information entropy of different feature maps. It actively amplifies the channel responses that are sensitive to geometric structure. Through this two-dimensional dynamic gating mechanism, SCSA can accurately filter out false edges and incorrect depths. Compared to traditional static weighting, SCSA provides more refined and controllable adjustments to the local feature distribution.

[0086] After feature enhancement, the process enters the multimodal fusion stage. In underwater scenarios, semantic tags are unaffected by illumination degradation and possess the highest physical reliability. Therefore, this invention establishes high-dimensional semantic features. As the dominant benchmark in the fusion process, the network first adds the three types of feature tensors element-wise:

[0087] in, This is the feature map resulting from the summation of three features. This initial summation operation breaks down the isolation between modalities. It initially aggregates heterogeneous information into a feature tensor of the same dimension. Subsequently, through a mapping function... Align and compress the fused features:

[0088] This is the output feature map after initial fusion, and the mapping function. Typically composed of multiple stacked convolutional kernels, its core task is to bridge the semantic gap between multimodalities and extract high-dimensional joint representations. To prevent the attenuation of high-level semantic information during complex convolutional operations, this invention introduces a residual connection mechanism. The network integrates the fused joint features with the original semantic features. Perform element-wise summation:

[0089] in This is the feature map that is ultimately input into the conditional control. This structure ensures that semantic information dominates the final output, while allowing depth and edge information to participate in the representation in a complementary form. From an overall structural perspective, each module has a clear division of labor. The encoder is responsible for extracting basic features. Convolution and normalization are used to adjust the feature distribution. SCSA is used to suppress noise and enhance structural regions. The fusion mapping achieves multimodal alignment. Residual connections are used to stabilize information transmission. The cooperation between these steps ensures good controllability of the fusion process.

[0090] Finally, output features This feature serves as a conditional input in the denoising process of the diffusion model. It strikes a balance between semantic consistency and structural representation, providing more stable constraints in complex underwater environments, thereby improving the structural rationality and detail representation of the generated images.

[0091] Furthermore, to improve the controllability and repeatability of the generation process, this invention proposes a Label-Guided Prompt Enhancement (LPE) module. This module employs a lightweight design, with all computations completed during the data preprocessing stage, thereby avoiding the introduction of additional computational overhead during diffusion model training and inference. This design ensures the stability of the model's main structure while enhancing the expressive power of conditional information.

[0092] In its implementation, LPE uses training samples As the basic processing unit, where Indicates degraded underwater images, For the corresponding semantic segmentation annotation. First, to obtain the initial semantic description, for each input image... Generate text descriptions using a pre-trained BLIP-2 model. (i.e., the initial text description). This description provides basic scene semantic information, such as the target category and its spatial relationships.

[0093] However, general visual language models such as BLIP-2 are primarily trained on terrestrial data, limiting their generalization ability in underwater scenarios. In practical applications, these models may exhibit semantic biases, such as misclassifying target categories or outputting overly generalized descriptions. These errors directly affect the conditional inputs of the diffusion model, causing semantic shifts in the generated results. Theoretically, this problem can be mitigated by domain-specific fine-tuning of BLIP-2, but such models have large parameter sizes, and the fine-tuning process requires significant computational resources and labeled data, resulting in high implementation costs. Therefore, directly relying on model fine-tuning is not feasible. Based on this, this invention does not... Instead of serving as final supervisory information, it is treated as a weak semantic prior, used only to provide a basic description.

[0094] Building upon this foundation, this invention introduces an explicit label enhancement mechanism to correct and supplement the initial semantics. Specifically, it constructs a four-level underwater degradation classification system based on three independent physical dimensions: sharpness, color distortion, and turbidity. Figure 7 As shown, this system covers continuous variations from clear environments to highly turbid, low-light conditions, and is expressed in the form of discrete labels. In this way, complex underwater imaging degradation processes can be transformed into structured semantic information, thereby enhancing the model's ability to perceive environmental attributes.

[0095] Regarding the annotation process, such as Figure 7 As shown, this invention employs a strategy combining machine prediction and manual verification. First, a Large Language Model (LLM) is used to perform a preliminary evaluation of each image across three dimensions, establishing a unified initial annotation benchmark. Based on this, a multi-stage manual verification mechanism is introduced. All samples are independently evaluated by three annotators with underwater vision research experience, and the annotation results are determined by majority vote. For samples with differing opinions, domain experts make the final decision. This process effectively reduces subjective bias, improves the consistency and reliability of the labels, and ensures the annotation results have strong physical plausibility.

[0096] Subsequently, the validated degenerate labels are mapped to structured natural language descriptions and stored in lightweight JSON format. This design eliminates the need for additional inference or label generation during the training phase, thereby improving overall training efficiency. In the prompt word construction phase, the degenerate labels and weak semantic descriptions are fused using a fixed template to obtain the final prompt words.

[0097] Final generated prompt words The text encoder input into the diffusion model serves as conditional information during training and inference. Compared to methods that rely solely on text or a single semantic description, this strategy can simultaneously express "environmental attributes" and "target information" at the semantic level. Specifically, degenerate labels provide global environmental constraints, while weak semantic descriptions provide target-level information, forming a complementary relationship.

[0098] Through the above design, the LEP module achieves explicit enhancement of conditional information without altering the main structure of the diffusion model. On one hand, this method avoids the need for costly fine-tuning of large-scale visual language models; on the other hand, it corrects semantics through structured labels, making the generated results more consistent with the real distribution of underwater scenes. Furthermore, since all enhancement processes are completed offline, this method does not increase the computational burden of the training and inference phases, demonstrating good engineering feasibility.

[0099] In summary, the LEP module reconstructs prompts using a combination of weak semantics and explicit labels, enabling the model to have more stable conditional modeling capabilities in complex underwater environments, thereby effectively improving the controllability and semantic consistency of the generated results.

[0100] Through the complete process described above, this invention simultaneously introduces semantic, geometric, and structural constraints during the generation process, and combines this with a text semantic enhancement mechanism to achieve fine-tuning of the generated content. This significantly improves the visual realism, semantic consistency, and structural rationality of the generated images, thereby effectively supporting the data requirements of tasks such as underwater semantic segmentation and depth estimation. The generated image is shown below. Figure 8 The fourth and last rows are examples of synthesized underwater images. The three rows above the examples of synthesized underwater images (from bottom to top) are the edge map, depth map, and semantic map of the corresponding examples of synthesized underwater images.

[0101] To verify the effectiveness of the method of the present invention, synthetic data was used to train existing semantic segmentation models and depth estimation models. Comparative experiments demonstrated the effectiveness and superiority of the method of the present invention.

[0102] In terms of data construction, the training data is divided into three types: the synthetic dataset Atlantis generated using the Atlantis method, the synthetic dataset SynTIDE generated using the TIDE method, and the SynMFLP dataset synthesized using the method of this invention. This setup can evaluate the independent effects of different data sources and analyze the role mechanism of synthetic data in underwater monocular depth estimation training.

[0103] To further verify the effectiveness and generalization performance of the synthetic data proposed in this invention in geometric structure modeling, a systematic quantitative comparative experiment was conducted on real underwater depth estimation datasets. Two representative benchmark datasets, Sea-Thru and SQUID, were selected to cover different water conditions and imaging degradation levels, thereby providing a more comprehensive analysis of the impact of different synthetic data on the cross-domain generalization ability of the depth estimation model.

[0104] Regarding model selection, this invention chooses three mainstream monocular depth estimation methods as evaluation benchmarks: NeWCRFs, PixelFormer, and MIM, to ensure the experimental results are sufficiently representative and comparable. In terms of training strategy, this invention strictly follows the settings in the Atlantis method, i.e., training with synthetic data and testing on real data. The overall training and evaluation process is consistent with the semantic segmentation experiment, both adopting a unified paradigm of "training with synthetic data and testing with real data" to ensure the comparability of experimental conclusions across different tasks. Results are as follows... Figure 9 The results shown are qualitative results on the SQUID dataset. A, T, and M represent models pre-trained on Atlantis, SynTIDE, and SynMFLP, respectively. After training on the dataset of this invention, the depth results are significantly improved. Qualitative analysis reveals that while the Atlantis-trained model achieves overall structural accuracy on real datasets, depth transitions in some areas are abrupt, failing to smoothly reproduce the scene's depth distribution. The SynTIDE-trained model improves upon the basic structure, reducing some discrete artifacts in model A, but its depth perception for mid-to-far scenes remains insufficiently nuanced. The SynMFLP-trained model exhibits the best macroscopic consistency, with its generated depth map most closely matching the outlines of near and far objects, indicating stronger representational capabilities in extracting global spatial features. Regarding detail preservation and edge sharpness, underwater imaging is prone to blurring depth map edges due to scattering effects. The Atlantis-trained model shows significant over-smoothing at object edges, resulting in substantial detail loss and making object boundaries appear "blurred" in the depth map. The SynTIDE-trained model optimizes edge sharpness to some extent, particularly in reproducing the outlines of near-field objects, outperforming model A, but still struggles with complex texture details. The model trained using SynMFLP significantly outperforms the previous two methods, exhibiting extremely high edge fidelity and clearly capturing the geometric details of objects (such as the fine branching structure of corals and the slender rods of detectors), displaying sharp edges closest to those of the ground truth (GT), demonstrating that this method has a stronger ability to preserve high-frequency information. Regarding background smoothness and noise robustness, the background region is often the most severely affected by underwater light scattering, frequently exhibiting artifacts. The model trained using Atlantis shows obvious depth inhomogeneity in the background region, with some noise interference, resulting in a "dirty" background. The model trained using SynTIDE shows improved background processing and reduced noise levels, but some non-uniform textures caused by water quality interference can still be faintly observed. The model trained using SynMFLP exhibits extremely high smoothness and purity in the depth background. Compared with GT, it is evident that this method effectively filters out false depth information caused by underwater scattering, successfully and accurately separating the background from the foreground while maintaining the coherence of the depth gradient.

[0105] In summary, the model trained using SynMFLP demonstrates significant advantages across all three dimensions. It not only achieves the highest consistency with the ground truth (GT), but also achieves optimal performance in preserving complex details and suppressing underwater noise, reflecting the stronger robustness and superiority of its algorithmic architecture in handling underwater light environment degradation and accurate scene perception.

[0106] In the semantic segmentation experiment, to systematically verify the effectiveness of the proposed method in underwater image synthesis, a total of fifteen sets of comparative experiments were constructed. Analysis was conducted from three dimensions: data source, training strategy, and model structure, to ensure the experimental conclusions have sufficient reliability and interpretability.

[0107] In terms of data construction, the training data was divided into five types: SynTIDE (using only the synthetic dataset generated by the TIDE method), SynMFLP (using only the synthetic dataset generated by the method of this invention), USIS (using only the real dataset), and two hybrid strategies (SynTIDE+USIS and SynMFLP+USIS). This setup allows for the independent evaluation of the effects of different data sources and analysis of the role of synthetic data in assisting real data training. Regarding model selection, this invention selects three representative semantic segmentation models: SegFormer (MiT-B4), Mask2Former (Swin-B), and ViT-Adapter (ViT-Adapter-B). These three models represent a lightweight Transformer structure, a unified segmentation framework, and an adapter-based visual Transformer model, respectively. They allow for the verification of the dataset's generalization ability from different modeling paradigms, and the average intersection-over-union ratio (IoU) is used to compare the results. The results are shown in Table 1 (qualitative analysis using different training data and methods on USIS10K): Table 1

[0108] To further validate the quantitative results, this invention provides a visual analysis of the semantic segmentation results predicted by the model, as shown below. Figure 10 As shown (visualization of tests on different models using different datasets): Qualitative analysis reveals a gradual change in model performance from left to right. The results on the left correspond to models employing only a single strategy, such as SynTIDE, SynMFLP, or USIS. These methods can locate the approximate position of the target well, but the overall segmentation results are relatively coarse, with unclear boundaries and obvious misclassification in local areas. As strategies are gradually stacked, the segmentation quality improves after incorporating SynMFLP or a combination of multiple strategies. In the intermediate results, the coherence of the target region is enhanced, the boundary transitions are smoother, and the overall shape closely matches the real structure. The segmentation effect of the red underwater detector shows the most significant difference. This target structure is relatively complex, containing a slender support underneath, which places high demands on segmentation accuracy. At the same time, underwater imaging presents light scattering issues, further blurring the boundaries of the fine structures. In the results of a single strategy or intermediate combinations, varying degrees of missing support structures can be observed, with some results only retaining incomplete main areas, or even blending into the background, and significant boundary adhesion.

[0109] In the rightmost result, where the SynMFLP and USIS combined strategy is employed, the overall segmentation performance is significantly improved. Regardless of the backbone network used, this combined strategy exhibits stable performance, effectively recovering the overall structure of the detector, especially the slender support structure, which is clearly represented in most cases and shows high consistency with the ground truth image. This indicates that the combined strategy has certain advantages in fine-grained structure modeling and, to some extent, mitigates the impact of boundary blurring in underwater images.

[0110] Furthermore, different backbone networks exhibit varying overall performance. Segformer demonstrates relatively stable performance in terms of overall connectivity within the target region, but still exhibits some errors at complex boundaries. Mask2former is more sensitive to the distinction of the target region, and after introducing strong policy combinations, it achieves better reconstruction of the detector structure. ViT-adapter shows relatively stable overall performance, effectively extracting feature information under different policy conditions, with a relatively clean background region and less noise interference.

[0111] Overall, the experimental results intuitively reflect the complexity of underwater semantic segmentation tasks and demonstrate that multi-strategy combinations play a role in improving the accuracy of fine-grained structure segmentation. Compared to single-strategy methods, combined strategies exhibit better stability in complex underwater optical environments and can improve the integrity and detail restoration of target structures to a certain extent. Furthermore, this result also indirectly verifies that the synthetic dataset constructed in this invention has a more significant advantage in improving the model's generalization ability and robustness.

[0112] Specifically, this invention constructs an underwater scene generation method that integrates multimodal prior modeling, structural consistency constraints, semantic enhancement guidance, and controllable generation mechanisms. It addresses the common problems in underwater vision tasks, such as data scarcity, annotation difficulties, and complex environments, and achieves high fidelity and reasonable physical structure of the generated images. At the same time, it supports the large-scale expansion of pixel-level aligned automatic annotation datasets and enables depth estimation and semantic segmentation models trained on this dataset to achieve optimal results.

[0113] To address the problems of severe light attenuation, strong scattering effects, blurred target boundaries, and unclear category representation in underwater environments, this invention effectively improves the consistency of generated images in semantic structure and geometric relationships through multimodal collaborative modeling of semantic maps, depth maps, edge maps, and text semantics. This significantly enhances the usability and reliability of generated data in complex scenarios, ensuring that the generated results have reasonable physical structure and high-fidelity visual performance.

[0114] At the level of multimodal perception and information representation, this invention constructs three types of structural prior information—semantics, depth, and edge—and combines them with textual semantic descriptions to form a unified multimodal input system. This enables the generation process to simultaneously characterize the category distribution, spatial geometric relationships, and boundary structural features of the scene. Compared to methods that rely solely on single-modal information, this invention can accurately recover object contours and spatial layouts, effectively mitigating information loss caused by water turbidity, uneven lighting, and visual degradation. This ensures the structural integrity and visual realism of the generated image, laying the foundation for expanding pixel-level aligned automatic annotation datasets.

[0115] At the level of feature fusion and structural constraints, this invention proposes a multimodal residual conditional fusion mechanism. Primarily driven by semantic information, it introduces attention-enhanced depth and edge features through residuals, thereby suppressing noise interference while strengthening effective structural representation. This mechanism avoids information redundancy and conflicts caused by simple multimodal splicing, enabling complementary relationships between different modalities. It simultaneously ensures semantic consistency and geometric rationality during generation, further enhancing the physical realism of the generated image.

[0116] At the semantic guidance and controllable generation level, this invention introduces a label-guided text enhancement mechanism, embedding degradation level information and semantic category information into text prompts to construct a conditional expression with structured semantic constraints. Compared to traditional natural language prompting methods, this mechanism can reduce the problem of inaccurate semantic expression in underwater degradation environments, enabling the generation model to maintain stable control under different turbidity, lighting conditions, and scene complexity. Furthermore, the generated results can be finely controlled by adjusting the degradation level and semantic input, achieving diversified automatic annotation dataset expansion.

[0117] Regarding the generation quality and model stability, this invention guides the generated results to converge towards the true distribution during the gradual denoising process of the diffusion model through joint constraints of multimodal conditions and semantic enhancement cues, thereby improving the visual realism and structural alignment of the generated images. Based on the pixel-level aligned dataset generated by this invention, the model performance reached its optimal effect when training the depth estimation and semantic segmentation models, verifying the significant value of this invention in underwater visual task data augmentation.

[0118] In terms of engineering applications and system expansion, the method of this invention has obvious modularity, with clear interfaces for each functional module, low coupling, and flexible combination. It can also access different types of semantic tags and environmental parameters, and has good versatility. It can be widely used in practical scenarios such as underwater exploration, visual perception of unmanned underwater vehicles, and marine environmental monitoring.

[0119] In summary, this invention effectively achieves high fidelity and physical structural rationality of generated images through multimodal information collaborative modeling, structural consistency constraints, and semantic enhancement guidance mechanisms. It supports large-scale expansion of pixel-level aligned datasets and achieves optimal performance when training depth estimation and semantic segmentation models, demonstrating significant engineering application value and promising prospects for wider adoption.

[0120] like Figure 11 As shown, this embodiment of the invention also provides an underwater scene generation device 900 based on multimodal fusion and tag-guided prompts, which can implement the above-mentioned method. This device may include: The first module 901 is used to acquire underwater images and construct multimodal data representations; wherein, the multimodal data representations include underwater images and their corresponding semantic maps, depth maps, edge maps and initial text descriptions; The second module 902 is used to process the semantic map, depth map and edge map, and use the multimodal residual conditional fusion module to obtain the fusion conditional features in sequence through independent encoding, spatial channel collaborative attention enhancement and semantic-dominated residual fusion. The third module 903 is used to classify water body types based on underwater images, and then combine semantic graphs and initial text descriptions to construct enhanced text prompts. Module 4, 904, is used to generate synthetic underwater images using a latent space diffusion model based on fusion conditional features and enhanced textual cues.

[0121] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0123] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] like Figure 12 As shown, Figure 12 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0125] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0127] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0128] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0129] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0130] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The underwater scene generation method, apparatus, electronic device, storage medium, and program product based on multimodal fusion and tag-guided prompts provided in this invention embodiment acquires underwater images and constructs a multimodal data representation. The multimodal data representation includes the underwater image and its corresponding semantic map, depth map, edge map, and initial text description. Based on the semantic map, depth map, and edge map, a multimodal residual conditional fusion module is used to sequentially process the data through independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion to obtain fusion conditional features. Water body type classification is performed based on the underwater image, and then enhanced text prompts are constructed by combining the semantic map and initial text description. Based on the fusion conditional features and enhanced text prompts, a synthetic underwater image is generated using a latent space diffusion model. This invention introduces semantic maps, depth maps, and edge maps through a multimodal residual conditional fusion module, and combines them with a latent space diffusion model to generate high-quality underwater images. This significantly reduces the reliance on expensive measured data and meticulous manual annotation. The introduction of the edge map effectively enhances semantic boundary consistency and alleviates pixel-level annotation ambiguity. Furthermore, based on water body type classification and enhanced text prompts, this invention can flexibly simulate the optical characteristics of different water bodies, effectively compensating for the limitations of simplified traditional physical models and significantly narrowing the domain gap between synthesized images and real-world scenes. Similarly, this invention utilizes spatial channel collaborative attention and semantically driven residual fusion to ensure a high degree of coupling between the generated image and the input depth, semantics, and other multimodal conditions. The latent space diffusion model avoids mode collapse and effectively maintains the realism of physical logic and texture details at high resolution.

[0132] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0133] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0136] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. An underwater scene generation method based on multi-modal fusion and label guided prompt, characterized in that, The method includes the following steps: Acquire underwater images and construct a multimodal data representation; wherein, the multimodal data representation includes the underwater images and their corresponding semantic map, depth map, edge map and initial text description; Based on the semantic map, the depth map, and the edge map, the fusion condition features are obtained by sequentially processing independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion using a multimodal residual condition fusion module. Based on the underwater images, water body types are classified, and then enhanced text prompts are constructed by combining the semantic map and the initial text description. Based on the fusion condition features and the enhanced text prompts, a synthetic underwater image is generated using a latent space diffusion model.

2. The method of claim 1, wherein, The construction of the multimodal data representation includes the following steps: Based on the underwater image, the semantic map is obtained by aggregating instance-level annotation information. The underwater image is forward-inferred using a pre-trained depth estimation model to obtain the depth map; The edge map is obtained by extracting the image boundary information of the underwater image through an edge detection network. Based on the underwater image, the corresponding initial text description is generated using a visual language model.

3. The method of claim 1, wherein, The process of obtaining fused conditional features based on the semantic map, the depth map, and the edge map, using a multimodal residual conditional fusion module, involves sequential independent encoding, spatial channel collaborative attention enhancement, and semantically dominated residual fusion, and includes the following steps: Based on the semantic graph, high-dimensional semantic features are extracted using the first encoder; Based on the depth map, high-dimensional depth features are extracted using a second encoder. Based on the edge map, high-dimensional edge features are extracted using a third encoder; The high-dimensional depth features and the high-dimensional edge features are preprocessed, and then feature enhancement is performed through the spatial channel collaborative attention module to obtain the corresponding depth enhancement features and edge enhancement features. Using the semantic high-dimensional features as the dominant benchmark, and combining the depth enhancement features and the edge enhancement features, residual fusion is performed to obtain the fusion condition features.

4. The method according to claim 3, characterized in that, The process of preprocessing the high-dimensional depth features and the high-dimensional edge features, and then enhancing the features through a spatial channel collaborative attention module to obtain the corresponding enhanced depth features and enhanced edge features, includes the following steps: Based on the aforementioned deep high-dimensional features, the deep preprocessed features are obtained by summing the results of all parallel processing element by element through parallel first convolution, second convolution, and batch normalization. Based on the aforementioned high-dimensional edge features, the edge preprocessing features are obtained by summing the results of all parallel processing element by element through parallel first convolution, second convolution, and batch normalization. Based on the aforementioned deep preprocessed features, feature enhancement is performed using a spatial channel collaborative attention module to obtain deep enhanced features; Based on the aforementioned edge preprocessing features, feature enhancement is performed using a spatial channel collaborative attention module to obtain edge enhancement features.

5. The method according to claim 3, characterized in that, The step of using the semantic high-dimensional features as the dominant benchmark, combining the depth enhancement features and the edge enhancement features to perform residual fusion to obtain the fusion conditional features includes the following steps: The semantic high-dimensional features, the depth enhancement features, and the edge enhancement features are added element-wise to obtain preliminary fusion features; The preliminary fused features are aligned and compressed using a preset mapping function to obtain joint features; wherein the mapping function is constructed by stacking multiple layers of convolutional kernels. The semantic high-dimensional features are joined with the joint features by performing element-wise addition of residual connections to obtain the fusion conditional features.

6. The method according to claim 1, characterized in that, The process of classifying water bodies based on the underwater image and then constructing enhanced text prompts by combining the semantic map and the initial text description includes the following steps: The underwater image and the preset water body classification prompt template are input into the visual big language model to determine the water body type and obtain the initial water body type. Obtain the evaluation voting results of all annotators regarding the initial water body type; If the proportion of agreement in the evaluation voting results is greater than a preset proportion threshold, the initial water body type is taken as the target water body type; otherwise, the underwater image is sent to a domain expert to obtain the final decision of the domain expert as the target water body type. Based on the target water body type, the initial text description, and the label category corresponding to the semantic graph, the enhanced text prompt is obtained by fusing them using a fixed template.

7. The method according to claim 1, characterized in that, The process of generating synthetic underwater images based on the fusion condition features and the enhanced text prompts using a latent space diffusion model includes the following steps: The underwater image is constructed by progressively adding Gaussian noise to the underwater image using the aforementioned latent space diffusion model through a forward process; In the reverse process of the underwater diffusion model, the fusion condition features and the enhanced text prompts are used as joint conditions to progressively remove noise from the noise distribution to restore the image content, thereby obtaining the synthetic underwater image.

8. An underwater scene generation device based on multimodal fusion and tag-guided prompts, characterized in that, The device includes: The first module is used to acquire underwater images and construct a multimodal data representation; wherein, the multimodal data representation includes the underwater images and their corresponding semantic map, depth map, edge map and initial text description; The second module is used to process the semantic map, the depth map and the edge map, and use the multimodal residual conditional fusion module to sequentially process independent encoding, spatial channel collaborative attention enhancement and semantic-dominated residual fusion to obtain fusion conditional features. The third module is used to classify water body types based on the underwater image, and then combine the semantic map and the initial text description to construct an enhanced text prompt. The fourth module is used to generate synthetic underwater images based on the fusion condition features and the enhanced text prompts using a latent space diffusion model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images