Underwater scene multi-modal generation method based on diffusion model

Abstract

This invention, a multimodal underwater scene generation method based on a diffusion model (UMDM-USG), belongs to the field of computer vision and image generation technology. Addressing the high cost of underwater scene data acquisition, this invention first encodes the features of the input text and multimodal data. Second, it randomly divides the modalities into generative and conditional modalities through a role assignment module. Then, it constructs a generative conditional alignment attention module in the multimodal alignment module to achieve bidirectional interaction and alignment between the generative and conditional modalities. Finally, it outputs the results of each modality through a decoder. Furthermore, a representation alignment regularization strategy is introduced, utilizing the visual prior of a pre-trained self-supervised model as a supervision signal to guide the generation distribution towards a realistic scene, significantly improving geometric fidelity. Experimental results show that this invention outperforms existing methods in terms of generation quality, semantic consistency, and multimodal collaboration, and can provide reliable data support for downstream vision tasks.

Description

Technical Field

[0001] This invention relates to a method for generating multimodal underwater scenes based on a diffusion model, belonging to the field of computer vision and image generation technology. Background Technology

[0002] With the development of computer vision technology, underwater data has significant application value in fields such as ocean exploration, underwater robotics, and autonomous navigation. However, due to the complex and variable underwater environment, severe light attenuation, and significant scattering effects, high-quality data acquisition is costly and difficult. Existing underwater datasets are limited in size, making it difficult to meet the large-scale data requirements of deep learning models. Furthermore, underwater scene data typically includes multiple modalities such as images, semantic segmentation, depth information, and surface normals, which are strongly correlated at the semantic and geometric levels. Existing methods often model single modalities or use loosely coupled approaches to process multimodal data, making it difficult to achieve unified modeling and resulting in deficiencies in structural and semantic consistency of the generated results. In recent years, diffusion models have made significant progress in image generation, but existing diffusion models are mostly applied to single-modal generation tasks, lacking the ability to uniformly model multimodal information. Therefore, there is an urgent need for an underwater scene generation method that can uniformly generate multimodal information and enhance structural consistency. Summary of the Invention

[0003] The purpose of this invention is to provide a diffusion model-based underwater scene multimodal generation method (UMDM-USG) to solve the problems of difficulty in acquiring underwater data, insufficient multimodal annotation, and poor cross-modal consistency of generation results in the prior art.

[0004] The underwater scene multimodal generation method based on the diffusion model is characterized by the following steps: (1) Collect multiple raw underwater images, and generate depth maps, semantic segmentation maps, surface normal maps and text information from these images to obtain multiple quintuple multimodal data.

[0005] The original underwater image can also be obtained from various underwater datasets. The original underwater image, along with its depth map, semantic segmentation map, and surface normal map, are collectively referred to as the visual modality, while the text information is referred to as the text modality.

[0006] (2) The quintuple multimodal features are digitized by using a text encoder and a visual encoder. This process transforms the image and text into a usable digital structure.

[0007] (3) The five modes of each quintuple multimodal are randomly divided into generating mode and conditional mode through the role assignment module. There is no less than one generating mode and one conditional mode. The text mode is used only as the conditional mode, the generating mode is used as the prediction target of the diffusion model, and the conditional mode is used to provide conditional constraint information for the generating mode.

[0008] The generating modes of all quintuple multimodal data constitute the generating mode set, and the conditional modes of all quintuple multimodal data constitute the conditional mode set.

[0009] (4) Input the set of generated modes and the set of conditional modes into the multimodal alignment module, wherein the multimodal alignment module includes: (4.1) Global alignment attention, used to uniformly model the various non-textual modalities of the input to obtain global alignment features; (4.2) Generate conditional alignment attention, used to construct joint generative conditional features between generative modalities and conditional modalities; (4.3) Gated fusion module, used to perform weighted fusion of the global alignment features generated in (4.1) and the joint features generated by the generation conditions generated in (4.2) to generate multimodal features; (4.4) Cross attention, when the conditional modality contains text, is used to inject text semantic information into the multimodal features to obtain a joint representation.

[0010] (5) Input the multimodal features or joint representations into a multilayer perceptron (MLP) to separate each modality, and then input them into a decoder to output the decoding results corresponding to the generated modalities.

[0011] (6) The underwater scene multimodal model is obtained by training through steps (1)-(5).

[0012] (7) Deployment of the model: Once training is complete, in practical applications, inputting any one of the conditional modalities from the quintuple multimodal model will result in the model outputting modalities other than the conditional modality and the text modality.

[0013] In step (1), the depth map, semantic segmentation map, and surface normal map are generated by processing the original underwater image through a pre-trained depth estimation model, semantic segmentation model, and surface normal estimation model, respectively; the text information is generated by inputting the original underwater image into a pre-trained large language model; thus, multiple five-tuple multimodal data containing the original image, semantic segmentation map, depth map, surface normal map, and text information are constructed.

[0014] In the process of constructing multimodal data in step (1), the pre-trained depth estimation model uses Priorda and VGGT to extract the geometric structure information of the image, the semantic segmentation model uses WaterMask, WaterSAM and SPDA-SAM to obtain pixel-level semantic category information, and the surface normal estimation model uses StableNormal, DSINE and Lotus to describe the spatial orientation information of the object surface.

[0015] The text encoder in step (2) uses Gemma3, and the visual encoder uses a deep compression autoencoder.

[0016] Step (4) addresses the problem of insufficient cross-modal structural consistency by constructing a multimodal alignment module to achieve joint modeling and information interaction among multimodal structures, wherein: (4.1) Global alignment attention: In each quintuple multimodal data, the latent features of each modality other than text are concatenated along the token dimension to construct a unified global sequence representation. A linear self-attention mechanism is then used to model this sequence in order to capture the long-range dependencies between different modalities, thereby obtaining global alignment features. (4.2) Generate conditionally aligned attention: Based on the role assignment results, cross-modal attention is constructed using the generative modality feature as the query and the conditional modality feature as the key and value, thereby achieving alignment between the generative modality and the conditional modality. At the same time, a reverse alignment relationship is constructed using the conditional modality as the query and the generative modality as the key and value, thus forming a bidirectional alignment of generative and conditional joint features between the generative and conditional modalities (i.e., establishing a cross-modal alignment relationship). (4.3) Gating fusion module: Adaptive weighted fusion of global alignment features and joint features generated by conditions is performed, and the weight parameters are dynamically adjusted through backpropagation to obtain multimodal features; (4.4) Text cross-attention: By performing cross-attention calculations on the text encoder output and multimodal features, textual semantic information is injected into the multimodal representation, achieving semantic alignment and obtaining a joint representation.

[0017] The decoder in step (5) is a deep compression self-decoder.

[0018] In step (6), during training, the model optimization process uses a joint loss function of reconstruction loss and representation alignment regularization to train the model, where: The reconstruction loss is used to constrain the prediction accuracy of the diffusion model in the denoising process, so that the generated result is close to the real data distribution in the pixel space or potential representation space. The representation alignment regularization is used to impose constraints on the intermediate features of the model. By introducing a pre-trained visual feature extraction model, feature representations of real multimodal data are obtained, and the hidden features of the model are mapped to the same feature space, so that the two are consistent at the feature level, thereby guiding the model to learn structural representations that conform to the statistical characteristics of natural scenes. The joint loss is obtained by combining the reconstruction loss and the representation alignment regularization in a weighted manner to balance the generation accuracy and the consistency of the multimodal structure.

[0019] Step (7) is as follows: After training is completed, in practical applications, input the five or fewer combinations of modalities in the quintuple multimodal as conditional modalities, and the text modalities are only used as conditional modalities. Other than the conditional modalities are used as generative modalities. Execute (2) to digitize each conditional modality, concatenate them, and execute (4.1) global alignment attention to obtain global alignment features. Then execute (4.2) to generate conditional alignment attention, using the conditional modalities as keys and values, and the generative modalities corresponding to the conditional modalities as queries, to obtain generative conditional joint features. Then, through the gating fusion module in step (4.3), the global alignment features and generative conditional joint features are weighted and fused to generate multimodal features. When there is text information, execute (4.4) cross attention to introduce semantic information to obtain joint representations. Finally, input the multimodal features or joint representations into the multilayer perceptron to separate each modality, and then input them into the decoder to output the decoding results corresponding to the generative modalities, thereby realizing multimodal generation based on various conditional modalities.

[0020] The method is applied to the generation of multimodal data for underwater scenes. By taking any one of the conditional modalities in the five-tuple multimodal data as input, it generates the modalities other than the text modality, which are then used as supplementary or auxiliary data for underwater vision tasks.

[0021] Advantages of the invention Compared with existing technologies, this invention has significant advantages. Based on a unified diffusion framework, this invention jointly models multimodal information such as image, semantic segmentation, depth, and surface normals in underwater scenes. This avoids the structural fragmentation and insufficient information utilization problems caused by the use of add-on or independent modal alignment modules in traditional methods. By constructing a unified multimodal alignment module, it achieves collaborative alignment of all modalities in the same feature space, thereby significantly improving structural and semantic consistency across modalities. Simultaneously, this invention introduces generative conditional alignment attention, explicitly modeling the bidirectional dependency between generative and conditional modalities. This allows the generation process to effectively utilize structural constraint information in the conditional modality, further improving the geometric rationality and detail representation of the generated results. Furthermore, a gating fusion mechanism adaptively balances global alignment information and local conditional constraint information, enabling the model to simultaneously maintain overall structural consistency and local detail representation capabilities in complex underwater scenes. Moreover, this invention introduces a representation alignment regularization strategy, constraining the intermediate representation of the model through feature priors provided by a pre-trained visual model. This makes the generated distribution closer to real-world natural scenes, thereby improving model training stability and accelerating convergence.

[0022] In summary, this invention can not only effectively improve the quality and consistency of multimodal generation results, but also generate high-quality multimodal data, providing reliable data support for downstream tasks such as underwater semantic segmentation, depth estimation, and normal estimation, and has good application prospects and promotion value. Attached Figure Description

[0023] Figure 1 This is the overall architecture and process framework of the UMDM-USG of the present invention.

[0024] Figure 2 This is the multimodal alignment module in this invention.

[0025] Figure 3 This is a demonstration of the multimodal generation results of the present invention. Detailed Implementation

[0026] The complete process framework of this invention is as follows: Figure 1 As shown. To facilitate understanding of the textual descriptions in the embodiments of this invention by those skilled in the art, some English texts in the patent application have been translated into Chinese. For example, Figure 1The Chinese text reads, "A multitude of silvery fish swim together in the deep blue ocean, creating a captivating underwater scene." The original English text describes it as, "A multitude of silvery fish swim together in the deep blue ocean, creating a captivating underwater display." Figure 3 The Chinese text description reads, "A small, vibrant orange butterflyfish gracefully swims through the underwater coral reef, showcasing its distinctive blue and yellow markings." The original English description is, "A small, vibrant orange butterflyfish gracefully swims through the underwater coral reef, showcasing its distinctive blue and yellow markings." It should be noted that the above Chinese content is for patent document explanation and understanding only; in actual invention experiments and model training, English text was used for input.

[0027] A multimodal generation method for underwater scenes based on a diffusion model includes: Step 1. Multimodal Data Construction. Acquire underwater scene image data, generate depth maps using pre-trained depth estimation models (Priorda and VGGT), generate semantic segmentation maps using semantic segmentation models (WaterMask, WaterSAM, and SPDA-SAM), generate normal maps using surface normal estimation models (StableNormal, DSINE, and Lotus), generate corresponding text using a large language model (Gemma3), and perform alignment processing on each modality.

[0028] Step 2: Feature Encoding. The input text is encoded using a text encoder, and features are extracted from the multimodal data using a visual encoder to obtain a unified latent representation for digitizing each modality.

[0029] Step 3: Role Assignment. The role assignment module is used to divide the multimodalities and determine the generative modality and the conditional modality.

[0030] Step 4: Multimodal alignment. For example... Figure 2 As shown, in the multimodal alignment module, shared features are first obtained through global alignment attention; then, bidirectional interaction between the generative modality and the conditional modality is constructed through generative conditional alignment attention; features are then fused through the gating fusion module; and finally, textual semantic information is introduced through cross attention.

[0031] Step 5: Decoding and Generation. Input the fused features into the decoder to generate the image, semantic segmentation map, depth map, and surface normal map.

[0032] Step 6: Model training. The underwater scene multimodal model is obtained through training in steps (1)-(5).

[0033] Step 7: Model Deployment. During training, noise is modeled using a diffusion model, and the model is optimized using reconstruction loss and representation alignment regularization.

[0034] The following specific examples will further illustrate this point.

[0035] Example The core of this invention lies in constructing an underwater multimodal scene generation method based on a diffusion model, and designing a multimodal alignment module to achieve structural constraints and semantic coordination between modalities, thereby generating highly consistent multimodal data. The implementation of this invention will be described in detail below with reference to the accompanying drawings and specific procedures.

[0036] 1. Image data acquisition and multimodal preprocessing First, raw RGB image data is acquired from multiple publicly available underwater datasets. Due to the high cost of acquiring multimodal annotations in underwater environments, this invention employs a "generative annotation" approach to construct multimodal data. Specifically, for the acquired RGB images, multiple pre-trained models are invoked for inference: a large language model (Gemma3) generates corresponding text; depth estimation models (such as Priorda and VGGT) process the input images to generate corresponding depth maps, reflecting the distance relationships and spatial structure information between objects in the scene; on the other hand, semantic segmentation models (such as WaterMask, WaterSAM, and SPDA-SAM) are used to obtain semantic segmentation results, thereby describing the semantic region distribution of different target categories; simultaneously, surface normal estimation models (such as StableNormal, DSINE, and Lotus) generate normal maps to characterize the geometric orientation and detailed changes of object surfaces.

[0037] Subsequently, the four modalities other than the text modality are normalized and spatially aligned to ensure strict pixel-level correspondence between the different modalities, thereby constructing a multimodal data representation other than the text modality: .

[0038] 2. Feature Encoding and Latent Space Mapping To reduce computational complexity and achieve unified modeling, this invention employs a deep compression autoencoder to encode multimodal data and map it to a shared latent space for data digitization. For the m-th modality, its latent representation is: ，L m Let C be the dimension of the m-th modality, and C be the number of layers.

[0039] 3. Role Assignment To achieve arbitrary modal combination generation, this invention introduces a role allocation module, which dynamically divides modal roles during training: In this context, the text element (Text) is used only as a conditional modality. To generate modes, For conditional modes, For the visual modality, diffuse noise (Gaussian noise) is introduced for the generated modality. For conditional modalities, the conditional modality remains unchanged. The advantage of this mechanism is that the model does not need to design separate generation paths for different modal combinations. Instead, it learns the conditional dependencies between multimodalities through a unified training strategy, thereby supporting arbitrary modal combination inputs during the inference phase.

[0040] 4. Multimodal alignment module The multimodal alignment module is the core of this invention. Its function is to achieve structural association modeling and semantic collaborative alignment between different modalities during the unified diffusion modeling process. For example... Figure 2 This module mainly comprises four components: global alignment attention, generative conditional alignment attention, gating fusion mechanism, and textual cross-attention. First, in the global alignment stage, latent features from all modalities are concatenated along the sequence dimension, and cross-modal global dependencies are modeled using a linear attention mechanism. This process effectively captures the overall correlation between different modalities, enabling the model to obtain a unified global semantic representation. However, relying solely on global attention is insufficient to characterize the directional dependencies between the generative and conditional modalities. Therefore, this invention further designs a generative-conditional aligned attention mechanism. In this mechanism, the generative modality is treated as a query, and the conditional modality as a key and value. Through cross-modal attention computation, the generative modality can proactively extract structural information from the conditional modality. Simultaneously, to enhance consistency, a reverse alignment process is introduced, enabling the conditional modality to perceive changes in the generative modality during joint modeling. After completing bidirectional alignment, the alignment result is fused with the original features using a residual approach, resulting in a more stable and constrained joint representation. .

[0041] Subsequently, this invention introduces a gating fusion mechanism to perform global alignment features. Features aligned with generation conditions Weighted fusion is performed. This mechanism can adaptively adjust the importance of different information sources according to the current input conditions, enabling the model to maintain overall structural consistency while enhancing the ability to express local details in complex scenarios.

[0042] Finally, the semantic information of the text is injected into the fused multimodal features through a cross-attention mechanism, so that the generated result can maintain semantic consistency with the input text description while satisfying structural constraints.

[0043] 5. Decoding and Generation After multimodal alignment, the fused unified latent representation is input into the decoder for reconstruction. The decoder maps the latent features back to each modal space, generating images, semantic segmentation maps, depth maps, and surface normal maps, respectively. Figure 3 Examples of generating modalities are shown, using text-only, depth map + normal map, normal map + segmentation map, and image-only as examples, after inputting the UMDM-USG model of this invention.

[0044] 6. Model Training and Optimization Strategies During model training, this invention employs joint optimization of the standard reconstruction loss of the diffusion model and representation alignment regularization. The diffusion reconstruction loss constrains the model's ability to predict noise, gradually approximating the generated results to the true data distribution. Its basic form is as follows: ,in, Cross-entropy loss, Original sample, noise, The time step indicates which step of the diffusion process is being performed. For the model based on The loss is influenced by temporal samples and conditional prediction noise. However, relying solely on this loss can easily lead to structural instability in the model. Therefore, this invention further introduces representation alignment regularization, which extracts feature representations of real data through a pre-trained visual model and aligns them with the features of the model's intermediate layers. This process guides the model to learn feature representations that better conform to the statistical regularities of natural scenes, thereby improving the geometric plausibility and cross-modal consistency of the generated results. Ultimately, the overall optimization objective of the model is a weighted combination of both, achieving a balance between generation accuracy and structural consistency.

[0045] 7. Experimental Verification and Effect Evaluation To verify the effectiveness of the proposed diffusion model-based underwater scene multimodal generation method, multiple evaluation metrics were uniformly used to quantitatively analyze the generation results during the experimental evaluation process to ensure comparability between different models. These metrics included Fréchet Inception Distance (FID), Natural Image Quality Evaluator (NIQE), Visual Question Answering Score (VQA Score), and Human Preference Score (HPSv2.1). Regarding the dataset, the experiment was conducted based on the constructed U-TMDN underwater multimodal dataset. Several representative generation models were selected as comparison methods, including general image generation models (SD3-Medium, SD-XL, Pixart-σ, Infinity, Jodi, Sana-1600M) and underwater scene-related methods (Atlantis, TIDE). All models were tested under the same evaluation protocol to ensure the fairness of the experimental results.

[0046] The results demonstrate that the method of this invention achieves superior performance across multiple metrics. Specifically, in terms of image quality, the method achieves the lowest FID score, indicating that the generated image distribution is closer to the real data; it also performs well in the NIQE metric, demonstrating higher naturalness and visual quality in the generated results. Regarding semantic consistency, the method outperforms most comparative methods in the VQA score, indicating that the generated content is better consistent with the input text description. Furthermore, the method achieves a high score in the HPSv2.1 metric, suggesting that the generated results are more in line with human preferences in overall visual perception. In summary, the method of this invention outperforms or is at least comparable to existing mainstream methods in terms of generation quality, semantic consistency, and overall visual effect, validating the effectiveness of the proposed multimodal alignment mechanism in unified generation tasks.

[0047] Table 1. Scores of the method of the present invention compared to other prior art indicators. .

[0048] 8. Downstream task validation and auxiliary training effect To further verify the practical application value of this invention in multimodal consistency modeling and data augmentation, the generated multimodal data is introduced into underwater vision downstream tasks for auxiliary training, including underwater semantic segmentation, depth estimation, and surface normal estimation. In the specific implementation, the model of this invention is first used to generate large-scale multimodal data samples, which include images, semantic segmentation maps, depth maps, and normal maps, and together with real labeled data, form an extended training set. During the model training phase, the generated data is used as auxiliary samples mixed with real data to improve the model's generalization ability to complex underwater scenes.

[0049] In the underwater depth estimation task, this invention is introduced to generate a fine-tuning dataset, and the AdaBins, NewCRFs, PixelFormer, and MIM methods are trained. Experiments are conducted on the Sea-thru and SQUID datasets. Based on the standard evaluation protocol, the absolute relative error (Abs Rel) is used as the evaluation metric. The experimental results show that the model has significantly improved the depth prediction accuracy. Compared with existing fine-tuning datasets, it exhibits its own strengths on different test datasets, indicating that it has achieved the same effect as existing technologies.

[0050] Table 2 shows the improvement effect of this invention on the model in underwater depth estimation tasks. .

[0051] In underwater semantic segmentation tasks, this invention generates a fine-tuning dataset to train three methods: SegFormer, Mask2Former, and ViT-Adapter. The datasets are evaluated on the UIIS and USIS10K datasets, with mean Intersection over Union (mIoU) used as the evaluation metric. Results show that using the multimodal data generated by this invention for auxiliary training effectively improves the target recognition capability and boundary segmentation accuracy of the segmentation model in complex underwater environments. Furthermore, compared to existing fine-tuning datasets, it exhibits distinct strengths on different test datasets, demonstrating that it achieves equivalent performance to existing technologies.

[0052] Table 3 shows the improvement effect of this invention on the model in underwater semantic segmentation tasks. .

[0053] In the surface normal estimation task, this invention generates a fine-tuning dataset to test three methods: StableNormal, DSINE, and Lotus, using Mean Absolute Error (MAE) as the evaluation metric. Experimental results show that this method significantly improves the mean angle error. Since high-quality labeled data is scarce, the data generated by this invention provides additional geometric supervision information for the model, thereby effectively improving the stability and accuracy of normal estimation.

[0054] Table 4 shows the improvement effect of this invention on the model in the surface normal estimation task. .

[0055] Furthermore, by comparing the performance of models trained using only real data with those trained using generated data, it can be found that in all downstream tasks, the data generated by this invention can achieve better results. This indicates that this invention can not only generate high-quality multimodal data, but also serve as an effective data augmentation method to improve the performance of various underwater vision tasks.

Claims

1. A method for multi-modal generation of underwater scenes based on diffusion model, characterized in that Includes the following steps: (1) Acquire multiple raw underwater images, and generate depth maps, semantic segmentation maps, surface normal maps and text information from these images to obtain multiple quintuple multimodal data; (2) The quintuple multimodal features are digitized by using a text encoder and a visual encoder; (3) The five modes of each quintuple multimodal are randomly divided into generating mode and conditional mode through the role assignment module. There is no less than one generating mode and one conditional mode. The text mode is used only as the conditional mode, the generating mode is used as the prediction target of the diffusion model, and the conditional mode is used to provide conditional constraint information for the generating mode. (4) Input the set of generated modes and the set of conditional modes into the multimodal alignment module, wherein the multimodal alignment module includes: (4.1) Global alignment attention, used to uniformly model the various non-textual modalities of the input to obtain global alignment features; (4.2) Generate conditional alignment attention, used to construct joint generative conditional features between generative modalities and conditional modalities; (4.3) Gated fusion module, used to perform weighted fusion of the global alignment features generated in (4.1) and the joint features generated by the generation conditions generated in (4.2) to generate multimodal features; (4.4) Cross attention, when the conditional modality contains text, is used to inject the semantic information of the text into the multimodal features to obtain a joint representation; (5) Input the multimodal features or joint representations into a multilayer perceptron (MLP) to separate each modality, and then input them into a decoder to output the decoding results corresponding to the generated modalities; (6) A multimodal model of the underwater scene is obtained through training in steps (1)-(5); (7) Deployment of the model: Once training is complete, in practical applications, inputting any one of the conditional modalities from the quintuple multimodal model will result in the model outputting modalities other than the conditional modality and the text modality.

2. The method of claim 1 wherein In step (1), the depth map, semantic segmentation map, and surface normal map are generated by processing the original underwater image through a pre-trained depth estimation model, semantic segmentation model, and surface normal estimation model, respectively; the text information is generated by inputting the original underwater image into a pre-trained large language model; thus, multiple five-tuple multimodal data containing the original image, semantic segmentation map, depth map, surface normal map, and text information are constructed.

3. The method according to claim 1, characterized in that... In the process of constructing multimodal data in step (1), the pre-trained depth estimation model uses Priorda and VGGT to extract the geometric structure information of the image, the semantic segmentation model uses WaterMask, WaterSAM and SPDA-SAM to obtain pixel-level semantic category information, and the surface normal estimation model uses StableNormal, DSINE and Lotus to describe the spatial orientation information of the object surface.

4. The method according to claim 1, characterized in that... The text encoder in step (2) uses Gemma3, and the visual encoder uses a deep compression autoencoder.

5. The method according to claim 1, characterized in that... Step (4) addresses the problem of insufficient cross-modal structural consistency by constructing a multimodal alignment module to achieve joint modeling and information interaction among multimodal structures, wherein: (4.1) Global alignment attention: In each quintuple multimodal data, the latent features of each modality other than text are concatenated along the token dimension to construct a unified global sequence representation. A linear self-attention mechanism is then used to model this sequence in order to capture the long-range dependencies between different modalities, thereby obtaining global alignment features. (4.2) Generate conditionally aligned attention: Based on the role assignment results, cross-modal attention is constructed using the generative modality feature as the query and the conditional modality feature as the key and value to achieve alignment between the generative modality and the conditional modality; at the same time, a reverse alignment relationship is constructed using the conditional modality as the query and the generative modality as the key and value, thereby forming a bidirectional aligned generative-conditional joint feature between the generative modality and the conditional modality. (4.3) Gating fusion module: Adaptive weighted fusion of global alignment features and joint features generated by conditions is performed, and the weight parameters are dynamically adjusted through backpropagation to obtain multimodal features; (4.4) Text cross-attention: By performing cross-attention calculations on the text encoder output and multimodal features, textual semantic information is injected into the multimodal representation, achieving semantic alignment and obtaining a joint representation.

6. The method according to claim 1, characterized in that... The decoder in step (5) is a deep compression self-decoder.

7. The method according to claim 1, characterized in that... In step (6), during training, the model optimization process uses a joint loss function of reconstruction loss and representation alignment regularization to train the model, where: The reconstruction loss is used to constrain the prediction accuracy of the diffusion model in the denoising process, so that the generated result is close to the real data distribution in the pixel space or potential representation space. The representation alignment regularization is used to impose constraints on the intermediate features of the model. By introducing a pre-trained visual feature extraction model, feature representations of real multimodal data are obtained, and the hidden features of the model are mapped to the same feature space, so that the two are consistent at the feature level, thereby guiding the model to learn structural representations that conform to the statistical characteristics of natural scenes. The joint loss is obtained by combining the reconstruction loss and the representation alignment regularization in a weighted manner to balance the generation accuracy and the consistency of the multimodal structure.

8. The method according to claim 1, characterized in that... Step (7) is as follows: After training is completed, in practical applications, input the five or fewer combinations of modes in the quintuple multimodal input as conditional modes, and the text mode is only used as a conditional mode, and the others are used as generative modes. Execute (2) to digitize each conditional mode, concatenate them and execute (4.1) global alignment attention to obtain global alignment features, and then execute (4.2) to generate conditional alignment attention. Use the conditional mode as the key and value, and the generative mode corresponding to the conditional mode as the query to obtain the generated conditional joint features. The global alignment features and the joint features of the generation conditions are weighted and fused through the gating fusion module in step (4.3) to generate multimodal features; when there is text information, cross attention (4.4) is performed to introduce semantic information to obtain joint representation; finally, the multimodal features or joint representation are input into the multilayer perceptron to separate each modality, and then input into the decoder to output the decoding result corresponding to the generated modality, thereby realizing multimodal generation based on various condition modalities.

9. The application of the method according to claim 1, characterized in that, The method is applied to the generation of multimodal data for underwater scenes. By taking any one of the conditional modalities in the five-tuple multimodal data as input, it generates the modalities other than the text modality, which are then used as supplementary or auxiliary data for underwater vision tasks.

Images