Multi-modal data generation method driven by large model

Abstract

The invention provides a large-model-driven multi-modal data generation method, which comprises the following steps of: firstly, carrying out demand analysis and scene definition on a data product, and defining a multi-modal data demand; then, by taking the large language model as a baseline, designing an expansion framework supporting multi-modal output; then, constructing a cue word project, and designing a cue word template for a special field; field information is retrieved through an RAG technology, and cue words are dynamically enhanced; integrating a condition generation network model to realize accurate control on output data parameters; and finally, carrying out automatic evaluation and manual sampling verification on the generated data. According to the method, high-quality and large-batch multi-field and multi-modal data generation is realized through hierarchical prompt word engineering, dynamic field knowledge supplementation, a conditional generation network and an automatic evaluation and feedback mechanism, and the method has the advantages of high professionality and accuracy, fine parameter control, good output consistency and collaboration, perfect evaluation and optimization system and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence data generation, and particularly relates to a large model-driven multi-modal data generation method. Background Art

[0002] In the field of artificial intelligence, data is the key driving force for technological development and the foundation for model training, verification, and application. High-quality large-scale data can not only improve the performance of models but also significantly expand their application scenarios. However, in many fields and modalities, problems such as data scarcity, collection costs, and privacy restrictions have hindered the further development of AI technology. Therefore, the research and application of data generation technology have become an important way to break through this bottleneck.

[0003] Data generation models can generate high-quality and realistic synthetic data by simulating the distribution of real data, providing richer training samples for models. These models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), have achieved certain results in tasks such as image generation, text generation, and speech synthesis by learning the latent data distribution.

[0004] The multi-domain multi-modal data generation method is a further extension based on this, aiming to simultaneously process and generate data from multiple domains and modalities. For example, the medical field needs to combine patient medical records (text), imaging data (images), and diagnostic speech (audio), while the transportation field involves real-time traffic data (numerical data), drone surveillance footage (images), and vehicle signals (audio). Such methods learn the commonalities and differences between different domains and modalities by training on multi-domain and multi-modal data, and generate synthetic data that integrates multiple types of information, thereby promoting the model's understanding and processing of complex data.

[0005] As one of the core tools of current AI technology, the Large Language Model (LLM) plays an important role in enhancing data generation capabilities. Through Prompting engineering techniques, users can provide specific instructions or contexts to the model to guide it to generate the required content. By adjusting the input prompts, the large language model can generate content that meets the needs of specific domains, making data generation more efficient and flexible.

[0006] In addition, the Retrieval-Augmented Generation (RAG) technology provides domain knowledge support for the generation process by integrating external knowledge bases or databases. RAG enables the model to access the latest and high-quality information in specific domains in real time, thereby improving the relevance and accuracy of the generated content, and is particularly suitable for the generation tasks of dynamically updated and professional domain content.

[0007] The conditional generation method provides more sophisticated control over the generation process. For example, through the ControlNet or T5 model, users can generate corresponding images based on given text descriptions and manipulate the parameters of the generated images. This method can not only flexibly generate multimodal content according to needs, but also ensure the semantic consistency of output and input conditions, meeting the personalized needs of users.

[0008] Current multi-domain multimodal data generation methods face some challenges, mainly reflected in: 1) Difficulty in integrating domain knowledge. Models often lack sufficient subdivided domain knowledge and find it difficult to generate content that has both domain characteristics and multimodal information. Lacking professional knowledge support, the generated data may lack depth and relevance in specific fields; 2) Insufficient controllability of multimodal generation. In multimodal outputs (such as text-generated images), it is still difficult to ensure that the generated content accurately meets the given conditions. How to use conditional generation technology to accurately control various aspects of the output, such as style, content, and parameters, requires further research and improvement.

[0009] Paper 1 (Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Shixiang Shane Gu. Aligning Text-to-Image Models using Human Feedback) proposed a text and image alignment method to align text-to-image generation models through human feedback. Specifically, the researchers used human preference data to train a reward model and then fine-tuned the generation model using reinforcement learning to make the generated images more in line with human expectations. However, there are also some drawbacks. It relies on a large amount of high-quality human feedback data, and the acquisition and annotation costs are relatively high. In addition, the reinforcement learning process may lead to unstable model training, and hyperparameters need to be carefully adjusted to ensure convergence. Paper 2 (Jiahui Yu, Yuanzhong Xu, Jing Yu, Koh Thang, Luong Gunjan, Baid Zirui, Wang Vijay, Vasudevan Alexander, Ku Yinfei, Yang Burcu, Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation.) proposed GigaGAN, a generative adversarial network for high-fidelity image generation. GigaGAN adopted a generator and discriminator to design a multi-scale feature fusion module, which can simultaneously focus on the global layout and local details of the image, ensuring the coordination of the generated image in terms of details and overall structure. By combining the self-attention mechanism and convolutional operations, it can efficiently capture long-range dependencies, generating images with more semantically consistent content and natural details. Moreover, this architecture can be extended to more modalities or task scenarios, such as conditional image generation, text-to-image generation, etc., showing strong adaptability and generalization ability. However, this method still has some defects. (1) While generating high-quality images, the model has a huge number of parameters, and the training and inference processes require a large amount of computing resources, limiting its application in resource-constrained environments. (2) Models based on GAN as the baseline network usually have problems with unstable training and need to be continuously monitored and tested with multiple parameter adjustments during the model training stage. Summary of the Invention

[0010] The objective of the present invention is to solve the problem that existing methods cannot efficiently and accurately generate multi-modal data in multiple fields, and a large model-driven multi-modal data generation method is proposed.

[0011] Step 1: Conduct a requirements analysis and scenario definition for the data product, clarify the multi-modal data requirements of the target field, and define the required modal types and their specific generation requirements.

[0012] Step 2: Use a large language model as the baseline model, and design a multi-modal extension framework on this basis to enable it to support the output of multi-modal data.

[0013] Step 3: Build a prompt engineering and design a prompt template for the data requirements of specific fields.

[0014] Step 4: Supplement domain knowledge, retrieve domain-related information by combining with the RAG technology, and dynamically enhance the prompt content and data generation description.

[0015] Step 5: Integrate a conditional generation network model, use ControlNet as the baseline network, and achieve precise control of the output data parameters.

[0016] Step 6: Automatically evaluate and manually sample and verify the generated data, and further optimize the data quality and modal consistency through self-supervised learning.

[0017] The beneficial effects of the present invention are as follows: The present invention proposes a large model-driven multi-modal data generation method. First, through the requirements analysis of the data product in a specific field, the large model is used to generate the corresponding required data in combination with the analysis results; by expanding and integrating the large model structure, multi-modal data output is realized, and for different knowledge in different fields, two optimization methods, namely prompt engineering and RAG, are proposed, enriching the description and background knowledge of data products in different fields, enabling the large model to better understand the specific data generation requirements; and by integrating a conditional generation model, a parameter control approach for generating data is provided to more finely ensure that the generated data meets the requirement standards; finally, the generated data products are evaluated through a combination of automatic evaluation and manual sampling verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of the large model-driven multi-modal data generation method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following further elaborates on this application in conjunction with the attached Figure 1 to make a more detailed description of this application.

[0020] Specific Embodiment 1: The method described in this embodiment is a large model-driven multi-modal data generation method, which is specifically implemented through the following steps:

[0021] Step 1: Comprehensively research the target industry and application scenarios. By referring to existing materials, competitor analysis, user interviews, etc., clarify the core value and functional positioning of multi-modal data in this field, and summarize the specific business pain points and high-priority usage scenarios to be solved. Decompose the generation requirements that multi-modal data may involve at different levels such as text, image, audio, or video one by one. Combining the application characteristics, compliance requirements, and user experience standards of the target field, sort out the constraints of each modality in terms of quality (such as resolution, sound quality, text length), format (such as file encoding, playback duration, text type), and style (such as formal or entertaining, conservative or creative). Organize all the requirement items closely related to the target field into a requirements specification, and refine the key indicators to the business logic and data annotation dimensions that may be required later;

[0022] Step 2: Based on the large language model, construct a multi-modal feature extraction layer to convert the input data of different modalities into a unified feature representation. For the visual modality, use Vision Transformer as the feature extractor, and divide the input image into fixed-size image patches (patches) denoted by, and each image patch is converted into an initial feature vector through a linear mapping, and positional encoding

[0023]

[0024]

[0025] For the audio modality, the Wav2Vec 2.0 model is used for feature extraction. This model first performs preliminary feature extraction on the original audio signal through a set of convolutional neural networks to extract local time-frequency features. Subsequently, the stacked Transformer layers are used to model these local features to capture long-range dependencies and global information in the audio, thereby outputting audio embeddings with rich representational capabilities to provide strong support for subsequent multimodal fusion. After completing the feature extraction of each modality, the present invention uses a linear projection layer to map the features of text and audio into a unified embedding space to ensure comparison and fusion in the same dimension. And a cross-modal attention mechanism is introduced to cross-align and fuse the features of different modalities. Specifically, a multi-head cross-attention module is constructed, where the features of one modality (such as text) are used as queries, while the features of the other modality (such as audio) are used as keys and values respectively. By calculating the dot-product similarity between the query and the key, and passing through softmax normalization to obtain the attention weights, and then weighted summing these weights with the values, a fused feature representation is generated.

[0026] ;

[0027] Step 3: In the stage of prompt engineering construction and template design, according to the detailed data requirement specifications, a set of prompt templates for the target domain need to be designed for different data requirements. The template needs to specify the input format, output style, and professional parameters, and by embedding domain-specific terms and dynamic variables, ensure that the generated data can accurately reflect the specific business requirements. In actual design, the present invention constructs a multi-level prompt system including basic prompts, style prompts, and detail prompts, thereby providing multi-dimensional and multi-level guidance for the model. The basic prompt part is mainly used to establish the overall context and goal orientation of the task. For example, for the financial market analysis task, the following basic prompt can be designed: "Please act as a senior financial analyst and generate a detailed market analysis report based on the current latest market data and macroeconomic indicators." Such a prompt clarifies the task identity, data background, and final goal. For the image generation task, the basic prompt can be: "Please generate an image showing the future urban landscape, which is required to have the characteristics of the integration of science fiction and modern elements. In the overall design process of the prompt template, Few-shots samples are introduced to help the model better understand the task requirements and output content and data that more conform to the target format by showing the format and content details of the expected output.

[0028] Step 4. In the stage of dynamic domain knowledge supplementation and prompt enhancement, the present invention adopts the Retrieval-Augmented Generation (RAG) technology to closely integrate the latest external domain knowledge with the generation model. Its basic principle is as follows: When a user proposes a generation task, the system first extracts key information from the prompt to generate a query statement, which will serve as the input to the retriever. The retriever uses a pre-trained semantic encoder (a bidirectional encoder based on Transformer) to convert the query into a vector representation and performs vector matching in a pre-constructed knowledge base (which consists of authoritative literature, patent databases, industry reports, etc.). By calculating the cosine similarity or other distance metrics between vectors, the system can quickly retrieve the most relevant documents or abstracts from the vast amount of data and sort these documents according to their relevance. Subsequently, the system will perform semantic parsing on the retrieved documents. With the help of techniques such as named entity recognition models, keyword extraction, and syntactic analysis, the complex literature content will be transformed into refined information that can be directly used in the prompt template. The extracted key information will be dynamically embedded into the original multi-modal prompt template to form an enhanced prompt that contains both the original task description and the latest domain knowledge, ensuring that the model not only relies on the requirements described by the original prompt but also can leverage real-time updated professional information to ensure the accuracy and frontier nature of the generated content;

[0029] Step 5. In the stage of conditional generation network integration and parameter control, to achieve fine control of the key parameters in the multi-modal data generation process, the present invention adopts a conditional generation network and uses ControlNet as the baseline. The core idea of ControlNet is to introduce an additional, trainable control branch to directly embed the key control variables into the generation network, thereby imposing fine constraints on the key parameters of each modality (such as image resolution, text length, audio sampling rate, etc.) during the generation process. Specifically, first, a set of control variables C is designed to represent the key parameters of each modality. For each modality, after passing through their respective encoders, preliminary feature representations F are obtained. ControlNet introduces a control signal extraction module, which extracts features from the input control variables C through a series of convolutional layers to obtain control features at each level , where i represents different levels in the network. During the generation process, each layer of the generation network will receive a modulation signal from the control branch. Specifically, let the original hidden layer feature of the i-th layer of the generation network be , then in ControlNet, the modulated feature of this layer is given by the following formula

[0030]

[0031] where is the learnable scaling factor for the i-th layer, which is used to balance the influence between the original features and the control signal. In this way, the control variable C is incorporated into the feature representations of each layer in a weighted manner, ensuring that the entire generation process is strictly guided by the preset conditions. Finally, the generation model generates the final output layer by layer based on the modulated features so that the output data strictly conforms to the format, style, and quality requirements preset for multi-modalities such as images, texts, and audios and progresses layer by layer to generate the final output

[0032] Step 6: In the data quality assessment, verification, and self-supervised optimization stage, the present invention designs a comprehensive evaluation system, starting from both automated evaluation and manual sampling verification to ensure that the generated data meets the expected standards in terms of accuracy, integrity, diversity, and cross-modal consistency

[0033] In the part of automated evaluation, the generated data is initially quantitatively analyzed through statistical metrics. For text data, metrics such as word frequency distribution, sentence length, grammar correctness, and spelling error rate are calculated to ensure that the generated text conforms to language norms and domain requirements; for image data, image resolution, clarity, color distribution, and contrast are detected, and metrics such as Structural Similarity Index (SSIM) and Fréchet Inception Distance (FID) are used to evaluate the visual quality and style consistency of the generated images; for audio data, basic metrics such as duration, sampling rate, and signal-to-noise ratio are detected, and evaluation criteria such as PESQ (Perceptual Evaluation of Speech Quality) and STOI (Speech Intelligibility) are used to measure the clarity and quality of the generated audio

[0034] In addition to these statistical metrics, the present invention also introduces a deep learning evaluator to conduct a more detailed evaluation of the semantics, style, and context coherence of the generated content. For example, in text generation, metrics such as BLEU, ROUGE, METEOR, and BERTScore can be used to quantitatively measure the closeness of the generated text to the reference text in terms of semantics and expression style; for the consistency evaluation between images and texts, a pre-trained multi-modal CLIP model is adopted, and the vector similarity between the image and the text description is calculated to ensure their semantic matching; for audio data, combined with speech recognition technology, the audio is converted into text and compared with the original prompt to detect whether the generated speech accurately conveys the expected information

[0035] Specific Embodiment 2: The difference between this embodiment and Specific Embodiment 1 is that an automated feedback mechanism is added and integrated in Step 4 to continuously evaluate the generation results. The evaluation module will automatically score according to indicators such as the accuracy, relevance, and professionalism of the generated content. At the same time, combined with the manual sampling verification of domain experts, it is fed back to the system to adjust the parameters of the retriever, update the knowledge base, and further optimize the prompt template. Through this closed-loop feedback, the prompt template and the retrieval enhancement module can be continuously iteratively optimized, so that the generated content is always highly consistent with the latest domain knowledge, further improving the generation quality;

[0036] Specific Embodiment 3: The difference between this embodiment and Specific Embodiment 1 is that in the training stage of the ControlNet model in Step 5, a joint training strategy is adopted to construct a multi-modal joint loss function to further constrain the consistency and synergy between the generation results of each modality. Specifically, for each modality output, a dedicated loss function is designed. For example, for image generation, loss or perceptual loss is used; for text generation, cross-entropy loss is used; for audio generation, spectral loss is used, etc. Let represent the loss of the i-th modality, then the multi-modal joint loss can be expressed as:

[0037]

[0038] where, is the weight of each modality loss, is the loss term used to measure the consistency of different modality outputs in semantics, style, or other high-level features, is its weight. Through this joint training method, each modality is not only optimized in its respective generation task, but also the synergy and consistency between the generation results are effectively guaranteed, thus significantly improving the accuracy and stability of the overall generation effect.

[0039] The present invention provides a large model-driven multi-modal data generation method, which provides a method for generating high-quality and large-volume data according to the data requirements of different modalities and different scenarios. By combining the use of prompt engineering and dynamic specialization information supplementation technology, accurate requirement input is achieved. And combined with related models such as the control network, the content, format, and style of the data output are further refined to meet the user's expectations.

[0040] The beneficial effects of the present invention are as follows: 1) By adopting hierarchical prompt engineering and domain knowledge dynamic supplementation technology, the generated data such as text, images, and audio not only meet the preset requirements in terms of format and style, but also can incorporate the latest industry knowledge and professional terms in real time, thus greatly improving the professionalism and accuracy of the generated results; 2) The conditional generation network is introduced, especially with ControlNet as the baseline, which can finely adjust the key parameters of each modality (such as image resolution, text length, audio sampling rate, etc.) to ensure that the output data strictly meets the business requirements. At the same time, the multi-head cross-attention mechanism is used to achieve information fusion and alignment between different modalities, effectively ensuring the consistency and coordination of the outputs of each modality in terms of semantics, style, and content; 3) An automated evaluation and self-supervised feedback closed-loop mechanism is constructed, which combines statistical indicators, deep learning evaluators, and manual sampling verification to continuously monitor and optimize the accuracy, integrity, and diversity of the generated data. This feedback mechanism can not only identify and correct problems in the generation process in a timely manner, but also continuously promote the iterative update of the model, thus achieving long-term stable high-quality data generation in practical applications.

[0041] The above examples of the present invention are only for illustrating in detail the calculation model and calculation process of the present invention, rather than limiting the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is impossible to enumerate all the implementation manners here. Any obvious changes or modifications derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A large model-driven multimodal data generation method, characterized in that, It includes the following steps: Step 1: Conduct a requirements analysis and scenario definition for the data product, clarify the multi-modal data requirements in the target field, and define the required modal types and their specific generation requirements; Step 2: Use a large language model as the baseline model, and design a multi-modal extension framework on this basis to support the output of multi-modal data; Step 3: Build a prompt engineering and design a prompt template for the data requirements in a specific field; Step 4: Supplement domain knowledge, combine the RAG technology to retrieve domain-related information, and dynamically enhance the content of the prompt and the data generation description; Step 5: Integrate a conditional generation network model, use ControlNet as the baseline network to achieve precise control of the output data parameters; Step 6: Automatically evaluate and manually sample and verify the generated data, and further optimize the data quality and modal consistency through self-supervised learning.

2. The method according to claim 1, wherein In Step 2, the multi-modal extension framework includes a multi-modal feature extraction layer, which is used to convert the input data of different modalities into a unified feature representation, and introduce a cross-modal attention mechanism to cross-align and fuse the features of different modalities.

3. The method according to claim 1, characterized in that, In Step 3, the prompt template includes a basic prompt, a style prompt, and a detail prompt, which are used to provide multi-dimensional and multi-level guidance for the model.

4. The method according to claim 1, wherein In Step 4, the RAG technology is used to retrieve the most relevant documents or abstracts from a pre-constructed knowledge base, and dynamically embed the extracted key information into the original multi-modal prompt template to form an enhanced prompt.

5. The method according to claim 1, wherein In Step 5, ControlNet directly embeds the key control variables into the generation network by introducing an additional, trainable control branch, so as to impose fine constraints on the key parameters of each modality during the generation process.

6. The method according to claim 1, wherein In Step 6, the automatic evaluation includes statistical index evaluation and deep learning evaluator evaluation, and the manual sampling verification is carried out in combination with domain experts.

7. The method according to claim 1, characterized in that, Add an automated feedback mechanism in Step 4 to continuously evaluate the generation results, and adjust the parameters of the retriever, update the knowledge base, and further optimize the prompt template according to the evaluation results.

8. The method according to claim 1, wherein In the training stage of the ControlNet model in Step 5, adopt a joint training strategy to construct a multi-modal joint loss function to further constrain the consistency and synergy between the generation results of each modality.

Images