Image generation method and device based on fine-grained multi-modal knowledge enhancement
By employing a fine-grained multimodal knowledge enhancement method, optimizing text features using a large language model and knowledge graph, and combining a dual DiT architecture and an introspective agent mechanism, the generation problem of AIGC models in specific domains is solved, achieving efficient and accurate image generation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-01
AI Technical Summary
When existing AIGC models generate images in specific domains, they suffer from insufficient fine-grained knowledge graph context, difficulty in multimodal knowledge mapping, and a lack of introspective proxy mechanisms, resulting in semantic bias and poor visual effects in the generated results.
We employ a fine-grained multimodal knowledge enhancement method, which obtains text semantic units through a large language model (LLM), combines a pre-trained multimodal retrieval model and knowledge graph to concatenate and reduce the dimensionality of text features and visual features, generates images using a dual DiT architecture, and performs closed-loop iterative optimization through an introspective agent mechanism.
It achieves high semantic relevance and high visual quality in image generation in specific domains, improves generation efficiency and accuracy, and reduces reliance on expertise and additional input.
Smart Images
Figure CN121962813A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to artificial intelligence generated content (AIGC), and more particularly to an image generation method and apparatus based on fine-grained multimodal knowledge enhancement. This invention achieves accurate matching of semantics and textual requirements in generated images while ensuring visual quality and generation efficiency. Background Technology
[0002] The rapid development of AIGC, especially diffusion models such as Stable Diffusion, DALL-E (an image diffusion model proposed by OpenAI), and Imagen (an image diffusion model proposed by Google), is reshaping the industrial landscape of digital content creation. These technologies have profoundly transformed the creation model of digital content, and image generation, as a core technology direction of AIGC, has become a crucial engine for promoting cultural innovation and facilitating industrial transformation and upgrading in the digital economy era. However, the demand for personalized and refined image generation in specific domains continues to rise. Existing AIGC models, trained with large amounts of general data, are ineffective at generating deep knowledge or rare concepts in specific domains. Problems such as text semantic restoration bias, lack of domain knowledge, and semantic disconnect are becoming increasingly prominent, making accurate text guidance, external knowledge supplementation, and deep semantic matching critical issues that urgently need to be addressed.
[0003] Against this backdrop, image generation schemes targeting specific domains or rare concepts have emerged to enhance the semantic relevance and visual quality of images generated by models within specific domains. Mainstream methods fall into three categories: cue word expansion, model fine-tuning, and knowledge augmentation methods, represented by Retrieval-Enhanced Generation (RAG). Cue word expansion guides the model to generate the desired output by writing detailed cue words, but this process is time-consuming and relies on expertise and an understanding of model preferences. Model fine-tuning typically involves training novel low-rank adaptations (LoRA). [1] Adjusting the parameters of the diffusion model can improve generation quality, but this approach is costly to train and limited by existing knowledge boundaries. Relational Acyclic Graphs (RAGs), a currently popular knowledge augmentation solution, enhance the model's understanding of input conditions by acquiring data highly relevant to the semantics of prompt words through an external retrieval library. However, it falls short in fine-grained context alignment. Furthermore, knowledge augmentation can also be achieved through pre-trained knowledge embedding and knowledge graph fusion. RAGs have the advantage of dynamically acquiring the latest knowledge, while knowledge graphs are more suitable for deep understanding and complex reasoning tasks. Combining the two represents a future development trend.
[0004] Existing knowledge-enhanced image generation schemes exhibit significant limitations in specific scenarios, primarily facing the following three key challenges: (1) Difficulty in embedding fine-grained knowledge into the context of knowledge graphs. Existing methods cannot accurately capture the fine-grained semantic relationships of the context, limiting the effectiveness of knowledge representation. (2) Difficulty in multimodal knowledge mapping. Data in specific domains often encompasses multiple modalities such as text and images, and the semantic mapping relationships between modalities exhibit cross-modality. A typical feature is that a single text content can construct corresponding mapping relationships with data from multiple different semantic dimensions across modalities, significantly increasing the difficulty of achieving multimodal knowledge association. (3) Lack of effective multimodal introspective proxy mechanisms. Existing generation models mostly adopt a "one-time generation" mode, lacking an introspective proxy mechanism based on multimodal knowledge. They cannot continuously optimize generation accuracy through real-time evaluation and dynamic feedback, making it difficult to form a closed-loop iterative quality improvement path.
[0005] Therefore, the purpose of this invention is to overcome the limitations of the prior art and provide an image generation scheme with fine-grained multimodal knowledge embedding that can overcome the problem of lack of knowledge in a specific domain. Summary of the Invention
[0006] This invention provides an image generation method and apparatus based on fine-grained multimodal knowledge enhancement. This invention ensures that, in a specific domain, images highly semantically related to the input conditions can be accurately generated using simple text prompts while maintaining high visual quality. It provides a practical technical solution for meeting the high-efficiency, high-precision generation requirements in specific domains, as detailed below:
[0007] A first aspect is an image generation method based on fine-grained multimodal knowledge enhancement, the method comprising:
[0008] Based on the obtained text prompts, multi-level text semantic units are obtained through LLM. Through a pre-trained multimodal retrieval model, unit-by-unit retrieval and semantic comprehensive retrieval are performed on the text semantic units to obtain topic images and reference images that are highly related to the text semantics.
[0009] The text features of the optimized and expanded prompt words are obtained by using a pre-trained text encoder, and the visual features of the corresponding topic image are obtained by using an image encoder. The obtained text features and visual features are concatenated in terms of feature dimension. The concatenated features are dimensionality reduced by a trainable MLP and the dimensionality-reduced features are transformed into embedding conditions. At the same time, the reference image is input into the reference branch to optimize the attention parameters.
[0010] The generated image is subjected to multi-dimensional quality evaluation. The final quality evaluation score is output by comprehensively considering the relevance of the generated image to the text prompt words and the semantic consistency with the domain knowledge graph. The score is compared with the preset threshold, and the introspection agent mechanism is triggered based on the current number of feedbacks.
[0011] The latent space representation of the image obtained from the output of the DiT branch is fed into the image decoder to restore the final generated image.
[0012] The obtained text prompt words are:
[0013] For the input text prompts, the core semantic units of the text prompts are obtained through a large language model, a domain knowledge graph is generated, fine-grained contextual information of different semantic units is obtained based on the knowledge graph, and the key feature boundaries of the core semantic units are clarified through multi-step chain reasoning of LLM by referring to the chain-like prompt thinking. Finally, the optimized text prompts are output by integrating contextual information.
[0014] Among them, the core semantic unit set Key elements that serve as effective references for image generation; implicit relationships Representing core semantic units Semantic connections between other core semantic units;
[0015] Based on the domain knowledge graph generated by LLM, the attributes of each core semantic unit are extracted to construct a set of prompt word attributes. It employs a contrastive chain thinking prompting strategy, based on implicit relationships. and the set of attributes of semantic units Pairs of positive examples are constructed through multi-step chain reasoning in LLM. and negative examples ; .
[0016] Among them, embedding conditions :
[0017] ;
[0018] in, Indicates the index of a word in the text prompt. The index of a sample in the subject image set is represented by ||, which represents the concatenation operation between text features and subject image features in the feature dimension direction.
[0019] The introspective proxy mechanism works by using a closed-loop iterative process of "generation → evaluation → feedback → correction" to ensure that the generated image matches the input conditions; and it addresses the input text prompts. The text encoder is used to transform the input condition C to generate the initial image latent representation. , .
[0020] The quality assessment score is as follows:
[0021] ;
[0022] in, This is a knowledge weighting coefficient, which is dynamically adjusted according to the task scenario. This refers to cross-modal similarity, used to measure global semantic consistency. It is an inspection score for the semantic relevance output of the knowledge graph and the generated image. Below the quality score threshold This triggers the introspective agent, which then outputs dual-path feedback signals through the introspective agent. and , Optimize the text description of prompt words using LLM. Then, the image details are directly optimized and adjusted through the gradient of the latent space.
[0023] A second aspect is an image generation apparatus based on fine-grained multimodal knowledge enhancement, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform the method described in any of the first aspects.
[0024] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.
[0025] The beneficial effects of the technical solution provided by this invention are:
[0026] 1. This invention proposes a text prompt optimization method that uses a large language model to expand and optimize simple text input, and combines knowledge graphs and fine-grained contextual information to deepen the semantic connections between different parts of the text prompt words; for the key information of the input text prompt words, the attributes of the core semantic units are extracted through the large language model (LLM), and a refined semantic representation is constructed through positive / negative sample comparison learning, which is then fused with the initial text, so that the automatic conversion from fuzzy description to compliant image generation conditions can be achieved without additional input;
[0027] 2. This invention proposes a fine-grained multimodal fusion strategy based on a dual-branch Diffusion Transformer (DiT) architecture. By using the text features of the expanded text prompts to obtain highly relevant visual features through cross-modal retrieval, and then concatenating them to achieve efficient fusion of multimodal information, the invention also uses a trainable multilayer perceptron (MLP) and reference branches to independently embed the subject image and fusion features, enriching the embedding conditions and improving the semantic accuracy and visual detail consistency of the generated image.
[0028] 3. This invention proposes a self-reflective proxy mechanism with cross-modal evaluation and knowledge graph constraints, constructing a logical closed loop of "generation-evaluation-feedback-correction". The generated image and the expanded text are evaluated for quality through similarity assessment using a contrastive language image pre-training model (CLIP) and verification using an LLM-based knowledge graph. For samples that do not meet the standards, a dual-path feedback mechanism of text correction and latent space feature optimization is enabled to ensure the consistency between the visual details of the generated result and the domain knowledge.
[0029] Therefore, this invention significantly improves the semantic consistency and visual detail representation of image generation tasks by achieving automatic optimization of text prompts, accurate fusion of multimodal features, and closed-loop verification of generation quality. Compared with existing technologies, this invention effectively solves the problems of semantic deviation and lack of detail in generated content caused by ambiguous text descriptions without relying on manually constructed refined prompts or additional visual reference inputs. At the same time, by leveraging knowledge graph constraints and cross-modal evaluation mechanisms, it ensures the rationality of the generated results at the domain knowledge level, providing an efficient and feasible technical solution for low-threshold, high-quality AIGC image generation. Attached Figure Description
[0030] Figure 1 This is a flowchart illustrating an image generation method based on fine-grained multimodal knowledge enhancement.
[0031] Figure 2 This is a schematic diagram of the framework for an image generation method based on fine-grained multimodal knowledge enhancement.
[0032] Figure 3 A schematic diagram illustrating the framework for calculating the image quality assessment score of the introspective agent module. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.
[0034] Example 1
[0035] This invention discloses an image generation method based on fine-grained multimodal knowledge enhancement. See [link to relevant documentation]. Figure 1 The method includes the following steps:
[0036] Step 101: For the input text prompt words, the core semantic units of the text prompt words are obtained through the Large Language Model (LLM), and a domain knowledge graph is generated accordingly. Fine-grained contextual information of different semantic units is obtained based on the knowledge graph. Referring to the learning method of comparative chain prompt thinking, the key feature boundaries of the core semantic units are clarified through the multi-step chain reasoning of LLM, and then the contextual information is integrated to output the optimized text prompt words.
[0037] Step 102: Based on the text prompts output in Step 101, obtain multi-level semantic units through LLM, and use a pre-trained multimodal retrieval model to perform unit-by-unit retrieval and semantic comprehensive retrieval based on the text semantic units to obtain topic images and reference images that are highly related to the text semantics.
[0038] Step 103: Use a pre-trained text encoder to obtain the text features of the optimized and expanded prompt words, and use an image encoder to obtain the visual features of the corresponding topic image. Then, concatenate the obtained text features and visual features in terms of feature dimensions. Use a trainable MLP to reduce the dimensionality of the concatenated features to further fuse the text features and image features, and transform them into embedding conditions. At the same time, input the reference image obtained in step 102 into the reference branch to optimize the attention parameters.
[0039] Step 104: Use a dual DiT architecture branch as the base model for image generation, which includes a frozen backbone DiT branch and a trainable reference DiT branch.
[0040] The reference DiT branch receives input from the reference image and optimizes the attention parameters by introducing conditional noise prediction mean square error loss, thus achieving end-to-end training. The backbone DiT branch performs denoising operations based on the embedding conditions obtained in step 103, and outputs the denoising result after multiple iterations, which is the latent representation of the generated image.
[0041] Step 105: By performing a multi-dimensional quality assessment on the generated image, the final quality assessment score is output by comprehensively considering the relevance of the generated image to the text prompt words and the semantic consistency with the domain knowledge graph; the final score is compared with the preset threshold, and the introspective proxy mechanism is determined by referring to the current number of feedbacks.
[0042] The quality assessment of the generated images is divided into two aspects: image-text similarity based on prompt words and semantic relevance based on domain knowledge graphs.
[0043] If the quality assessment score of the generated image is lower than the preset threshold and the number of iterations does not exceed the maximum number of iterations, the introspective agent is triggered. At this time, the introspective agent outputs a dual-path feedback adjustment signal, optimizes the text prompt word description through the instruction parsing model, and directly corrects the local features through gradient optimization in the latent space of the diffusion model, and re-executes steps 102-105; otherwise, the current step ends.
[0044] Step 106: The latent space representation of the image obtained from the output of the DiT branch of the main branch is fed into the image decoder to restore the final generated image.
[0045] The image decoder uses a VAE decoder.
[0046] In summary, the embodiments of the present invention complete the training of the image generation framework through the above steps 101-106, realize the image generation method based on fine-grained multimodal knowledge enhancement, and complete the embedding of multimodal knowledge in a specific domain of the image while maintaining the visual quality of the image, thus meeting the needs of accurate image generation in specific domains in practical applications.
[0047] Example 2
[0048] The training process in Example 1 will be further described below with reference to specific calculation formulas:
[0049] 201: Generate conditions (i.e., text prompts) based on the input image. Each element This represents a single word or semantic unit within the prompt, and invokes an LLM (such as GPT, Deep Seek, and Doubao AI) with strong semantic understanding capabilities to process the prompt. We conduct in-depth analysis to accurately extract the core information from the prompt words and obtain the set of core semantic units within them. and their implicit relationships ;
[0050] Among them, the core semantic unit set This refers to key elements that serve as effective references for image generation, including but not limited to the generating subject, scene, and style requirements, removing set elements. Meaningless embellishments and redundant information. Implicit relationships. Specifically representing the core semantic unit Semantic connections between other core semantic units.
[0051] Based on the domain knowledge graph generated by LLM, the attributes of each core semantic unit are extracted to construct a set of prompt word attributes. To further enhance semantic detail, a contrastive chain-like thinking prompt is employed. [2]Strategy, based on implicit relationships and the set of attributes of semantic units Pairs of positive examples are constructed through multi-step chain reasoning in LLM. and negative examples .
[0052] ;
[0053] Among them, negative examples The design needs to be consistent with the positive example Based on semantic similarity, the key features of the core semantic units are precisely distorted, rather than randomly modified. Common distortion methods include: key attribute replacement, core relationship disruption, and core semantic unit replacement, ensuring that the difference between negative and positive examples focuses only on the key feature dimensions of the original prompt words. The contrast chain-like thinking prompt strategy uses positive examples... With negative examples The systematic comparison guides the model to more accurately identify the differences in key features, define clearer decision boundaries, and establish more accurate semantic representations and reasoning paths in the bidirectional learning of "what should be done" and "what should not be done".
[0054] For the generated positive and negative examples, the key feature boundaries of the core semantic units are clearly defined to ensure that the generated positive and negative examples accurately focus on the core semantic differences, combined with the input prompt words. Output prompts that incorporate contextual information .
[0055] 202: Based on the text prompt words obtained in step 201 Utilizing pre-trained cross-modal alignment models (e.g., OpenCLIP) [4] Semantic parsing and decomposition of semantic units (such as CLIP) are performed to construct a multi-level semantic unit set. .
[0056] in, The semantic unit in the text is the text prompt word. Structured extraction of core information, where each element Representing a single semantic unit, it encompasses descriptive elements related to image generation, such as the subject object, style attributes, and action state, constructing a complete semantic profile and providing accurate text anchors for subsequent cross-modal matching.
[0057] At the same time, for real image databases All images in the dataset were processed using pre-trained models based on the ViT architecture (such as ViT-B / 16, ViT-H / 14, etc.) to extract regionalized visual features. Constructing an image feature set .
[0058] in, The image embedding features are represented by p, p represents the length of the image patch sequence, and d represents the embedding feature dimension of a single image patch (e.g., if the image size is reset to d). Furthermore, using ViT-B / 16 as the image encoder, the embedded features of the image will be presented as ).
[0059] For real image libraries By calculating multi-level semantic unit sets Similarly, this The semantic relevance between them was used to ultimately select the reference image set. .
[0060] The overall semantic relevance is evaluated using a cosine similarity aggregation formula:
[0061] ;
[0062] in, The text features representing the k-th semantic unit of the text. Represents the dimension of text features. Representing an image Chinese and text The visual features of the corresponding region, for a single text unit Corresponding regional image features Use a similar screening method.
[0063] Specifically, through text semantic unit features Same regional visual features The semantic units of the text are determined by quantifying the cosine similarity and sorting them in reverse order of similarity. Corresponding theme image .
[0064] ;
[0065] Simultaneously, a thematic image set highly aligned with the semantic units of the prompt words is constructed. and topic image index , the elements Used to represent a semantic unit of text Images on the same theme The mapping relationship.
[0066] 203: For the obtained text prompt words and themed image collection Using a pre-trained text encoder and image encoder Encode the two as embedded features , .
[0067] Where n represents the total number of words in the text prompt. This indicates the number of samples in the subject image set.
[0068] To enhance text cues, a multilayer perceptron (MLP) is used to reduce the dimensionality of the concatenated features, further fusing multimodal features and leveraging visual features extracted from thematic image patches to enhance the text embedding, resulting in the final conditional embedding. :
[0069] ;
[0070] Where i represents the code of a word in the text prompt. The index of a sample in a set of subject images (used to represent subject images) (The semantic relationship between a word in the same text prompt), || represents the concatenation operation of text features and topic image features in the feature dimension direction.
[0071] 204: The dual DiT architecture is adopted as the main network for image generation. It consists of two DiT branches: one is the frozen base generation model branch (backbone DiT), and the other is the trained reference network branch (reference DiT). Through the collaborative working mode of the two branches, multimodal information is effectively integrated and the semantic accuracy and detail consistency of the generated results are improved.
[0072] The backbone DiT branch freezes parameters, preserving the image generation capabilities of the trained model. As the core execution unit for image generation, it is responsible for receiving multimodal fusion features from the reference DiT branch and conditional representations after topic embedding enhancement. The DiT branch is responsible for receiving reference images. By dynamically optimizing the key and value matrices in its own attention mechanism, it constructs an efficient multimodal information fusion channel, solving the problems of insufficient multimodal information fusion and visual-semantic alignment deviation in traditional generative models, and providing accurate visual reference for the subsequent generation process.
[0073] During the model training process, the input real image is used. It is transformed into a latent representation through a variational autoencoder (VAE Encoder). By adding noise and spreading, Gradually transform into white noise .
[0074] Noise Addition Features at Time Step t The backbone DiT is based on enhanced conditional embedding. Following the iterative denoising logic of the diffusion model, the noisy features are gradually denoised. Denoising and detail reconstruction are performed. Multiple iterative optimization steps are used to improve the consistency of detail in the generated images and the accuracy of text semantics. To guide the model in accurately learning the noise distribution patterns, this architecture uses the conditional noise prediction mean square error (MSE) loss as the overall optimization objective to quantify the true noise. and model prediction noise The mathematical expression for the difference between them is as follows:
[0075] ;
[0076] in, This represents the true image representation in the latent space without added noise. This refers to multimodal knowledge embedding that integrates textual semantic features and topical image features. This refers to the squaring operation of the L2 norm. This represents the standard Gaussian noise used in the forward noise addition step; The noise value predicted by the model based on the current input is the core representation of the learning effect of the diffusion model. The time step parameter represents the diffusion process. Represents the first step in the forward diffusion process Noisy features at each time step.
[0077] ;
[0078] in, , , It is the preset noise variance.
[0079] By minimizing the MSE loss, the diffusion model is gradually learned to predict noise accurately, thereby achieving high-quality image generation. In the inverse denoising stage, the latent representation of the output image is completed after iterations at all time steps T. .
[0080] 205: To prevent the generation process from failing to complete in one go, a reflection agent framework with an adaptive feedback mechanism is applied. Through a closed-loop iterative process of "generation → evaluation → feedback → correction," the generated image is ultimately ensured to be highly consistent with the input conditions. This includes input text prompts. The text encoder is used to transform the input condition C to generate the initial image latent representation. This process can be represented as .
[0081] in, Gaussian white noise is used as the input for the denoising process. This represents the conditional embedding of DiT, where G refers to the base image generation model (i.e., the pre-trained DiT image generation architecture), and the DiT Block uses a variant with cross-attention, which is obtained by adjusting the conditional embedding. Cross-attention is performed on the image's embedding features to complete the deep embedding of conditional information.
[0082] Latent representation of the generated image Conduct multi-dimensional quality assessments and output assessment scores. Based on a predefined threshold and assessment score Determine whether the introspection proxy mechanism needs to be triggered.
[0083] The overall score consists of two parts: cross-modal similarity and semantic consistency score based on knowledge graph.
[0084] ;
[0085] in, This is a knowledge weighting coefficient, which can be dynamically adjusted according to the task scenario. Cross-modal similarity refers to a measure of global semantic consistency. Specifically, it utilizes a pre-trained evaluation model (such as CLIP) based on the latent representation of the generated image. and conditional embedding calculate: . The scoring function is designed based on the pre-trained CLIP feature space and is implemented using cosine similarity. It is an output score for semantic consistency checks on generated images based on domain knowledge graphs, ensuring that generated samples conform to the knowledge constraints of a specific domain.
[0086] like And the number of feedback iterations is less than This triggers the introspective agent, which generates text correction feedback signals for both the text description and the embedded features. and feature adjustment feedback signal Otherwise, the latent representation of the resulting generated image is denoted as .
[0087] Among them, text correction feedback signal It processes semantic biases from user feedback or automatic detection through instruction parsing models (such as large language models like GPT or DeepSeek), transforms them into structured correction instructions, and generates updated conditions. Embedded feature adjustment feedback signal Local features are directly corrected through gradient optimization within the latent space of the diffusion model. By co-optimizing text correction and embedding adjustment, both high-level semantic correction and low-level feature optimization are considered, and knowledge graph constraints are incorporated to ensure that the final generated result meets high-quality standards in both visual detail and domain consistency.
[0088] Feedback signal , After injection, the model regenerates the latent representation of the image. ,Right now The process of "generation → evaluation → feedback → correction" is iteratively executed until the quality requirements are met. Or reach the maximum number of iterations The latent representation of the output This is recorded as the final output of the introspection proxy process. .
[0089] 206: The latent space representation of the obtained output image The image is sent to the image decoder and restored to an RGB image. :
[0090] ;
[0091] in, This indicates a differential automatic decoder.
[0092] In summary, the embodiments of the present invention complete the training of the image generation framework through the above steps 201-206, realize an image generation method based on multimodal fine-grained knowledge embedding, accurately identify the core semantic requirements of text prompt words, and reliably embed multimodal knowledge in a specific domain while maintaining the quality of the generated image. This generates images that conform to domain knowledge constraints and whose visual details are highly consistent with the text description, significantly improving the adaptability and efficiency of image generation tasks, while reducing the technical requirements of users for accurate image generation tasks.
[0093] Example 3
[0094] An image generation apparatus based on fine-grained multimodal knowledge enhancement, the apparatus comprising: a memory storing program instructions, and a processor calling the program instructions stored in the memory to cause the apparatus to execute the following method steps in Embodiment 1:
[0095] Based on the obtained text prompts, multi-level text semantic units are obtained through LLM. Through a pre-trained multimodal retrieval model, unit-by-unit retrieval and semantic comprehensive retrieval are performed on the text semantic units to obtain topic images and reference images that are highly related to the text semantics.
[0096] The text features of the optimized and expanded prompt words are obtained by using a pre-trained text encoder, and the visual features of the corresponding topic image are obtained by using an image encoder. The obtained text features and visual features are concatenated in terms of feature dimension. The concatenated features are dimensionality reduced by a trainable MLP and the dimensionality-reduced features are transformed into embedding conditions. At the same time, the reference image is input into the reference branch to optimize the attention parameters.
[0097] The generated image is subjected to multi-dimensional quality evaluation. The final quality evaluation score is output by comprehensively considering the relevance of the generated image to the text prompt words and the semantic consistency with the domain knowledge graph. The score is compared with the preset threshold, and the introspection agent mechanism is triggered based on the current number of feedbacks.
[0098] The latent space representation of the image obtained from the output of the DiT branch is fed into the image decoder to restore the final generated image.
[0099] The obtained text prompts are:
[0100] For the input text prompts, the core semantic units of the text prompts are obtained through a large language model, a domain knowledge graph is generated, fine-grained contextual information of different semantic units is obtained based on the knowledge graph, and the key feature boundaries of the core semantic units are clarified through multi-step chain reasoning of LLM by referring to the chain-like prompt thinking. Finally, the optimized text prompts are output by integrating contextual information.
[0101] Among them, the core semantic unit set Key elements that serve as effective references for image generation; implicit relationships Representing core semantic units Semantic connections between other core semantic units;
[0102] Based on the domain knowledge graph generated by LLM, the attributes of each core semantic unit are extracted to construct a set of prompt word attributes. It employs a contrastive chain thinking prompting strategy, based on implicit relationships. and the set of attributes of semantic units Pairs of positive examples are constructed through multi-step chain reasoning in LLM. and negative examples ; .
[0103] Among them, embedding conditions :
[0104] ;
[0105] Where i represents the code of a word in the text prompt. The index of a sample in the subject image set is represented by ||, which represents the concatenation operation between text features and subject image features in the feature dimension direction.
[0106] The introspective proxy mechanism works by using a closed-loop iterative process of "generation → evaluation → feedback → correction" to ensure that the generated image matches the input conditions; and it addresses the input text prompts. The text encoder is used to transform the input condition C to generate the initial image latent representation. , .
[0107] The quality assessment score is as follows:
[0108] ;
[0109] in, This is a knowledge weighting coefficient, which is dynamically adjusted according to the task scenario. This refers to cross-modal similarity, used to measure global semantic consistency. It is an inspection score for the semantic relevance output of the knowledge graph and the generated image. Below the quality score threshold This triggers the introspective agent, which then outputs dual-path feedback signals through the introspective agent. and , Optimize the text description of prompt words using LLM. Then, the image details are directly optimized and adjusted through the gradient of the latent space.
[0110] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.
[0111] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.
[0112] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.
[0113] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.
[0114] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.
[0115] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the method description in Embodiment 1, and the embodiments of the present invention will not be repeated here.
[0116] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.
[0117] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.
[0118] References
[0119] [1]Edward J. H, YeLong S, Phillip W, et al. LoRA: Low-Rank Adaptation of Large Language Models.[C] / / Proceedings of the IEEE Conference onInternational Conference on Learning Representations (ICLR). 2022: 1-20.
[0120] [2]Yewken C, Guizhen C, Luuanh T, et al. Contrastive Chain-of-ThoughtPrompting. / / CoRR abs / 2311.09277 (2023).
[0121] [3]Peebles W, Saining X. Scalable diffusion models with transformers.[C] / / Proceedings of the IEEE Conference on International Conference onComputer Vision (ICCV). 2023:4195-4205.
[0122] [4]Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision[C] / / Proceedings of the International Conference on Machine Learning (ICML). 2021: 8748-8763.
[0123] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.
[0124] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An image generation method based on fine-grained multimodal knowledge enhancement, characterized in that, The method includes: Based on the obtained text prompts, multi-level text semantic units are obtained through LLM. Through a pre-trained multimodal retrieval model, unit-by-unit retrieval and semantic comprehensive retrieval are performed on the text semantic units to obtain topic images and reference images that are highly related to the text semantics. The text features of the optimized and expanded prompt words are obtained by using a pre-trained text encoder, and the visual features of the corresponding topic image are obtained by using an image encoder. The obtained text features and visual features are concatenated in terms of feature dimension. The concatenated features are dimensionality reduced by a trainable MLP and the dimensionality-reduced features are transformed into embedding conditions. At the same time, the reference image is input into the reference branch to optimize the attention parameters. The generated image is subjected to multi-dimensional quality evaluation. The final quality evaluation score is output by comprehensively considering the relevance of the generated image to the text prompt words and the semantic consistency with the domain knowledge graph. The score is compared with the preset threshold, and the introspection agent mechanism is triggered based on the current number of feedbacks. The latent space representation of the image obtained from the output of the DiT branch is fed into the image decoder to restore the final generated image.
2. The image generation method based on fine-grained multimodal knowledge enhancement according to claim 1, characterized in that, The obtained text prompt words are: For the input text prompts, the core semantic units of the text prompts are obtained through a large language model, a domain knowledge graph is generated, fine-grained contextual information of different semantic units is obtained based on the knowledge graph, and the key feature boundaries of the core semantic units are clarified through multi-step chain reasoning of LLM by referring to the chain-like prompt thinking. Finally, the optimized text prompts are output by integrating contextual information.
3. The image generation method based on fine-grained multimodal knowledge enhancement according to claim 2, characterized in that, The core semantic unit set Key elements that serve as effective references for image generation; implicit relationships Representing core semantic units Semantic connections between other core semantic units; Based on the domain knowledge graph generated by LLM, the attributes of each core semantic unit are extracted to construct a set of prompt word attributes. It employs a contrastive chain thinking prompting strategy, based on implicit relationships. and the set of attributes of semantic units Pairs of positive examples are constructed through multi-step chain reasoning in LLM. and negative examples ; .
4. The image generation method based on fine-grained multimodal knowledge enhancement according to claim 2, characterized in that, Embedding conditions : ； in, Indicates the index of a word in the text prompt. The index of a sample in the subject image set is represented by ||, which represents the concatenation operation between text features and subject image features in the feature dimension direction.
5. The image generation method based on fine-grained multimodal knowledge enhancement according to claim 1, characterized in that, The introspective proxy mechanism works by using a closed-loop iterative process of "generation → evaluation → feedback → correction" to ensure that the generated image matches the input conditions; and it also addresses the input text prompts. The text encoder is used to transform the input condition C to generate the initial image latent representation. , .
6. The image generation method based on fine-grained multimodal knowledge enhancement according to claim 1, characterized in that, The quality assessment score is: ； in, This is a knowledge weighting coefficient, which is dynamically adjusted according to the task scenario. This refers to cross-modal similarity, used to measure global semantic consistency. It is an inspection score for the semantic relevance output of the knowledge graph and the generated image. Below the quality score threshold If so, then introspection is triggered; The introspective agent outputs dual-path feedback signals. and , Optimize the text description of prompt words using LLM. Then, the image details are directly optimized and adjusted through the gradient of the latent space.
7. An image generation device based on fine-grained multimodal knowledge enhancement, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-6.